FFMpeg encoding CPU limit for new jobs

Use this forum to request features. Maybe you'll get lucky!
Post Reply
veks
Posts: 15
Joined: Fri Oct 25, 2019 6:51 am

FFMpeg encoding CPU limit for new jobs

Post by veks » Thu Nov 21, 2019 1:13 pm

Hi all!

I see a bottleneck in my workflow, or FFAStrans as such.
Everything works fine, but when FFAStrans takes 1 file to transcode, and when it gets to multiple nodes for encoding multiprofile (6 profiles) it takes randomly let's say 3 jobs (3 jobs set as maximum), and then starts transcoding 3 profiles at the same time hogging a CPU to 99%.

Problem is that if I set a job to 1, some of the condition nodes won't fire, and then FFAStrans would just wait and wait, and won't do nothing.

What if you could set the number of simultaneous ENCODING FFMpeg jobs and CPU percentage when to take new ENCODING jobs?
I mean, it might be a problem with my workflow, some example of a workflow with at least 6 multi profile h264 transcoding would be also great.

If I missed an topic, please move it to somewhere else.

Thanks a lot!

EDIT 1: or ability to set maximum threads per encoding job. It's a must feature. I see no way to adding this option, only if using a totally custom FFMpeg command.

emcodem
Posts: 445
Joined: Wed Sep 19, 2018 8:11 am

Re: FFMpeg encoding CPU limit for new jobs

Post by emcodem » Thu Nov 21, 2019 2:16 pm

Hey veks!
i am sure that @admin has something to say about that; personally i don't like the CPU limit feature at all, it should be set in transcoding slots only than in CPU percentage but that is not a matter of discussion.
However, i have an important question for you: do i understand correct that you are using a single machine for your encodings and your ultimate goal is to utilize it's resources as best as possible? - additionally you cannot affort to just add enough System Resources or Servers to your Farm, correct?

Thanks!
-emcodem

veks
Posts: 15
Joined: Fri Oct 25, 2019 6:51 am

Re: FFMpeg encoding CPU limit for new jobs

Post by veks » Thu Nov 21, 2019 3:16 pm

emcodem wrote:
Thu Nov 21, 2019 2:16 pm
Hey veks!
i am sure that @admin has something to say about that; personally i don't like the CPU limit feature at all, it should be set in transcoding slots only than in CPU percentage but that is not a matter of discussion.
However, i have an important question for you: do i understand correct that you are using a single machine for your encodings and your ultimate goal is to utilize it's resources as best as possible? - additionally you cannot affort to just add enough System Resources or Servers to your Farm, correct?

Thanks!
-emcodem
We're using an virtual machine with Xeon 6130 CPU.
But yeah, we would like to add another nodes, servers into farm.

admin
Site Admin
Posts: 1303
Joined: Sat Feb 08, 2014 10:39 pm

Re: FFMpeg encoding CPU limit for new jobs

Post by admin » Thu Nov 21, 2019 4:53 pm

Hi veks,

First I just want to be clear on what you problem is. So you have the 99% CPU hogging, and the nodes not starting when you have limited simultaneous jobs to 1. Now, the latter worries me the most. Could you please share some details, or better yet, export and post your workflow?

So regaring CPU control... utilizing all the CPU power that is is sort of what encoders should do. I can see this is a challenge if you're using the same host for other tasks but the nature of this program is to run on separate server dedicated to transcoding. If you want to do other stuff, setting max jobs to 1 should do but not if there is a bug of course preventing FFAStrans from working correctly with this setting. So we need to address that first, before adding ad-hoc CPU control.

-steinar

Post Reply