Hi
I have a FFAStrans farm with 8 nodes.
I am curious on the decision process that decides how the tasks are allocated to each node, and what decides the order in which the tasks are allocated?
Does it give the next task to the node with the lowest CPU, for example?
Is there any way to prioritise the nodes in an order of preference?
Thanks
Transcode Farm - Processing of Node Allocation?
-
yekootmada
- Posts: 1
- Joined: Thu Oct 12, 2023 1:19 pm
Re: Transcode Farm - Processing of Node Allocation?
Hi yekootmada, welcome to the Forum and thank you for using FFAStrans!
In a Farm, Each host checks constantly for new tickets as described here: https://ffastrans.com/wiki/doku.php?id=system:processes
If there is a new ticket, each host decides on it's own if it has the resources free to execute the job, e.g. if it's cpu is utilized less than the cpu setting.
There is no communication between the hosts. In fact from this perspective, each host kind of believes it is running alone.
In the end, it is the first one that "accidently" successfully moves the ticket json file from pending folder to running.
The point where you control about hosts vs workflows is in the include/exclude list of the workflow.
In a Farm, Each host checks constantly for new tickets as described here: https://ffastrans.com/wiki/doku.php?id=system:processes
If there is a new ticket, each host decides on it's own if it has the resources free to execute the job, e.g. if it's cpu is utilized less than the cpu setting.
There is no communication between the hosts. In fact from this perspective, each host kind of believes it is running alone.
In the end, it is the first one that "accidently" successfully moves the ticket json file from pending folder to running.
The point where you control about hosts vs workflows is in the include/exclude list of the workflow.
emcodem, wrapping since 2009 you got the rhyme?
Re: Transcode Farm - Processing of Node Allocation?
Hi yekootmada,
There are some logic to how the different hosts pick up jobs and in the long run there should be an even distribution modified by the resource bias on each host. So a "powerful" host is more likely to pick up a job before a "weaker" host. But for every job a host picks up, the bias shifts in disfavior of the host picking up the job. Or rather, picking up the next node cause a job ticket is only relevant for one node. Throughout a complete job there are many tickets realted to the same mother job.
Anyway, in the upcoming 1.4 we have addes some interresting features that enables you to allocate nodes to certain hosts. So for example, if you have a node that does heavy encoding, you can assign this to ONLY go to the host that has the most powerful configuration.
Another new feature is to allocate a node to use more job slots than what is available on weaker hosts. So with that feature you can make sure all nodes with higher job slots set only goes to the hosts that has enough CPU cores to deal with them. So for example if you set a node to take 4 job slots and you have one host set to 3 slots and one set to 8, the node will allways go to the one with 8 slots cause the one with 3 can't fit the 4-slot node.
Please note that in this comment a "host" is the computer running ffastrans and a "node" is the node boxes you use in your workflow.
-steinar
There are some logic to how the different hosts pick up jobs and in the long run there should be an even distribution modified by the resource bias on each host. So a "powerful" host is more likely to pick up a job before a "weaker" host. But for every job a host picks up, the bias shifts in disfavior of the host picking up the job. Or rather, picking up the next node cause a job ticket is only relevant for one node. Throughout a complete job there are many tickets realted to the same mother job.
Anyway, in the upcoming 1.4 we have addes some interresting features that enables you to allocate nodes to certain hosts. So for example, if you have a node that does heavy encoding, you can assign this to ONLY go to the host that has the most powerful configuration.
Another new feature is to allocate a node to use more job slots than what is available on weaker hosts. So with that feature you can make sure all nodes with higher job slots set only goes to the hosts that has enough CPU cores to deal with them. So for example if you set a node to take 4 job slots and you have one host set to 3 slots and one set to 8, the node will allways go to the one with 8 slots cause the one with 3 can't fit the 4-slot node.
Please note that in this comment a "host" is the computer running ffastrans and a "node" is the node boxes you use in your workflow.
-steinar
Re: Transcode Farm - Processing of Node Allocation?
Thats why i call processors processors and not nodes
emcodem, wrapping since 2009 you got the rhyme?