HOLD-Node in Farming environment

Questions and answers on how to get the most out of FFAStrans
ThomasM
Posts: 59
Joined: Wed Feb 22, 2017 6:36 am

HOLD-Node in Farming environment

Post by ThomasM » Fri Feb 08, 2019 10:39 am

Hi all,

I fugured out a problem when using the HOLD-node in a farming-Environment.

I have two PC (win10pro) dedicated to farming. They start FFAStrans from a NAS. They have global workingFolders on that NAS.

I have a more or less complex workflow where a hold-node should wait for files. directly after that hold-node there is a Command-Executioner-node with a MOVE-command on that directory, the foregoing holde-note waits for files.

Basically, this workflow works. BUT - you guess right, now it comes - on several transcodings the hold node waits for files although the Command executioner already moved the directory.

Workflow In short: PickUpFolder - ThreeTranscodeNodes in parallel - DeliveryFolder - HoldeNode - CommandExecutioner. I can provide the whole workflow if someone is interested in it.

Any suggestions?

regards,

tom
Attachments
Scanner_Recherche_und_Ansichtskopie_aus_ProRes_SHARED.xml
(60 KiB) Downloaded 9 times

jordanh
Posts: 130
Joined: Wed Sep 19, 2018 8:11 am

Re: HOLD-Node in Farming environment

Post by jordanh » Fri Feb 08, 2019 12:26 pm

Mahlzeit Thomas,
if i guess right, you want to check the "Synchronize" box in the hold node.

lg,
Harald

ThomasM
Posts: 59
Joined: Wed Feb 22, 2017 6:36 am

Re: HOLD-Node in Farming environment

Post by ThomasM » Fri Feb 08, 2019 2:40 pm

Harald,

thank you for your fast reply. I use the syncronize-function of the hold-node in the same workflow for ending two transcoding nodes. But...

I have to wait for another workflow to end. The other Workflow puts one big AVI ind the DeliveryFolder. So I populated the Hold-node in the actual Workflow with ALL files to be expected in the folder. This is the folder which have to be moved one node after the hold-node. You can see the node in the workflow, it is the second Hold-Node in the flow top-do-down.

In short:

Workflow "A" puts TESTFILE_01_Archivdigitalisat.AVI
TESTFILE_01_Archivdigitalisat.LOG in DeliveryFolder
Workflow "B" puts TESTFILE_01_Recherchekopie.mp4
TESTFILE_01_Ansichtskopie.mp4
TESTFILE_01_Produktionskopie.MOV in DeliveryFolder

Both workflows have different runtimes. At the end of Workflow "B" (which is the Workflow I attached in Post 1) a CommandExecutioner-Node is moving the DeliveryFolder to the Archive-Folder. And this last node executes while or at the same time as the Hold-Node!

A workaround could be to start the Workflow "B" after the AVI-File ist completely delivered. I will check that on monday.

Schönes Wochenende!
Have a nice weekend,

lg / regards
tom

jordanh
Posts: 130
Joined: Wed Sep 19, 2018 8:11 am

Re: HOLD-Node in Farming environment

Post by jordanh » Fri Feb 08, 2019 3:39 pm

Hehe sorry i didn't pay enough Attention.
You do know that when you split from one processor into into 2 or more, you actually generate one additional Job per split.
What you might not know is that when you do not use the "synchronize" Option for reconnecting the split up Jobs into a single Job again, it will actually continue to execute as separate Jobs. In that case, what you see in the workflow Editor is not actually reflecting real-life as all following components even if they are only in one branch are actually executed for each branch you created with the first split.

I hope this was understandable.

Cheers,
Harry

ThomasM
Posts: 59
Joined: Wed Feb 22, 2017 6:36 am

Re: HOLD-Node in Farming environment

Post by ThomasM » Sat Feb 09, 2019 3:16 pm

hmmmm...... as you can see in the attached workflow, the first HoldNode (1) is set to syncronize (After the two FOLDER-nodes). Then there are some Command-Executioner-Nodes followed by a last HoldNode (2) which waits for all files to be delivered in a specified directory. When all files are present, there are the last Command-Executioner-Nodes, one who moves the directory monitored by the last HoldNode (2) to another place.

As you can see I set the Timeout very high to be sure to get all files needed. But the workflow goes on, the directory is moved and the HoldNode (2) is still counting down the timeout in the StatusMonitor. Strange behaviour.

I expected this last HoldNote (2) to just wait for files an then process the rest of the workflow.

What do I get wrong?

any help appreciated...
cheers,

tom

jordanh
Posts: 130
Joined: Wed Sep 19, 2018 8:11 am

Re: HOLD-Node in Farming environment

Post by jordanh » Sat Feb 09, 2019 5:22 pm

Hey Tom,

"Wait For Rech Ans MP4" is not set to synchronize from what i see after importing the workflow; that is why i currently concentrate on this point.
In case you really set the "Wait For Rech Ans MP4" node to synchronize we might have found a bug where exported workflows do not import correctly on another system.

While creating the debug workflow attached, i learned that even using "synchronize" will not cause the 2 branches to get together into 1 again, so please be aware that all nodes after "Wait For Rech Ans MP4" are executed twice.
So, as you execute all the waiting and moving twice, i guess it is just logical that one branch loses (in german "zieht den kürzeren") and waits forever for a file that the other branch already moved.

I think you will succeed when not merging mutliple branches back into the same tree again. So maybe just delete the connection between "Deliver Ansicht 2 WD" and "Wait For Rech Ans MP4"

Here a workflow i use to debug the branch merging behaviour of ffastrans.
Please tell me if the "synchronize" option on the "Hold-sleep1" node is enabled after you imported the workflow and if you like, test it once with the synchronize option and once without - you might always see 4 text files in c:\temp, but once with 10 seconds delay and once only with one second delay.
thomas_debug.xml
(11.76 KiB) Downloaded 7 times
And by the way, a little self-advertising: as you seem to be using ffastrans heavily, i would like to know if my webinterface is of use for you or whats missing to make it useful for you ;-)

cheers,
Harry

ThomasM
Posts: 59
Joined: Wed Feb 22, 2017 6:36 am

Re: HOLD-Node in Farming environment

Post by ThomasM » Sat Feb 09, 2019 10:19 pm

guten Abend Harry,

Thank you so much for your support. I tried your debug-workflow. The behaviour of the HoldNotes is somewhat strange.

#1: I programmed my workflows with the last 64-Bit version (9.4.0 or 9.4.1 - I cannot tell correctly what I installed in the studio last time...)
#2: I have the latest version of FFAStrans without the latest patch here on my laptop at home on win 10 pro
#3: I imported my workflow from post 1 of this thread and the HoldNode was set to "syncronize"
#4: I imported your debug-Workflow (thx-a-lot...!) - ALL HoldNodes are set to Syncronize with timeout 3600...
#5: So I checked my imported Workflow from post 1. BOTH HoldNodes are set to Syncronize Timeout 3600.

Maybe, is it possible for you to import your ownor my workflow to another machine at your place and check the Nodes? It seems that they all do what the first node does. I set the second node to a brutal 72000 because the .AVIs are VERY big, now the node is set to sync timeout 3600.

You maybe right that there could be a bug by interpreting these Nodes...

By the way: I will check out the benefits of your web-Interface. I just have some problems with the REST-settngs... but that is another story.

Anyway: Thank you Harry for the help (and of course Steinar for this wonderful piece of really helpful software)!

enjoy your weekend,
all the best,
tom

jordanh
Posts: 130
Joined: Wed Sep 19, 2018 8:11 am

Re: HOLD-Node in Farming environment

Post by jordanh » Sun Feb 10, 2019 7:54 pm

Hey Tom,
let's not concentrate on the issue about if the synchronized is enabled after importing or not for the moment. I am not 100% certain that we got down to the bottom of your problem. Can you try this- i think it should solve your problem:
-) delete the connection between "Deliver Ansicht 2 WD" and "Wait For Rech Ans MP4"

Also, as you mentioned you don't know how to use user variables in a PM, i had a deeper look at what you do in the first processing nodes. I believe this minified version should work for you, it uses only one user variable and parses the FPS value dynamically from the filename and inserts the parsed value at the avisynth script (check out the one remaining avisynth node). The function used in the populate node is:

Code: Select all

$regext("%s_source%","(\d+)FPS")
Which basically translates to this regex doing all the magic:

Code: Select all

(\d+)FPS
Tis means "take all digits before FPS in the filename. So it assumes that the 2 digits before the string "FPS" in the filename are standing alone as digits, e.g. for 24FPS, the filename should not be like 21398924FPS, this would cause the variable to be set to the value 21398924. A filename like "tolles-file_24FPS_schauDirDasAn.mov" or "framerate24fps.mov" is OK.

ThomasM
Posts: 59
Joined: Wed Feb 22, 2017 6:36 am

Re: HOLD-Node in Farming environment

Post by ThomasM » Mon Feb 11, 2019 3:48 pm

Harry,

mea culpa - I installed ffastrans into c:\programs, but that is supervised by windows. So changes to scripts were not possible. Later I figured that out. Outside the Program-Folder everything works and the nodes are "sync" or "wait for Files..." as expected.

Thank you very much for your help on that Workflow. That is more than I could expect. I am busy the next days and will try everything on Wednesday.

In the Meantime I checked my Workflow in the studio and you are right, it was NOT set to sync. I switched this to sync. Report follows...

It is a good advise that the Hold node also functions as a junction back to one workflow - I was not aware of that.

Again thanks a lot

regards,

tom

ThomasM
Posts: 59
Joined: Wed Feb 22, 2017 6:36 am

Re: HOLD-Node in Farming environment

Post by ThomasM » Mon Feb 11, 2019 6:07 pm

jordanh wrote:
Sun Feb 10, 2019 7:54 pm

(...)
The function used in the populate node is:

Code: Select all

$regext("%s_source%","(\d+)FPS")
Which basically translates to this regex doing all the magic:

Code: Select all

(\d+)FPS
Tis means "take all digits before FPS in the filename. So it assumes that the 2 digits before the string "FPS" in the filename are standing alone as digits, e.g. for 24FPS, the filename should not be like 21398924FPS, this would cause the variable to be set to the value 21398924. A filename like "tolles-file_24FPS_schauDirDasAn.mov" or "framerate24fps.mov" is OK.
puuuh...... I am just studying regex (https://www.debuggex.com/cheatsheet/regex/pcre) for understanding. I never understood what that is. Thank you Harry!

I am no programmer and do not have any deep experience in cmd or batch programming, so ss64 and Stackoverflow are my places to go to. Hard work... thats why I try to get variables done by For /f and text-files until now.

I will give it a try!

regards,

tom

Post Reply