How to trigger FFAStrans workflow only on complete folder arrival

[gabrielladawn]

Hi all!

I am using FFAStrans to monitor an incoming FTP folder where clients upload batches of video files. Sometimes; large files or entire folders take time to sync & my workflow starts before all expected files have finished arriving, leading to incomplete processing / errors due to missing dependencies.

I would like to trigger the workflow only when the folder is fully uploaded; not when the first file appears. I have tried using a wait node and delay logic, but it's unreliable with varying file sizes and transfer speeds. What I need is a way to detect when the upload activity has stopped for a certain duration (e.g., 5 minutes of no new changes). Checked https://www.ffastrans.com/frm/forum/viewtopic.php?t=1343with Golang Course Online guide for reference.

Has anyone implemented a robust solution for this using FFAStrans nodes or external scripting? I’d love to hear best practices or any creative tricks for handling incomplete FTP syncs gracefully before starting a time-sensitive encoding job.


Thank you!!

[emcodem]

Hi @gabrielladawn,

welcome to the forum and thank you for using FFAStrans! Sorry for the delay.
Unfortunately this is not really straight forward, we cover such complex watchfolder scenarios only in specific cases of e.g. Panasonic P2 but it required some internal programming logic to do it. In case of P2, we have some xml that describes the files we must wait for in order to start the process. In other words, we have a "single file (xml)" that triggers the watchfolder plus some programming logic that checks the xml and waits for the described files.

Of course there are many ways to get to your goal.
Up in front, and this is very important, i would always do the ftp download in a complete separate job but this just downloads the files and does nothing else. So the rest of my thoughts here always assumes that we have a normal local or UNC folder where all files reside.

Let me concentrate on the question: how to detect if there are new files and the oldest one is minimum 5 minutes.
At work i do such stuff only with scheduled jobs, not using normal watchfolders. E.g. schedule a job on webinterface every 1 or 5 mins.
This job can use "files list" processor to count the files and if count > 0, find all files younger than 5 minutes, if no then dispel. If any younger found, dispel.

Another idea can be to avoid webinterface scheduled jobs, use a normal watchfolder but this can only work if you KNOW that there will be a single unique file to watch for that starts the process e.g. "index.xml". If you have such a unique filename you can watch for, all you need to do is to again use files find and count all files younger than 5 minutes, if one or more found then sleep and retry, and if again then sleep longer and retry...