Check for updated files

Questions and answers on how to get the most out of FFAStrans
Post Reply
DsmithTopGun
Posts: 9
Joined: Mon Sep 17, 2018 12:30 am

Check for updated files

Post by DsmithTopGun » Mon Oct 08, 2018 6:05 pm

Is there any way for the folder monitor to check for files that have changed size since the last time it ran? I tried the "check growing files" and it's not re-encoding files that have changed sizes since last time it was run.

admin
Site Admin
Posts: 1040
Joined: Sat Feb 08, 2014 10:39 pm

Re: Check for updated files

Post by admin » Mon Oct 08, 2018 7:58 pm

Hi DsmithTopGun,

"Check growing files" does not work that way: Once the monitored file is deemed finished with growing, the file record will be created and FFAStrans does not pick up that file again unless the record is deleted. So no changes to the file will make any difference.

Let me ask you, is the file in your user scenario a file that's once in a while being updated?

-steinar

DsmithTopGun
Posts: 9
Joined: Mon Sep 17, 2018 12:30 am

Re: Check for updated files

Post by DsmithTopGun » Mon Oct 08, 2018 9:49 pm

They're being pulled from our phone system(every 10 minutes currently) but for calls longer than 10 minutes FFAStrans is simply copying the file once and then done and not "rechecking" if the file size still matches what it did originally and creating a 10 minute chunk. I can reprocess the folder manually but that defeats the point of automation XD

admin
Site Admin
Posts: 1040
Joined: Sat Feb 08, 2014 10:39 pm

Re: Check for updated files

Post by admin » Tue Oct 09, 2018 5:25 pm

Hi,

Please import the attached workflow. It should point you in the right direction. Change the monitor and delivery paths and file names to meet your needs.

-steinar
Attachments
Dynamic monitor.xml
(13.17 KiB) Downloaded 6 times

DsmithTopGun
Posts: 9
Joined: Mon Sep 17, 2018 12:30 am

Re: Check for updated files

Post by DsmithTopGun » Tue Oct 09, 2018 7:10 pm

So it looks like this might be a solution? But I'm trying to understand how it works and don't want to reprocess more than I have to.
I'm monitoring at a "Year" level so I'm monitoring here "D:\Public\Recordings\2018" but the files are placed at a day level Ex "D:\Public\Recordings\2018\10\09" is deleting the file record(step 2) going to wipe all my file records or just the records of the files that have changed size?

admin
Site Admin
Posts: 1040
Joined: Sat Feb 08, 2014 10:39 pm

Re: Check for updated files

Post by admin » Tue Oct 09, 2018 8:40 pm

If FFAStrans is going to pick up any file size changes for the monitored files, the record must be deleted every run, or else it will not work. If you just need monitoring each new day you can use the time-variables to change monitor folder dynamically:

D:\Public\Recordings\%i_year%\%i_mon%\%i_mday%

This way the amount of files monitored is limited to the ones created "today".

Hope this helps :-)

-steinar

DsmithTopGun
Posts: 9
Joined: Mon Sep 17, 2018 12:30 am

Re: Check for updated files

Post by DsmithTopGun » Tue Oct 09, 2018 9:56 pm

It does a a little although with 700+ calls a day it's still going to reprocess more files than i'd prefer, any chance we'll see a different solution down the road?

admin
Site Admin
Posts: 1040
Joined: Sat Feb 08, 2014 10:39 pm

Re: Check for updated files

Post by admin » Wed Oct 10, 2018 4:31 pm

Oh, I was under the impression that there was just one file being updated once in a while. Then the workflow I sent you will fail if you have several files. But ok, a small modification and we can make it work in your scenario. But of course, doing this workflow every minute on 700+ files is a lot of work and is not very smooth but it's the best I've got for you at this point. However, I'm going to take a closer look at how we can overcome this limitation.
Actually, FFAStrans once did take changed file-size into account and also the date modified but it came in conflict with the "Growing files" functionality and it had other problems so I was force dto remove it in order to get a reliable watch folder approach.

-steinar

DsmithTopGun
Posts: 9
Joined: Mon Sep 17, 2018 12:30 am

Re: Check for updated files

Post by DsmithTopGun » Wed Oct 10, 2018 9:41 pm

I did go ahead and add it to my workflow, the files are only being dumped once an hour and 700 could be a slow day, yesterday was 878 actually XD I've slowed it down to ingest every hour(there's a separate process that's bringing them from the phone system to the ingest folder) but reprocessing the whole days files towards the last few hours means it takes almost the entire hour to update new files(is there a way to speed it up? It's running on a rather large E5-2650 V3 server and seems like it should be much quicker converting wav files to mp3) I've attached my workflow(with your additions) if it helps
Attachments
FTP-2018_copy.xml
(15.71 KiB) Downloaded 2 times

Post Reply