How to setup farm correctly? Errors when farming

Questions and answers on how to get the most out of FFAStrans
ThomasM
Posts: 120
Joined: Wed Feb 22, 2017 6:36 am

Re: How to setup farm correctly? Errors when farming

Post by ThomasM » Tue Jul 09, 2019 3:22 pm

...sorry, the attachment-function is not stable on this machine. So... next try:

ThomasM
Posts: 120
Joined: Wed Feb 22, 2017 6:36 am

Re: How to setup farm correctly? Errors when farming

Post by ThomasM » Tue Jul 09, 2019 3:34 pm

WorkingFolders_05JUL2019.7z
(948.05 KiB) Downloaded 41 times
nu aber...

emcodem
Posts: 445
Joined: Wed Sep 19, 2018 8:11 am

Re: How to setup farm correctly? Errors when farming

Post by emcodem » Tue Jul 09, 2019 7:40 pm

Those logs are fantastic, thanks a lot for taking the time to collect all the stuff.

In difference to the last error we detected which was about a user_variable not being set, this time we have a totally different error:

In the file ~20190705-152651-129-1705188E69AF.mov_ffreport.txt, we see that the last line is

Code: Select all

frame= 5019 fps=2.6 q=-0.0 size= 5854464kB time=00:03:20.08 bitrate=239702.9kbits/s speed=0.103x
which looks like the encoding process was just interrupted while it was running.

The first error which was about a non set user_variable did not indicate if we face file access errors or maybe some ffastrans internal error.
This second error more or less indicates that there is no error in ffastrans as ffmpeg was writing the "not finished" ffreport file.

So most likely we should take care about the storage connection to the cache Z:.
As a first thing, please check the network stability by executing

Code: Select all

ping-t %STOAGEIP% > c:\temp\ping.txt

for some minutes. Take the IP of the storage that is mapped to Z: ... After some minutes, check if the ping looks stable. A typical example for a normal ping result is like the time always < 5ms and not varying a lot. If you have a single "lost" ping, you are definitely facing a networking problem. If the ping time varies a lot above 100ms and such, the same...

ThomasM
Posts: 120
Joined: Wed Feb 22, 2017 6:36 am

Re: How to setup farm correctly? Errors when farming

Post by ThomasM » Wed Jul 10, 2019 10:10 am

Hey Harry,
thank you for this ping-tip! great help.

I let it run from either Renderer, so I got two Ping.txt-files. No lost pings on both. Max time is <12ms on renderer 1 for one time (as I can see...) and <9ms on renderer 2 once.

I attach the files for reference.

I let the Files run through renderer 2 (those files which came out with the logs I sent yesterday from both renderes farming) - no error. The thing here is that also renderer 2 alone also looks on Z: and Y: via Network. In other words: I do not let FFAStrans run locally on renderer 1 or 2 when not farming. I just exclude one of them from the Farming-Tab in Workflow-properties. Maybe this helps a bit...

As mentioned earlier, in another thread, maybe it is the size of the files which lead to network-problems? When I do a little stress-test with some 10 files about 300MB of size each, I got 80% to 100% success. When working on real-life-files which are about 150GB ... 350GB of size the errors occur. The last workflow-run (from the logs I sent) was 0% success (output to the Archive-Storage Y:\).

Regards,
tom
Attachments
Ping_R1_and_R2.7z
(1.83 KiB) Downloaded 23 times

emcodem
Posts: 445
Joined: Wed Sep 19, 2018 8:11 am

Re: How to setup farm correctly? Errors when farming

Post by emcodem » Wed Jul 10, 2019 12:32 pm

Well i guess we should not poke around in the dark. As in any other complex problem, we need to simplify and reproduce.
To simplify, we need to take out components one by one, especially software. E.g. are we able to reproduce any problem when not using ffastrans.

Can you try to run the transcoding command manually a few times and see if you succeed to get it to fail once?

Code: Select all

"S:\\NICHT LOESCHEN FFAStrans\\FFAStrans\\Processors\\FFMpeg\\x64\\ffmpeg.exe" -analyzeduration 33554432 -i "Z:\\WATCHFOLDER\\FA_Scanner\\SCANNER-STUMM\\S8FA10170INV19504_Archivdigitalisat.avi" -c:v prores_ks -b:v 1500k -c:a pcm_s16le -b:a 128k -vendor apl0 -profile:v 3 -pix_fmt yuv422p10le -max_muxing_queue_size 125 -metadata "creation_time=now" -y "Z:\\FFasTrans_CACHE\\20190626104342\\20190705-152436-363-DC8805101ECA\\~20190705-152651-129-1705188E69AF.mov"
Maybe you have to change the output directory, but keep it in Z:\FFASTRANS_Cache\. While the transcoding is running, check the ping from time to time please. It will go up and down a little but it should not be over ~100ms at any time. Also check the ping to the server behind S:

ThomasM
Posts: 120
Joined: Wed Feb 22, 2017 6:36 am

Re: How to setup farm correctly? Errors when farming

Post by ThomasM » Wed Jul 17, 2019 2:18 pm

Hey,

sorry for this delay - I was a few days out in the field doing some documentaty. Now the summer-holidays are next and I am afraid, I cannot get to testing for the next three weeks.

Sorry for that, I will be back ASAP.

cheers,

tom

emcodem
Posts: 445
Joined: Wed Sep 19, 2018 8:11 am

Re: How to setup farm correctly? Errors when farming

Post by emcodem » Thu Jul 18, 2019 7:46 pm

Happy holidays Thomas :)

ThomasM
Posts: 120
Joined: Wed Feb 22, 2017 6:36 am

Re: How to setup farm correctly? Errors when farming

Post by ThomasM » Thu Aug 15, 2019 3:24 pm

Hi there, right back again.

@Harry: thanks... :)

I came back in the studio and the other guys made some new scans. 22 Files all between 150 GB - 300 GB in size. I let it run through FFAStrans on a single renderer.

All files were transcoded fine.

BUT...

It seems that some files are treated twice or on different threads at the same time or slightly time-delayed. I don't know how to make this clear, have a look at the status-log. ALL files are processed fine but there are in fact some "job ended..." entries. This was an RealLife-run. When I do the same wit small files (about 120 ... 200 MB in size) all runs through without any hassle.

So I assume there is a kind of timeout or timing-issue.

BTW: I set the Properties -> General -> Timeout Level to 10.

@Harry: I will check the CommandLine the next days. Sorry for this delay...

cheers,
tom
Attachments
log.txt
(7.36 KiB) Downloaded 29 times

Post Reply