How to setup farm correctly? Errors when farming

[ThomasM]

...sorry, the attachment-function is not stable on this machine. So... next try:

[ThomasM]

WorkingFolders_05JUL2019.7z

(948.05 KiB) Downloaded 365 times

nu aber...

[emcodem]

Those logs are fantastic, thanks a lot for taking the time to collect all the stuff.

In difference to the last error we detected which was about a user_variable not being set, this time we have a totally different error:

In the file ~20190705-152651-129-1705188E69AF.mov_ffreport.txt, we see that the last line is

Code: Select all

frame= 5019 fps=2.6 q=-0.0 size= 5854464kB time=00:03:20.08 bitrate=239702.9kbits/s speed=0.103x

which looks like the encoding process was just interrupted while it was running.

The first error which was about a non set user_variable did not indicate if we face file access errors or maybe some ffastrans internal error.
This second error more or less indicates that there is no error in ffastrans as ffmpeg was writing the "not finished" ffreport file.

So most likely we should take care about the storage connection to the cache Z:.
As a first thing, please check the network stability by executing

Code: Select all

ping-t %STOAGEIP% > c:\temp\ping.txt

for some minutes. Take the IP of the storage that is mapped to Z: ... After some minutes, check if the ping looks stable. A typical example for a normal ping result is like the time always < 5ms and not varying a lot. If you have a single "lost" ping, you are definitely facing a networking problem. If the ping time varies a lot above 100ms and such, the same...

[ThomasM]

Hey emcodem,
thank you for this ping-tip! great help.

I let it run from either Renderer, so I got two Ping.txt-files. No lost pings on both. Max time is <12ms on renderer 1 for one time (as I can see...) and <9ms on renderer 2 once.

I attach the files for reference.

I let the Files run through renderer 2 (those files which came out with the logs I sent yesterday from both renderes farming) - no error. The thing here is that also renderer 2 alone also looks on Z: and Y: via Network. In other words: I do not let FFAStrans run locally on renderer 1 or 2 when not farming. I just exclude one of them from the Farming-Tab in Workflow-properties. Maybe this helps a bit...

As mentioned earlier, in another thread, maybe it is the size of the files which lead to network-problems? When I do a little stress-test with some 10 files about 300MB of size each, I got 80% to 100% success. When working on real-life-files which are about 150GB ... 350GB of size the errors occur. The last workflow-run (from the logs I sent) was 0% success (output to the Archive-Storage Y:\).

Regards,
tom

[emcodem]

Well i guess we should not poke around in the dark. As in any other complex problem, we need to simplify and reproduce.
To simplify, we need to take out components one by one, especially software. E.g. are we able to reproduce any problem when not using ffastrans.

Can you try to run the transcoding command manually a few times and see if you succeed to get it to fail once?

Code: Select all

"S:\\NICHT LOESCHEN FFAStrans\\FFAStrans\\Processors\\FFMpeg\\x64\\ffmpeg.exe" -analyzeduration 33554432 -i "Z:\\WATCHFOLDER\\FA_Scanner\\SCANNER-STUMM\\S8FA10170INV19504_Archivdigitalisat.avi" -c:v prores_ks -b:v 1500k -c:a pcm_s16le -b:a 128k -vendor apl0 -profile:v 3 -pix_fmt yuv422p10le -max_muxing_queue_size 125 -metadata "creation_time=now" -y "Z:\\FFasTrans_CACHE\\20190626104342\\20190705-152436-363-DC8805101ECA\\~20190705-152651-129-1705188E69AF.mov"

Maybe you have to change the output directory, but keep it in Z:\FFASTRANS_Cache\. While the transcoding is running, check the ping from time to time please. It will go up and down a little but it should not be over ~100ms at any time. Also check the ping to the server behind S:

[ThomasM]

Hey,

sorry for this delay - I was a few days out in the field doing some documentaty. Now the summer-holidays are next and I am afraid, I cannot get to testing for the next three weeks.

Sorry for that, I will be back ASAP.

cheers,

tom

[emcodem]

Happy holidays Thomas

[ThomasM]

Hi there, right back again.

@emcodem: thanks...

I came back in the studio and the other guys made some new scans. 22 Files all between 150 GB - 300 GB in size. I let it run through FFAStrans on a single renderer.

All files were transcoded fine.

BUT...

It seems that some files are treated twice or on different threads at the same time or slightly time-delayed. I don't know how to make this clear, have a look at the status-log. ALL files are processed fine but there are in fact some "job ended..." entries. This was an RealLife-run. When I do the same wit small files (about 120 ... 200 MB in size) all runs through without any hassle.

So I assume there is a kind of timeout or timing-issue.

BTW: I set the Properties -> General -> Timeout Level to 10.

@emcodem: I will check the CommandLine the next days. Sorry for this delay...

cheers,
tom