Too Many Connectors

Here you can submit bugreports
Post Reply
Noqras
Posts: 36
Joined: Mon Dec 16, 2019 4:32 am

Too Many Connectors

Post by Noqras » Wed May 27, 2020 5:02 am

Well, some time ago I managed to use the maximum number of processors, that was fixed and I can have tons of processors now. But just now as I was about to clean up my workflow to be a bit more sparing on the processors, it looks like I broke my workflow by having too many connectors between processors, see the attached screenshots.

I was able to create the workflow fine without error, but then I deleted a few processors and then tried to add a new connector and I get the Line 58774 error. If I deleted a processor and then try to delete another I get the second error Line 58504.

I found a workaround which is delete some connectors first, then I can delete processors or make some new connectors all I want without errors. The errors only occur after first having a huge amount of connectors and processors and then deleting a processor. If I get rid of some connectors first, its fine.

So I've come to the conclusion that my problem is too many connectors. As you can see my workflow is a little intense, so it actually doesn't surprise me.
Attachments
Error 2 Line 58504
Error 2 Line 58504
ffastrans_error_2.JPG (134.93 KiB) Viewed 1080 times
Error 1 Line 58774
Error 1 Line 58774
ffastrans_error.JPG (212.34 KiB) Viewed 1080 times
FFAStrans 1.0.0.5
HP DL360p Gen8 - 2x Intel Xeon E5-2670 - 32GB RAM
2x HP SL230s
Windows Server 2008 R2 SP1 64-bit

Noqras
Posts: 36
Joined: Mon Dec 16, 2019 4:32 am

Re: Too Many Connectors

Post by Noqras » Wed May 27, 2020 5:16 am

Nevermind, my workaround doesn't fully work, I think I'm going to need to delete all connectors and then maybe it will be normal again. If I delete a bunch of connectors now and then start delete processors, the remaining connectors start moving to random places.
FFAStrans 1.0.0.5
HP DL360p Gen8 - 2x Intel Xeon E5-2670 - 32GB RAM
2x HP SL230s
Windows Server 2008 R2 SP1 64-bit

admin
Site Admin
Posts: 1446
Joined: Sat Feb 08, 2014 10:39 pm

Re: Too Many Connectors

Post by admin » Wed May 27, 2020 6:46 am

Hi Noqras,

Thanks for reporting! :-) I don't suppose you could share a version of your broken workflow? It would be much easier to investigate ;-)

-steinar

emcodem
Posts: 629
Joined: Wed Sep 19, 2018 8:11 am

Re: Too Many Connectors

Post by emcodem » Wed May 27, 2020 8:25 am

Aye,
i spent some time checking what you do with all the nodes and i am under the impression it would be better for you to get in touch with variables. This way you might reduce your workflow to 1-3 encoding nodes only.

For example, to get started, lets reduce all the "is there no audio" nodes and also reduce half of the encoding nodes:
1) take a deep breath, get a coffee, backup your existing workflow
2) insert one "is ther no audio" conditional in the main branch (before fps check)
3) insert 2 "populate" nodes in the main branch after the before inserted "is there no audio" check
4) in the success path (there is really NO audio), set 2 user variables:

Code: Select all

%s_lavfi% =  -f lavfi -i anullsrc=r=48000 -shortest 
%s_ac% = -ac 2
after3.png
after3.png (7.18 KiB) Viewed 1058 times


5) in the error path's populate parocessor, set the s_lavfi and s_ac variables to "nothing"
6) both populate connector's outputs are connected to the "there is 1 or less audios processor
before1.png
before1.png (10.84 KiB) Viewed 1064 times
after1.png
after1.png (12.35 KiB) Viewed 1064 times

6) delete all ffmpeg nodes for the old "no audio" path and all "is there no audio" conditionaly before the encoding nodes
7) alter the old "there is audio" ffmpeg command from this

Code: Select all

%ComSpec% /c ""%s_ffmpegx64%" -i "%s_source%" -c:v dnxhd -vf "scale=1920x1080:in_range=tv:out_range=tv,fps=30000/1001,format=yuv422p" -vsync 1 -b:v 145M -c:a pcm_s24le -ac 2 -ar 48000 "M:\DPShare\Aurora\DNxHD Profile\output\QC Pass\%s_original_name%.mxf""
to this:

Code: Select all

%ComSpec% /c ""%s_ffmpegx64%" -i "%s_source%" %s_lavfi% -c:v dnxhd -vf "scale=1920x1080:in_range=tv:out_range=tv,fps=30000/1001,format=yuv422p" -vsync 1 -b:v 145M -c:a pcm_s24le %s_ac%  -ar 48000 "M:\DPShare\Aurora\DNxHD Profile\output\QC Pass\%s_original_name%.mxf""
You see? ...all i did was inserting the variables %s_lavfi% and %s_ac% into the existing "there is audio" ffmpeg command. If the variables are set, audio will be created, if not, we go with the contained audio..
before2.png
before2.png (25.1 KiB) Viewed 1064 times
...i am pretty sure we could delete of most of encoding nodes by using further variables for fps, resolution and encoding settings (bitrate)

Also, it would be good if you insert a populate processor at start to set the output path, instead of writing the output path into each and every ffmpeg line:
%s_output_path% = M:\DPShare\Aurora\DNxHD Profile\output\QC Pass\
This way, if you need to change the path, you only need to change it at ONE processor, not at 50...

admin
Site Admin
Posts: 1446
Joined: Sat Feb 08, 2014 10:39 pm

Re: Too Many Connectors

Post by admin » Wed May 27, 2020 9:27 am

Noqras, I found the cause of the first bug and it was a nasty one! :oops: It's fixed and I'm testing now. The second bug is probably related but I have not been able to reproduce that one. I will send you a patch for you to test when I'm done.

Thanks again for reporting! :-)

-steinar

Noqras
Posts: 36
Joined: Mon Dec 16, 2019 4:32 am

Re: Too Many Connectors

Post by Noqras » Wed May 27, 2020 9:17 pm

Emcodem!

Thank-you so much for your help on this, yesterday I was thinking as this was happening that first of all there is probably a better way to do with involving variables and second of all I really just need to sit down and work that out.

I have removed 1/3 of the encode processors by leaving a lanczos flag on all processors thus being able to remove the redundant processors for 4k and 2k, then using variables for audio that you laid out will make this much more efficient.
FFAStrans 1.0.0.5
HP DL360p Gen8 - 2x Intel Xeon E5-2670 - 32GB RAM
2x HP SL230s
Windows Server 2008 R2 SP1 64-bit

emcodem
Posts: 629
Joined: Wed Sep 19, 2018 8:11 am

Re: Too Many Connectors

Post by emcodem » Thu May 28, 2020 6:49 am

Oh you will be so much more mighty when you work with variables, i am looking forward to it. Let me know all your doubts and experiences.
If you have troubles, please spend some time on the workflow example thread, maybe it is of help: viewtopic.php?t=849

And also please don't forget to thank @admin for locating the bug.. he works his ass of for all of us!

Noqras
Posts: 36
Joined: Mon Dec 16, 2019 4:32 am

Re: Too Many Connectors

Post by Noqras » Fri May 29, 2020 12:21 am

Steinar,

I tested your patch and it works perfectly! I don't get the first error anymore and it looks like the second error was more or less an extension of the first error as it would only ever happen after the first. Thank-you so much!

I am reducing my workflow down quite a bit now, but we now know that workflows have the potential of being huge without any issues now which is great.

Thanks!
FFAStrans 1.0.0.5
HP DL360p Gen8 - 2x Intel Xeon E5-2670 - 32GB RAM
2x HP SL230s
Windows Server 2008 R2 SP1 64-bit

admin
Site Admin
Posts: 1446
Joined: Sat Feb 08, 2014 10:39 pm

Re: Too Many Connectors

Post by admin » Fri May 29, 2020 7:55 am

That's great news, Noqras :-)

Again, thanks for reporting the bug!

-steinar

Post Reply