Too Many Connectors

Here you can submit bugreports
Noqras
Posts: 40
Joined: Mon Dec 16, 2019 4:32 am

Too Many Connectors

Post by Noqras »

Well, some time ago I managed to use the maximum number of processors, that was fixed and I can have tons of processors now. But just now as I was about to clean up my workflow to be a bit more sparing on the processors, it looks like I broke my workflow by having too many connectors between processors, see the attached screenshots.

I was able to create the workflow fine without error, but then I deleted a few processors and then tried to add a new connector and I get the Line 58774 error. If I deleted a processor and then try to delete another I get the second error Line 58504.

I found a workaround which is delete some connectors first, then I can delete processors or make some new connectors all I want without errors. The errors only occur after first having a huge amount of connectors and processors and then deleting a processor. If I get rid of some connectors first, its fine.

So I've come to the conclusion that my problem is too many connectors. As you can see my workflow is a little intense, so it actually doesn't surprise me.
Attachments
Error 2 Line 58504
Error 2 Line 58504
ffastrans_error_2.JPG (134.93 KiB) Viewed 9704 times
Error 1 Line 58774
Error 1 Line 58774
ffastrans_error.JPG (212.34 KiB) Viewed 9704 times
FFAStrans 1.2.0
HP DL360p Gen8 - 2x Intel Xeon E5-2670 - 32GB RAM
2x HP SL230s
Windows Server 2008 R2 SP1 64-bit
Noqras
Posts: 40
Joined: Mon Dec 16, 2019 4:32 am

Re: Too Many Connectors

Post by Noqras »

Nevermind, my workaround doesn't fully work, I think I'm going to need to delete all connectors and then maybe it will be normal again. If I delete a bunch of connectors now and then start delete processors, the remaining connectors start moving to random places.
FFAStrans 1.2.0
HP DL360p Gen8 - 2x Intel Xeon E5-2670 - 32GB RAM
2x HP SL230s
Windows Server 2008 R2 SP1 64-bit
admin
Site Admin
Posts: 1658
Joined: Sat Feb 08, 2014 10:39 pm

Re: Too Many Connectors

Post by admin »

Hi Noqras,

Thanks for reporting! :-) I don't suppose you could share a version of your broken workflow? It would be much easier to investigate ;-)

-steinar
emcodem
Posts: 1631
Joined: Wed Sep 19, 2018 8:11 am

Re: Too Many Connectors

Post by emcodem »

Aye,
i spent some time checking what you do with all the nodes and i am under the impression it would be better for you to get in touch with variables. This way you might reduce your workflow to 1-3 encoding nodes only.

For example, to get started, lets reduce all the "is there no audio" nodes and also reduce half of the encoding nodes:
1) take a deep breath, get a coffee, backup your existing workflow
2) insert one "is ther no audio" conditional in the main branch (before fps check)
3) insert 2 "populate" nodes in the main branch after the before inserted "is there no audio" check
4) in the success path (there is really NO audio), set 2 user variables:

Code: Select all

%s_lavfi% =  -f lavfi -i anullsrc=r=48000 -shortest 
%s_ac% = -ac 2
after3.png
after3.png (7.18 KiB) Viewed 9682 times


5) in the error path's populate parocessor, set the s_lavfi and s_ac variables to "nothing"
6) both populate connector's outputs are connected to the "there is 1 or less audios processor
before1.png
before1.png (10.84 KiB) Viewed 9688 times
after1.png
after1.png (12.35 KiB) Viewed 9688 times

6) delete all ffmpeg nodes for the old "no audio" path and all "is there no audio" conditionaly before the encoding nodes
7) alter the old "there is audio" ffmpeg command from this

Code: Select all

%ComSpec% /c ""%s_ffmpegx64%" -i "%s_source%" -c:v dnxhd -vf "scale=1920x1080:in_range=tv:out_range=tv,fps=30000/1001,format=yuv422p" -vsync 1 -b:v 145M -c:a pcm_s24le -ac 2 -ar 48000 "M:\DPShare\Aurora\DNxHD Profile\output\QC Pass\%s_original_name%.mxf""
to this:

Code: Select all

%ComSpec% /c ""%s_ffmpegx64%" -i "%s_source%" %s_lavfi% -c:v dnxhd -vf "scale=1920x1080:in_range=tv:out_range=tv,fps=30000/1001,format=yuv422p" -vsync 1 -b:v 145M -c:a pcm_s24le %s_ac%  -ar 48000 "M:\DPShare\Aurora\DNxHD Profile\output\QC Pass\%s_original_name%.mxf""
You see? ...all i did was inserting the variables %s_lavfi% and %s_ac% into the existing "there is audio" ffmpeg command. If the variables are set, audio will be created, if not, we go with the contained audio..
before2.png
before2.png (25.1 KiB) Viewed 9688 times
...i am pretty sure we could delete of most of encoding nodes by using further variables for fps, resolution and encoding settings (bitrate)

Also, it would be good if you insert a populate processor at start to set the output path, instead of writing the output path into each and every ffmpeg line:
%s_output_path% = M:\DPShare\Aurora\DNxHD Profile\output\QC Pass\
This way, if you need to change the path, you only need to change it at ONE processor, not at 50...
emcodem, wrapping since 2009 you got the rhyme?
admin
Site Admin
Posts: 1658
Joined: Sat Feb 08, 2014 10:39 pm

Re: Too Many Connectors

Post by admin »

Noqras, I found the cause of the first bug and it was a nasty one! :oops: It's fixed and I'm testing now. The second bug is probably related but I have not been able to reproduce that one. I will send you a patch for you to test when I'm done.

Thanks again for reporting! :-)

-steinar
Noqras
Posts: 40
Joined: Mon Dec 16, 2019 4:32 am

Re: Too Many Connectors

Post by Noqras »

Emcodem!

Thank-you so much for your help on this, yesterday I was thinking as this was happening that first of all there is probably a better way to do with involving variables and second of all I really just need to sit down and work that out.

I have removed 1/3 of the encode processors by leaving a lanczos flag on all processors thus being able to remove the redundant processors for 4k and 2k, then using variables for audio that you laid out will make this much more efficient.
FFAStrans 1.2.0
HP DL360p Gen8 - 2x Intel Xeon E5-2670 - 32GB RAM
2x HP SL230s
Windows Server 2008 R2 SP1 64-bit
emcodem
Posts: 1631
Joined: Wed Sep 19, 2018 8:11 am

Re: Too Many Connectors

Post by emcodem »

Oh you will be so much more mighty when you work with variables, i am looking forward to it. Let me know all your doubts and experiences.
If you have troubles, please spend some time on the workflow example thread, maybe it is of help: viewtopic.php?t=849

And also please don't forget to thank @admin for locating the bug.. he works his ass of for all of us!
emcodem, wrapping since 2009 you got the rhyme?
Noqras
Posts: 40
Joined: Mon Dec 16, 2019 4:32 am

Re: Too Many Connectors

Post by Noqras »

Steinar,

I tested your patch and it works perfectly! I don't get the first error anymore and it looks like the second error was more or less an extension of the first error as it would only ever happen after the first. Thank-you so much!

I am reducing my workflow down quite a bit now, but we now know that workflows have the potential of being huge without any issues now which is great.

Thanks!
FFAStrans 1.2.0
HP DL360p Gen8 - 2x Intel Xeon E5-2670 - 32GB RAM
2x HP SL230s
Windows Server 2008 R2 SP1 64-bit
admin
Site Admin
Posts: 1658
Joined: Sat Feb 08, 2014 10:39 pm

Re: Too Many Connectors

Post by admin »

That's great news, Noqras :-)

Again, thanks for reporting the bug!

-steinar
Pogle
Posts: 12
Joined: Fri Mar 27, 2020 10:03 am

Re: Too Many Connectors

Post by Pogle »

Hello!
I also have a workflow with (too) many nodes. Now I cant delete nodes anymore and get the error message
Line 58504... Array variable has incorrect number of subscripts or subscript dimension range exceeded. I have no backup version with fewer nodes ):
Could you help me repairing my existing workflow? I deleted most of the connectors, but still get the error message when trying to delete nodes.
Could I send you my broken workflow or would this mentioned patch help me?

Thank you!
Post Reply