Rc Buffer Overflow?

[knk]

Hey guys,

If anyone can help:

I was deploying a new WF for older IMX media that has very large clips, hours long.
When I'm rewrapping using the IMX encoder to make some tweaks I'm getting an error "RC buffer overflow".
WF works fine if clips are up to 2h long but it "blows" beyond it...

Our good new friend GPT helped me with a custom FFMpeg with some "qmax" and "qmin" values that seems to do the trick, but I was wondering if there is anything I can to to prevent this error and keep using the FFAS default encoder Node for IMX, which is much more "friendly" in the long run whenever I'll need to tweak or readapt this WF in the future.

All the best!

[FranceBB]

Are you actually just remuxing the source or re-encoding it?
Are you using the default FFMpeg mxf muxer or the BBC BMX one?
Which version of FFAStrans are you using?

You mentioned -qmax and -qmin which are used when the source is re-encoded.
In the IMX encoder we're currently using

Code: Select all

-qmin 1 -lmin "1*QP2LAMBDA" -qmax 8 -lmax "8*QP2LAMBDA"

The bitrate is set to

Code: Select all

-b:v ' & $tpl.video.bitrate & ' -minrate ' & $tpl.video.bitrate & ' -maxrate ' & $tpl.video.bitrate 

where, for IMX50, this is 50 Mbit/s. Then, given that it's all intra, we specify

Code: Select all

-g 0 

and we have

Code: Select all

-bufsize ' & Int($tpl.video.bitrate/$tpl.video.fps)+1 & ' -rc_init_occupancy ' & Int($tpl.video.bitrate/$tpl.video.fps)+1

This all looks absolutely fine to me.
The rule of thumb is to have 50 Mbit/s with the bitrate / fps for all intra formats.
In this case it would be 50000 / 25 = 2000 kbit/s for the buffersize.
In the past, like few versions ago, we were using

Code: Select all

-qmax 10 -lmax "10*QP2LAMBDA" 

but that changed later on, although I really don't think it affected this.

Tomorrow I'll try to find a long feed to trigger an IMX50 encode and see how it behaves on BitrateViewer, but in theory I don't see how it could overshoot. A good encoder, when put in front of those constraints, should, in theory, undershoot and zero-fill, namely it should always raise the quantization pass if it can't achieve the target bitrate so that it uses less bitrate and then add zeroes to get to 50 Mbit/s. To be fair, the open source MPEG-2 encoder we're using has been pretty unreliable in the past in terms of bitrate management, so much so that in other MPEG-2 encoders *coff coff XDCAM-50 coff coff* we had to go as high as -qmin 5 -qmax 28, but I really hope we won't have to do the same for the IMX50 one. We'll see.

Out of curiosity, what are the -qmin and -qmax values you used to get the bitstream to be correct?

[knk]

Hey @FranceBB thanks a lot for the reply!

I'm trimming IMX30 (4:3 PAL) content in very large clips, so there seems to be a need for A/V decoding before the custom AV Synth that sets in/out points from the web browser video player.

The webUI is amazing to get give me the possibility to rename media and redirect it anywhere from the custom forms, while providing the video and thumbnail preview for the content, so this would be my prefered entry point for clips.

Our good frient GPT advised me to use the maximum allowed q of 28. It works, I mean the final clip is correctly generated, but the colorspace is being messed with, colors are clearly affected with the code below...

Here's the custom command I'm using in FFmpeg node, which I got from the IMX encoder node running a file without segmentation, and with the added "tweak" for the qmax:

"-r 25 -pix_fmt yuv422p -non_linear_quant 1 -intra_vlc 1 -ps 1 -dc 10 -qmin 1 -lmin "1*QP2LAMBDA" -qmax 28 -lmax "8*QP2LAMBDA" -rc_min_vbv_use 1 -rc_max_vbv_use 1 -g 0 -b:v 30000000 -minrate 30000000 -maxrate 30000000 -bufsize 2400002 -rc_init_occupancy 2400002 -profile:v 0 -level:v 5 -chroma_sample_location topleft -signal_standard bt601 -aspect 4:3 -color_primaries bt470bg -color_trc smpte170m -colorspace bt470bg -color_range tv -field_order tt -top 1 -flags +ildct+ilme+low_delay"

[ngocnhungplaza]

I’ve run into a similar issue before with long IMX clips. From what I experienced, the default encoder node in FFAStrans struggles with bitrate control on very long inputs—especially if there's not enough buffer allocation internally. One workaround I tried was splitting the input into smaller segments, running the encode, then stitching them back. Not ideal, but it avoided the overflow. Curious to know if you’ve experimented with any buffer or GOP settings inside the node itself?

[emcodem]

@ngocnhungplaza
how nice from you to share these experiences, thanks a bunch.

Mister @FranceBB is still working on thisj. My guess is that the bitrate control must be changed to the same as it works in the XDCAMHD encoder, e.g. different qmin qmax presets for different picture content.

[FranceBB]

Our good frient GPT advised me to use the maximum allowed q of 28

Good friend? Maybe yours, not mine XD
Jokes aside, I raised -qmax to 28 and changed -lmax to "28*QP2LAMBDA" and then I recompiled a new processors.a3x for FFAStrans 1.4.1 Stable.

processors.a3x

(889.69 KiB) Downloaded 177 times

Copy-paste it in B:\Processors\ in the FFAStrans installation folder

Capture.PNG (11.46 KiB) Viewed 6999 times

Then trigger the encode again and let me know how it goes.
I wanna know whether the issue goes away and most importantly whether the quality stays "acceptable" through the content or if there are some scenes in which the quantizer went crazy and totally destroyed them.

My guess is that the bitrate control must be changed to the same as it works in the XDCAMHD encoder, e.g. different qmin qmax presets for different picture content.

Your guess is right and that would be the right approach, however that would potentially come in 1.4.2 if we make that change 'cause it would involve changing the gui.au3 and proc.au3. For now I've just recompiled the processors.a3x to allow an higher quantization pass by default.