Corrupt Files Transcode

[Pippo]

Dear community, I'm trying to transcode corrupt files. The process starts in the status monitor and stops after a while. It starts with Reading source media properties, then Verifying media content, then Verifying media (TTL: 27996), then Indexing, please wait, then 0 / 43200, 0% @ 0 fps, speed: 0x, ETA: 00:00:00 :00 (15:30:07) and then it's gone. nothing is displayed in the status monitor anymore.

[FranceBB]

Well, nothing is gonna guarantee that the file will be correctly processed if it's actually corrupted.
For instance, some containers like .mov have the header written at the very end, which means that if the file gets truncated halfway through a transfer, there's no way of decoding it for FFAStrans (well, I mean for ffmpeg and FFVideoSource) 'cause there are just no info.
In that case, the only way would be to recreate the header from scratch but to do that you gotta know a bunch of info on the file like the resolution, the framerate etc.
It's doable, but it's a manual process that I have (unfortunately) very painfully done in the past as a last resort.

Luckily some other containers are not that bad and one of these is, for instance, mxf.
In the mxf container the header is written at the beginning so that if the file gets truncated you will still see "isTruncated" == True in Mediainfo, but you'll have all the info to be able to decode it.
In fact, using FFVideoSource() and FFAudioSource() from the A/V Decoder node will allow you to handle it correctly but of course it will stop at a certain point 'cause it's truncated.

In other words, there's no definitive answer here, but you could for now do two things:

1) Send us a complete Mediainfo of the file in Advanced Mode (possibly with "Full Parsing" ticked in the GUI) so that we know how bad the situation is
2) Try to set the A/V Decoder to Full Decode first and to No Decode then and try both ways to see if something changes.

[Pippo]

Many thanks for the answer. I'll try using the decode. Where can I find the advanced mode for sending?

The files are from Metus Ingest Software
MXF Op Atom
1080i
XDCAM HD422 50

[FranceBB]

MXF Op Atom
1080i
XDCAM HD422 50

Oh, so it's an mxf in XDCAM, nice, we have a fairly good chance of recovering what's left of it then I would say.

Where can I find the advanced mode for sending?

If you're using the Mediainfo GUI, it's in "Debug" -> "Advanced Mode".

Screenshot from 2023-11-07 13-31-55.png (26.5 KiB) Viewed 1960 times

The idea will be to use an indexer to basically index the file.
This is pretty much what the A/V Decoder does, basically it indexes a file thus creating an uncompressed audio and video stream living in RAM that is then passed to the encoder.
Even if the container has its index compromised, an indexer should, in theory, be able to reconstruct it and deliver *something*.
Please note that if the file is truncated, though, we will only be able to recover until the part in which it's truncated, while if it has other issues like missing reference frames etc we'll see some repeated frames every now and then as the indexer will try to compensate for missing references by duplicating the previous "good" frame. (this depends on the indexer, but in Avisynth generally all indexers like ffms2, the one included in FFAStrans, and lwlibav, the one I generally use externally, favor repeated frames than partially decoded ones. Some other indexers/decoders might instead decode those as grey or green).

Anyway, fingers crossed for your file, let me know how it goes with the A/V Decoder first, though!

Oh, please also note that every time we index a file, a part of emcodem dies. (just joking).

[Pippo]

Many thanks for the answer. I tried both ways with the Decoder. The Same Problem. aaaah okay, media info, for sure. my fault. I Exported to TXT Files. Corrupt File and Not Corrupt to see the difference.

[FranceBB]

Ouch.
This is bad. Assuming that both files are identical and come from the same source, it looks like the corrupted one isn't truncated at the end as I suspected, but rather it doesn't have a proper header.

You now have 3 options in front of you:

Option 1: You bear with me, trying a few things like installing Avisynth and different indexers and AVSPmod mod to see if one of them can index it, if it can, happy days, if it can't, we'll try to remux it using different remuxers, if they can't then we'll go the hard way and we'll recreate the faulty header in the corrupt file using the one from the file that is good. This is a long process but we can totally achieve the result in the end and it's useful if the content is sensitive and can't be shared.

Option 2: You send me the file privately (I can send you an Aspera Faspex invitation) or publically (you can share a link here) and I'll try to fix it for you and then I'll comment on what I did and I'll explain.

Option 3: You don't wanna mess with any of this and you just want to "get things done" regardless and you want a very easy and clicky solution, so you upload the file to https://restore.media/ click next next and you're done. It's a service in no way linked or affiliated to me or this community, but essentially it comes from the former devs of Recover_MP4 (which used to be a command line tool) and what it does is analyzing the source file, asking you a bunch of info like from which camera it comes from, whether you know the resolution, framerate etc and whether you have another file to upload from the same camera, then it will use those info to recreate the header and it will show you a low res proxy to check that everything went well. If it did and you're satisfied, it asks you to pay a fee to download the file.


which one you pick is up to you, but in case you wanna go with option 2, send me a PM.

[Pippo]

many thanks for the answer. Ok, Option 2 is not possible, cause i cant send you Material. If my bosses find out, it will be difficult. We're filming the show right now and it's ''secret''. Let's try option 1 even if it takes a long time. I know that files from Metus Ingest are encoded using FFMpeg (which I don't understand at all) or FFAstrans. This was done by a former employee. Unfortunately I can no longer reach it. Thanks so much for trying to help me. I will try it tomorrow, i have to leave the Location, its late

[Pippo]

so, if youre ready, let me know

[FranceBB]

so, if youre ready, let me know

Ok, so, first thing first we need Avisynth to be installed.
Go to this page: https://github.com/AviSynth/AviSynthPlus/releases
Download and install the latest AviSynthPlus_3.7.3_20230715_vcredist.exe build.
After that, we're gonna need indexers like LWLibav, so head to https://github.com/HomeOfAviSynthPlusEv ... s/releases and download the latest L-SMASH-Works-r1147.0.0.0.7z build.
From the zip, take the x64 version and put it in C:\Program Files (x86)\AviSynth+\plugins64+
At this point we need an IDE to check what's going on, so we need AVSPmod mod (yes, two "mod" 'cause it was started by a guy, then picked up by another guy and then picked up by another guy who is the current maintainer): https://github.com/gispos/AvsPmod/releases
From that page, download the latest AvsPmod_v2.7.5.5_.Windows_x86-64.zip and extract it somewhere like on a folder on the desktop or whatever (it's portable).

Now we get to business: open AVSPmod mod and create your first script.
It will be as simple as:

LWLibavVideoSource("normal_file.mxf")

then click on the bottom left black arrow to "compile" and wait until the index is over (it should show you a preview).
If it works, then we're good to go and you can try with the faulty mxf and pray.
According to the result, we'll decide where to go from there.
Please note that you need the full path there, so like LWLibavVideoSource("D:\whatever\file.mxf")

p.s out of curiosity does "pippo" stand for "Filippo"?

[Pippo]

Thanks a lot, but sorry if im too stupid

If i try -> LWLibavVideoSource("normal_file.mxf") that comes -> Script error: There is no function named 'LWLibavvideoSource'.
If it ry -> ("normal_file.mxf") that comes -> Not a Clip

I have to open my Corrupt File there?


PS: Stand for Philippe, just a Nickname