Issue with AVC Files and missing audio data

D.Anne wrote on 11/2/2014, 10:32 PM

I recorded 30 minutes of video at 23.97 fps, 1080p. On the SD Card, this appears as 3 MTS files. (00000.MTS. 00001.MTS, and 00002.MTS).

Using Device Explorer, there is only 1 clip to import, and it appears to be fine.

If I import the 3 MTS files separately, the video frames are all correct, and match 1 to 1 with the file from Device Explorer window. Problem is that the final 0.48 to 0.49 seconds of audio is missing from the first two files. The audio data matches exactly with the audio data from the device explorer file except for the missing pieces.

I just tried both Catalyst Browse and Catalyst Prepare. The clips created by them have the same missing data issue. Using Catalyst Prepare and a Storyboard export has the same missing data issue. When viewing with Catalyst Prepare, there is a noticeable audio gap as play proceeds from one clip to the next (storyboard view).

Camera used for test was Sony CX760V. I believe the same issue happens for all AVC based cameras though. Vegas Pro 13.

Any idea what is happening here? Clearly the data is not missing since device explorer gets it right.

Comments

Kimberly wrote on 11/3/2014, 12:02 AM

Did you record the entire 30 minutes without pressing the Stop button? If yes, you are bumping into a file size/length constraint that is particular to AVCHD. I also have a cx760v and I recall seeing this on the owner's manual. I think it's 10 minutes but cannot recall for sure.

The solution is to concatenate the files together using Sony Play Memories Home or similar.

Regards,

Kimberly

OldSmoke wrote on 11/3/2014, 7:35 AM

Using Device Explorer, there is only 1 clip to import, and it appears to be fine.

Device Explorer concatenates the split files and is the correct way to import those into Vegas.
The main reasons the camera splits a long recording into several files is the files size limitation of either the camera itself or the SD card used. SDHC cards are formatted as FAT32 which has a files size limit of 2GB. If your camera can accept SDXC cards then you might be able to get around those split files completely. SDXC cards are exFAT32 which has no files size limit. You could also try and re-format the SDHC card as exFAT32 and see if it works. If all that doesn't help, then your camera doesn't write bigger files then 2GB.

musicvid10 wrote on 11/3/2014, 7:53 AM

Use the software that came with your camera, or Device Explorer.
Don't use Windows Explorer.

PeterDuke wrote on 11/3/2014, 5:17 PM

Sony supplies software called Play Memories Home (formerly called Picture Motion Browser) which is primarily a database program, but it can perform other functions too. If you use it to transfer your files from your camera to the computer via USB, it will concatenate the files making up a single scene into a single file and rename all files to shooting date and time.

You can also concatenate the files using tsMuxer, VideoReDo, the COPY /B command, etc. but the one thing you shouldn't do is use the scene fragments individually if you want to avoid the lost data at the joins. The audio is the worst affected, but the the video is affected as well.

OldSmoke wrote on 11/3/2014, 6:02 PM

I just don't know why one would recommend an external software when VP does it perfectly.

D.Anne wrote on 11/3/2014, 9:48 PM

Both PlayMemoriesHome and DeviceExplorer work perfectly well and do concatenate the 2GB segmented clips just fine.

I was trying to use a new workflow based on Catalyst Prepare/Browse and cannot get either of them to recognize these are continuous clips. Both Prepare and Browse treat them individually and there is a significant loss of audio data at each clip junction point.

Was hoping someone had an explanation of why this is happening. Obviously the data IS IN THE FILES, otherwise PMH and DeviceExplorer would be failing too!

PeterDuke wrote on 11/4/2014, 1:37 AM

"I just don't know why one would recommend an external software when VP does it perfectly"

Some reasons for me:

I have a tidy mind and I like one scene to occupy one file. At a later date I will never remember what scenes are fragmented.

If you wish to view the scene in something other than Vegas, you are in trouble.

The transfer software supplied by Sony and Panasonic also renames the clips to shooting date and time. This is sufficient reason in itself for me. Unfortunately, the Panasonic software doesn't concatenate the clips as well, but the file names now make it obvious that a scene has been fragmented. Fortunately, Panasonic uses 4GB chunks vs. Sony's 2GB, so it is less of an issue.

PeterDuke wrote on 11/4/2014, 1:52 AM

"Was hoping someone had an explanation of why this is happening."

Because of the file system used in cameras, a file is limited to 2 GB if the software uses signed 32 bit integers, or 4 GB if unsigned 32 bit integers are used. If the scene would be longer than this, the file is simply cut near each 2 GB or 4 GB point more or less regardless of the data and the parts are stored in separate files.

Both compressed video and audio use prediction to get the best compression, so to uncompress you may need to know what has been and what is yet to come. If this info is in the other file part, prediction cannot be carried out, hence the gap. If you chopped at I-frames, the video at the start of a file part would be OK but the audio wouldn't.

OldSmoke wrote on 11/4/2014, 9:06 AM

I have a tidy mind and I like one scene to occupy one file. At a later date I will never remember what scenes are fragmented.

Sorry Peter but I don't understand your reasoning here. Device Explorer does import the truncated files already as a single file hence there is no need for any additional software.

D.Anne wrote on 11/4/2014, 3:05 PM

Re: "Was hoping someone had an explanation" I was referring to why Vegas Pro does not import all the audio data from the file. I do understand why the clip is broken into 2GB segments.

The audio data IS IN THE FILE. otherwise PlayMemoryHome and DeviceExplorer could not get the data when it joins the files together.

But when Vegas imports each individual file segment there is a noticeable audio data loss at the end of each segment. I measure between 0.48 and 0.49 seconds of lost audio.

OldSmoke wrote on 11/4/2014, 3:25 PM

Vegas Pro does "import" it correctly via Device Explorer. The information is spread over two or more files by the camera and MUST first be written into one new file containing all the truncated information. There is no NLE where you can drop the separate files on the timeline and expect them to play without out a gap.

Rob Franks wrote on 11/4/2014, 3:27 PM

" I was referring to why Vegas Pro does not import all the audio data from the file. I do understand why the clip is broken into 2GB segments."

Actually if you look hard enough you will find a few frames missing too.
Peter explained it to you correctly

When you hit the stop butoon on the cam that last clip is closed (or "finalized") but when the camera divides the long segments to fit the formatting it simply chops the video. It does not finalized the end, so a GOP gets chopped as a result. (A gop is a system which contains the combination to properly uncompress that particular segment of the video. You now have two separate clips with an incomplete gop at the end of one and an incomplete gop at beginning of another. Vegas can't read the incomplete gop and ends up discarding a frame or two as a result.... and you end up with gaps.

You need to connect these long segments back together to re complete before importing to Vegas otherwise they are looked at as separate clips with unreadable gops

OldSmoke wrote on 11/4/2014, 3:29 PM

Ah! Much better then my explanation.

D.Anne wrote on 11/4/2014, 9:03 PM

Okay. I am not getting the answers I am looking for.

I have used the AVCHD camera for a more than a year now, and yes, Device Explorer gets it right.

To answer Peter, I do look at the video and I have not seen any dropped frames from this camera when Device Explorer or PlayMemoryHome brings in spanned clips. And yes I have looked carefully. at more than one clip. The video frames brought in separately, and from DeviceExplorer, match up one to one in every case that I have examined. I have not seen the video loss that Peter is talking about (yet).

Anyway, I don't understand how the video Codec and long GOP has anything to do with the audio data which is encoded separately in a separate stream. My initial question remains unanswered and is only concerned with the audio data.

My point to this thread was about the audio data loss, because I would like to use Catalyst Prepare/Browse for initial ingest, but it seems to not be possible. I have started this thread in that forum, but very few people are reading that forum so far.

I will wait for someone to answer this in that forum.

PeterDuke wrote on 11/5/2014, 8:25 AM

If you import a split video correctly into Vegas (either concatenate it before hand with Play Memories Home, tsMuxer, Copy /B, etc. or use device explorer) the video and the audio will have no breaks.

If instead you insert the parts directly into the timeline by dragging or double clicking them from the normal explorer window in Vegas, or dragging from Windows Explorer, and place them below the properly imported version, you will notice that the parts total is slightly shorter than it should be. With my limited testing of Sony and Panasonic PAL AVCHD, there is always two frames of video missing at the join(s). In addition, the audio is silent for a short period (several frames long, from memory) at the end of each part except the final one. That is what you are talking about, I think.

The principle is the same with both video and audio. Predictive coding is being used to efficiently compress the signal (audio or video), but when the complete file is arbitrarily split, some of the information needed to fully decode say the end of one part may be located in the start of the next part and vice versa. However when you drag the parts to the timeline, each part is decoded by Vegas independently of the others, so the info in the adjacent part is not read. That is the behaviour you would expect with normal independent clips.

The reason why the audio gap is longer than the video gap must have something to do with the way each is encoded, and is beyond my knowledge.