Transcoding All Home Video to H.264/MP4

VideoP wrote on 10/6/2011, 4:53 PM
I've been capturing family video for the past 15 years. Over the course of that time I've been very good about organizing and editing that video and have it at the point now where I've got around 200 individual video files ranging from 30 seconds to 20 minutes each. The video was encoded and stored in whatever codec/container made sense to me at the time i.e. HDV/AVI, MPEG1/MPG, AVCHD/M2TS, MPEG2/M2TS, MPEG2/MTS. The resolutions include 720x480, 720x480 Widescreen, 1440x480, 1280x720 and 1980x1080 (both 30fps and 60fps). The early files were created using early versions of Premiere but most everything else was created with Vegas 3 through Vegas Pro 10. Most of these files play flawlessly on my Windows 7 based machine but suffer from compatibility problems on my MAC, AppleTV, PS3 and XBOX 360. I have been able to transcode to H.264/MP4 and create compatible files that play fine on all platforms but am facing a dilemma, which transcode workflow is best for my purposes. While I want to retain quality if I can I'm more concerned about future compatibility. I have found in my experiments that not all transcodes are equal and wanted to list some of the experiences/thoughts I've had to see what others think. I know there is a large thread here concerning rendering for YouTube but since I'm not as concerned about bitrate and 1280x720 I'm not sure if it is apple to apples.

Some things that concern me are how some transcoders by default want to change the pixel aspect ratio and thus change the resolution. For example a 1440x1080 file will get written as 1905x1072 by Handbrake.

I'm also not sure if I should try and make my files progressive if they were originally interlaced. I'm finding that by default the fps come out to 59.94 when they were originally 29.97 if I leave things as interlaced when rendering using MainConcept. I've already run into problems where iTunes does not like the higher fps.

I thought I had found the perfect solution in Handbrake but for whatever reason I am getting strange playback results in Windows Media Player and QT on my MAC. If I jump around within the video using the progress bar there is a distinct pause / catch up as play resumes. Sony AVC and MainConcept do not seem to have the same issue.

I had thought the ideal solution was to use Apple Compressor but it has serious compatibility problems with AVCHD along with pretty much every other HD format I have. I've tried demuxing, switching containers, etc.. etc.. but have not found the perfect workflow. Clipwrap also does not work as I often lose sound or the first few seconds of the converted video (the developer told me they don't support Vegas exported files, just fresh from the camera).

If you have any advice on this matter I'd love to hear about it. I'm considering settling on Sony AVC since I can transcode everything using a single codec so would like to hear if anyone thinks that is a big mistake from a future proof standpoint. I've already been burned by some bad AVCHD renders via Vegas where audio was out of synch so that worries me.

Thanks !

Comments

johnmeyer wrote on 10/6/2011, 5:16 PM
This sounds like a LOT of work, and probably for no reason.

1. There is no reason to transcode a file until and unless you truly cannot deal with it in its native format. You will almost certainly be able to play it and watch it, pretty much forever, as long as the media on which it resides is intact.

2. You will always degrade the quality of the video when you transcode. Any further editing, followed by rendering, that you do at a later date will degrade it even further. If you don't transcode now, you will save yourself one level of degradation.

3. Don't transcode just to save disk space. Disks are amazingly cheap.

4. Even if you do decide to go ahead with this project DO NOT deinterlace footage that is interlaced. This will degrade the quality even further and is simply not necessary.
VideoP wrote on 10/6/2011, 5:31 PM
Hi John,

Thanks for the feedback. The issue I'm trying to deal with here is making these video files wife proof. There are only two playback devices that are capable of dealing with the multitude of formats I have, a Windows 7 PC and WDTV both of which are not connected to our television. In case I ever get hit by a bus I want to make sure that playback of these files is not a huge technical undertaking. Things need to be simple ala MP3 and AAC in the music world where you just hit play and it works. And while I have noticed a hit in quality, even when going to progressive, it is so minor that even with my very picky tastes I am unable to see a huge difference when doing A / B comparisons. The other thing I forgot to mention is that these videos are final. There will be no more editing, they are already fully edited and worked up with music etc..this will be a one time transcode. I will also of course keep the originals should something better in the future come along. I've got terrabytes of backups so DASD is not the issue, just convenience.
john_dennis wrote on 10/6/2011, 6:10 PM
I agree with johnmeyer that you seem to be going to a lot of effort to ensure your legacy video is "wife proof". Many of us have similar concerns. Your time would be better spent if you:

1) Use the time you would have spent transcoding making money to pay for term life insurance on your life in case you get hit by a bus. Your wife could then pay johnmeyer to deal with the compatibility issues.

"There are only two playback devices that are capable of dealing with the multitude of formats I have, a Windows 7 PC and WDTV both of which are not connected to our television."

2) Buy your wife a Windows 7 laptop for Christmas (or your holiday preference) with an HDMI port and wireless adapter and teach her how to hook it to your TV. Share your files on your current machine using the DLNA streaming service in Windows Media Player and teach her how to navigate to your files.

This method may enrich your life rather than have you sitting in the basement transcoding files by yourself.
johnmeyer wrote on 10/6/2011, 6:22 PM
For the record, I am not qualified to deal with wife compatibility issues, although if you want to pay me to deal with them ... oh, wait, that makes me ... never mind.

OK, so the issue is that you want to be able to play all your video on devices that your wife uses. I'm sure you've looked into this, but you might want to see if you can find something, at least for the Mac, that is the equivalent to VLC (is it available for the Mac?). Put another way, I'd spend at least a little bit of time trying to solve the problem by finding a player that will play your existing files on each of the platforms in question, rather than transcoding the files to fit the players you already have on each of these computers/consoles.

VideoP wrote on 10/6/2011, 7:43 PM
I'm not sure why you have the misconception that this is such as big time waster. I've actually transcoded everything all ready with very little time and energy using Handbrake. I simply broke the files down into groupings by resolution and type and then added them to the appropriate job queue. It took maybe an hour of my time and a weekend of unsupervised processing time on a spare computer. The same is true of Compressor. I compressed 70 movies in no time by simply dragging and dropping them into a batch job. Sure I'm tinkering with one or two files trying to see the difference in results but how is this any different than all the work others have done to see what works best in YouTube. Was that a waste of time?
john_dennis wrote on 10/6/2011, 8:59 PM
Whether your efforts are a waste of time is very personal and subjective. You certainly have the right to spend your time as you please. Depending on how you frame the project, the solution could be different. If the problem is finding the killer workflow, codec, wrapper or cross-platform solution it might be one thing. If the problem is providing easy access to your video to non-video enthusiasts the solution could be quite different. You might even be able to buy it as a product. You said the WD TV was able to play all of your files but you have not chosen to hook the device to your TV. A network enabled version of this device may just be the hardware solution to making the video available to others. ~$100. There a number of other devices that play video from the network or locally attached hard drives, including most of the current Blu-ray players.

John's comment about generation loss and deinterlacing still stands.
musicvid10 wrote on 10/6/2011, 10:19 PM
For home/family use, standardizing playback formats is actually an admirable goal. For a network media server, monitor playback, or if it is bound for the web, AVC/MP4 Progressive is a good choice. It is also cross-platform with Mac-like devices.

"For example a 1440x1080 file will get written as 1905x1072 by Handbrake."
I think you need a little more experience with Handbrake. For instance, 1440x1080 will resolve to 1920x1080 if you create a preset that does not allow automatic cropping and set a macroblock size of 8. That's exactly the way it should be.

In general, keeping the size the same as the source is a good idea. So mixed sizes are still OK. The decomb in Handbrake is so good you should not have any concern about using it on your home videos, and preemptive decombing will improve monitor and browser playback as opposed to letting the software players do it for you. This is especially important for web delivery, including Youtube. I've run many tests over time that confirm this.

You need to beware of feeding Handbrake RGB source, such as DV-AVI (MPEG-2 is fine). Levels are going to suffer because of the way the encoder mishandles these sources. Better to level in Vegas Pro and use a YUV intermediate to deliver to Handbrake, as has been discussed before.

"If I jump around within the video using the progress bar there is a distinct pause / catch up as play resumes."
Keep your bitrates reasonable and don't use variable frame rate to encode! Limiting the number of High Profile options you choose will improve playback, immensely.

The batch encoder queue in Handbrake makes like-source-to-output easy. And the encoding is twice as fast as Vegas in many instances. I'll set up a batch job, have wine and dinner, and when I come back it's done. Not like you're spending hours slaving over a computer. I've got over a TB of videos done this way, and they are completely portable between the media server, PC, and web delivery, taking up roughly 1/2 to 1/3 the space of their original source counterparts. If this is a waste of time, it really hasn't seemed that way to me.
VideoP wrote on 10/6/2011, 11:06 PM
musicvid - Thanks. I'm glad someone shares my opinion about standardization.

The Handbrake preset I used was High Profile which has Detelecine and Decomb set to default. I'm certainly not an advanced Handbrake user but I believe what you are saying is that I should use Anamorphic of None with a Modulus of 8 as long as I choose a Width and Heigh that will fit the Displayed video i.e. output of 1920x1080 for the 1440x1080 and 1920x1080 source I have.

That would still leave the wide screen anamoprhic stuff I have at 720x480 which is resolving to 853x480 in both the High Profile Handbrake and Apple Compressor. I experimented with upconverting it to 1280x720 and it actually looked fine.

Do you have any idea why the x.264 stuff would be a little sluggish while jumping around on the timeline of the media player? I was thinking it had something to do with reference frames and the fact that there were fewer in the Handbrake encodes versus the Sony AVC and MainConcept. I'm completely grasping at straws on that but it just feels like it is processing a larger delta of "changes' in the scene which I believe is how the compression works.
musicvid10 wrote on 10/6/2011, 11:32 PM
720x480 widescreen resolves to 853x480. That is perfectly normal.

Why are you using High Profile? It doesn't increase quality, only compression efficiency, and at a significant detriment to both encoding times and playabilty, period. High Profile was invented to maximize streaming bandwidth efficiency, which you implied you are not concerned about.

Try the Normal preset (with CFR), or even my Normal-Dumb preset below, and see how it plays.
ref=2:bframes=2:subq=6:8x8dct=0:trellis=0:weightp=0:b-pyramid=none:deblock=-2,-1

But without knowing the rest of your settings, I absolutely can't predict how it will go for you. Unnecessarily high bitrates (or unnecessarily low RF), and VFR are huge factors in playback issues as I already mentioned. I suggest you go to 7:09 in the Youtube tutorial, set the output height to match your source, and follow the other suggestions. You can even use RF 20-22 without noticeable quality loss. You may notice how much better and faster things will go for you if you keep the process simple. Best of luck.




corug7 wrote on 10/7/2011, 2:36 PM
Never transcode video to a frame size with an odd number like 853x480. 856x480 will keep with your modulus of 8 and will look great.
Hulk wrote on 10/7/2011, 10:40 PM
I would have a look at Ripbot. I've found it to have excellent video quality, a simple yet effective interface, and it is very stable.

The downside? It is slow. But you can batch process with the ability to load/set up the batch while other videos are transcoding. You can basically just keep feeding the machine.