Best Archival Format?

xstr8guy@sbcglobal.net wrote on 11/23/2008, 11:43 AM
We currently produce content for web use. But we have near-future plans to release our content on DVD so I need to re-encode all of our old videos. These will be regular resolution DVDs not Hi-Def although most our source material was shot in HDV.

But while I'm rendering old videos for DVD, I'd like to make copies in the most versatile, future-proof format available. And then I'd like to store those copies (along with the original source material) in a safe-deposit box to prevent any unforeseen tragedy.

Any suggestions?


Thanks!

Comments

johnmeyer wrote on 11/23/2008, 12:45 PM
Since it is hi-def, you want to render in an HD format; otherwise it does not meet your requirement of "future proof" (since in the future, SD will eventually be dead).

So, what I've been doing is rendering out to m2t, using the supplied m2t template in Vegas, and printing back to tape. There is some resolution loss, although if you have Vegas 8.x, I am pretty sure that the cuts-only portions of your footage will be saved without re-rendering. Put these m2t files back onto DV/HDV tape. Some will no doubt argue about the wisdom of using tape as a storage format, with both pro comments (tape has proven to last a long time) and con (how long will HDV equipment be around to read these).

You could also store these on BD discs (as a data file, not as actual video content). This would give you about two hours of m2t per disc. Since you are storing in a safe deposit box, I don't know how the storage volume/gigabyte for DV/HDV tape compares to BD, but it is probably pretty close.

I would also archive these files onto some cheap hard disk drives so you have quick access for future edits.

I guess you could also render to WMV, but that format take forever to render, and I don't know how it compares if you want to re-use the stuff later on.

farss wrote on 11/23/2008, 1:37 PM
If you've got the funds LTO3/4 data tape is pretty much the industry standard for archiving. You just backup everything as data and you get a lot of error correction.
The whole issue of future proof archival storage is a huge issue for this industry with millions of words written about it. I can't really say if LTO is any better than anything however given the mass of archival material both from video, film and audio being commited to it it's a fair bet there'll be people with the kit to read it for a long time to come.

Bob.
xstr8guy@sbcglobal.net wrote on 11/23/2008, 2:04 PM
Thanks for the replies so far!

I have lots of local hard disc storage so I'm mainly concerned about storing BD discs offsite.

I've learned a hard lesson. I recently made an awful mistake. I bought a new RAID storage tower but in my efforts to set it up, I accidentally formatted another drive that held some of my archived files. Hopefully I can recover all of the old files. Pray for me. Lol.

How about uncompressed AVI? Or would the file size be way too massive?

Of course, I'll be saving the original video files and Vegas files along with a rendered version too. But I don't imagine that I will always be using Vegas as my NLE years in the future. So who knows if I'll be able to open .veg files 5 years from now.
TheHappyFriar wrote on 11/23/2008, 3:03 PM
print back to tape via HDV. all set. :)
xstr8guy@sbcglobal.net wrote on 11/23/2008, 3:54 PM
I'm editing .m2t files. I have no access to tape... I'm just the editor. ;)
farss wrote on 11/23/2008, 4:07 PM
Uncompressed HD does produce MASSIVE file sizes. In general you should use a lossless or less lossy codec than your acquisition codec. I quite like the Sony YUV codec. You only need the 8bit variant for most things, other apps inc FCP can read it and it does multichannel audio.
The advantage of that codec is its 4:2:2 chroma sampling. Your graphics will hold up pretty well thanks to that. Anything acquired as NTSC DV will be going from 4:1:1 to 4:2:2 so that'll hold up very nicely. HDV assets will also not suffer in the process. On the other hand using all but the highest bitrate mpeg-2 encoding you could be introducing losses in several directions plus at the higher bitrates mpeg-2 isn't going to save that much space anyway.

Bob.
TheHappyFriar wrote on 11/23/2008, 4:51 PM
I'm editing .m2t files. I have no access to tape... I'm just the editor. ;)

you wanted the MOST future proof & there's a TON of HDV camera's + deck's out there. :D IE I can still get 8mm tape cameras to play/capture from!

any particular reason to not to render to a normal m2t file, put on a drive & then when that drive is full pull it out & put a sticky label on it?
xstr8guy@sbcglobal.net wrote on 11/23/2008, 5:23 PM
I've tried rendering to a few different file types. The problem arises when I try to import those files back into Vegas. BluRay .m2v files and Sony YUV .avc files crash or refuse to open in 8.1 and 8.0c. Uncompressed AVIs are huge but import into Vegas although they bog it down when previewing. The only effective high-quality file type that I'm able to import back into Vegas is HDV .m2t. But am I going to lose a significant amount of quality if I try to render a BluRay version from .m2t?
xstr8guy@sbcglobal.net wrote on 11/23/2008, 5:30 PM
any particular reason to not to render to a normal m2t file, put on a drive & then when that drive is full pull it out & put a sticky label on it?



I am doing that too. In addition to archiving on drives, I'd like to make further backup copies to keep offsite.

And these high quality renders could also be used in the future when we want to release our titles on BluRay. Currently our videos are only encoded for internet use. We do have immediate plans to release standard-def DVDs so I need to load all of the old files and re-render them (and tweak the edits a bit). And while I'm at it, I might as well render to a high-def format for future release.
Spot|DSE wrote on 11/23/2008, 5:40 PM
For a multi-repurposable future format, I'd consider 4:2:2 XDCAM/.mxf format. It's a standard, it'll be readable for a long time to come, high quality vid, small relative file sizes, very fast to render to from HDV, XDCAM, and even AVCHD file formats. For output to BD, very fast render. For output to web, reasonably fast render. Uncompressed HD is huge....you'll spend a lot more on storage space than necessary, IMO
farss wrote on 11/23/2008, 6:12 PM
"BluRay .m2v files and Sony YUV .avc files crash or refuse to open in 8.1 and 8.0c."

If you cannot open Sony YUV files reliably on your system there's something remiss. I haven't tried them in 8.0c, for good reason I'm sticking with 8.0b as so far I've no compelling reason to upgrade to 8.0c. If there's a problem with the Sony YUV files in 8.0c someone really needs to report this. That codec has worked for years, it is pretty much the BMD codec.
Bob.
xstr8guy@sbcglobal.net wrote on 11/23/2008, 6:26 PM
For a multi-repurposable future format, I'd consider 4:2:2 XDCAM/.mxf format.


Under what "save as type" is that option?



If you cannot open Sony YUV files reliably on your system there's something remiss.


I must be missing some codecs. I assume that when you install Vegas, it also installs all supported codecs or at least checks for them before making them an option in the menu. And if I can render those formats, I must have the codecs, no? Is this a Vista 64bit problem maybe? I also can not render H.264 on one of my Vista 64bit machines but can on another one. Strange.

Also YUV and .m2v seemed to mess up my colors and/or brightness.
xstr8guy@sbcglobal.net wrote on 12/5/2008, 2:01 AM
I am trying the .mxf format and I'm pretty happy with the results except for the output is has a bit more contrast than my original files.

As for Main Concept MPEG2 (m2t), I'm a little confused by the plethora of settings,,, especially under the advanced tab such as DC Coeffcient (8, 9 or 10 bit), Color Primaries, Transfer, Matrix Coefficients, etc. Does anyone know what the heck to do with these settings?.

I have another question. Since most of my source video is PAL 50i, should I maintain the PAL interlacing throught the process? Should I stay at 50 or go to 25 fps interlaced? For example, the preset settings for 'HD 422 1920x1080-50i 50Mbps' ,mxf has a framerate of 25 but you can select 50 fps in the dropdown menu. Too many options...UGH!
kairosmatt wrote on 12/5/2008, 7:21 AM
I think that you should maintain your exact framerate and interlacing. I have found that rendering to different framerates can and does effect the quality, though I haven't used PAL (but I'm guessing it would be the same).

50i is the same as 25 fps interlaced. I also find this confusing, in NTSC land they say 60i which means 29.97 interlaced frames.
If you select 50 from the pulldown does it change it to progressive?

1-Does any archive using cineform? I have been using Raylight (which is similar).

2- Does Sony YUV change the color space, and if so, does this effect the quality?

kairosmatt
BrianStanding wrote on 12/5/2008, 7:56 AM
I render a digital master out to Cineform Intermediate.

I've also been converting all my timelines to Cineform Intermediate. (To date, I've been using GearShift from VASST to do this from rough-cut m2t timelines, but I may spring for the full version of Cineform Neo HDV, and just do all my work in Cineform.) Then, use Vegas' "Save As | Save Trimmed Media" function to get just the trimmed files to save space. If you have a lot of nested Veggies, VASST's Ultimate S is a good investment, since it has a feature that will automatically trim and save all the media files inside your nested vegs.

I couldn't afford a new LTO-3 or 4 deck, but I found a reconditioned LTO Ultrium 2 tape drive and a SCSI card for $500. So now I save all the trimmed intermediate files, the VEG files and the digital master to LTO tape, using the free NTBACKUP. THE LTO-2 tapes are 200gb native capacity and cost about $30 apiece. The LTO tape goes into the safe deposit box, and meanwhile, I keep all the camera-original HDV tapes for at least some kind of redundancy. (Too bad Vegas doesn't recapture offline HDV media like it does for DV media.)

If I just need to render out into a different format, I can restore the digital master. If I need to re-edit, I can restore all the veggies and their associated trimmed files.

Working so far. Ask me again in 30 years.
johnmeyer wrote on 12/5/2008, 9:05 AM
Absolutely, positively maintain the interlacing. Otherwise you will degrade the footage. No question on this one. Even if you eventually want to deinterlace for some reason, do it then, and NOT now when you archive. When you deinterlace, you ALWAYS lose actual information that you will NEVER recover. Therefore, this is why I say that it is always the wrong thing to do if you are archiving footage.

Also, just use the standard presets when creating m2t files. Don't change ANYTHING. If you have Vegas 8.x, it should simply copy -- without any re-rendering -- footage which hasn't been altered from your original (i.e., cuts-only edits). Thus, if you then print this back to tape, you will get precisely, to the pixel, exactly the same thing as was on your original HDV tape.

xstr8guy@sbcglobal.net wrote on 12/7/2008, 7:35 AM
Mr. johnmeyer you are indeed a Vegas guru. And your advice on keeping the interlaced video while archiving is great information. But I'm not so sure my source video is actually interlaced. I've started a new thread explaining my predicament here...
http://www.sonycreativesoftware.com/forums/ShowMessage.asp?MessageID=628101

I am so confused. The more I learn, the less I seem to know. :(