OT: Backup your backups.

farss wrote on 2/1/2013, 3:00 AM

I've had a Thecus 5200 NAS running just fine for 5 years now with 5 Samsung 200GB drives in it in RAID 5. What could possibly go wrong?

The obvious single point of failiure, something in the NAS box itself.
Now it just sits there saying "Self Testing...". :)

It's a bit frustrating knowing that the data is still perfectly intact on the 5 disks with redundancy as well. I've found a number of ways to recover it, none exactly easy and even if I cannot, it will not be the end of the world as I know it.
What is really furstrating is how I've fallen for one of the greatest dangers in engineering, creep. Over the years I've come to put more and more faith in this box. The only error it's ever reported is a single S.M.A.R.T error from one drive years ago. Despite its advancing years I'd sort of assumed it'd last for ever or at least I'd replace it once one of the disks died, hah, very bad thinking.

If I can't recover the data the most annoying loss , the data that will be the most frustrating to rebuild is my WhereIsIt database of over 20 HDDs sitting in a cupboard.

There's another dimension to this. One of the reaons I don't have any complete backups of the contents of this NAS is simple. When I first put it together the 1.5TB of storage it had was bigger than the biggest single disk one could buy. I'd had dreams of buying an LTO tape drive but that was all getting seriously expensive and I've done that tape dance before but that was on someones else's dime.
So here again today I face the same challenge. Run out and buy an 8 bay Thecus or QNAP box, fill it with 1 or 2TB drives and then back it up to....???
It has occured to me that instead of buying one huge box with RAID 5 or 6 or 10 it may make more sense to just buy a couple of small 2 bay NAS boxes and use RAID 1. So long as the drives are formatted with NTFS if one fails, no drama, I can always access the other one on any Windows PC easily. Not a very elegant solution and a big waste of drives technically but simple and no single point of failure. Even if one burns to the ground taking both disks with it by splitting vital data over different units in different rooms the risks are mitigated.

Bob.

Comments

Grazie wrote on 2/1/2013, 3:30 AM

You wearing that Lycra again?? - I told you that colour just wasn't you...

G

farss wrote on 2/1/2013, 6:22 AM

I've ordered it in black, makes me look slimmer :)

Bob.

Grazie wrote on 2/1/2013, 6:28 AM

You think huh?

Chienworks wrote on 2/1/2013, 7:05 AM

That's why we always have redundant backups. The drives themselves are often robust enough, but the supporting hardware goes bad with frightening frequency. Back when we were running a cycle of 6 replicated DB servers usually at any given time or or two of them were down for repairs.

malowz wrote on 2/1/2013, 9:29 AM

im about to start "just now" on the tapeless world. already got 2 external storages, for saving the videos in 3 places. also, the 2 storages i put them in different places (my house and business partner house). my house can burn down, no valuable data loss ;P

also, will get the FMU128 to record in 2 places on-camera, so, more safety...

i really like my problem-less sleep ;P

riredale wrote on 2/1/2013, 11:06 AM

Sorry to hear about your issue.

Having a clone would be a good idea, as you said, so there was no single failure point.

Taking your disks out and sending the box for repairs can be done, yes? Or are these things throwaway?

farss wrote on 2/1/2013, 1:38 PM

"Taking your disks out and sending the box for repairs can be done, yes? Or are these things throwaway?"

Yes, I eventually convinced the local agent they had an obligation according to the manufacturer to repair the unit. The other option is to buy a 2nd hand unit off eBay.
Another option that may work is to buy the latest model NAS and put the drives into that, I'm waiting for Thecus to confirm if that will work.

One thing I have learned is to be very careful of data recovery services. One guy I've spoken to sounds totally honest, another I'm not 100% certain about and I know a lot are total rip offs.
As it turns out these Thecus boxes run Linux and I believe you can mount the drives onto most Linux distros and quite easily recover the data yourself. It's also possible and more desirable to first clone the disks onto another drive. Those images can then be mounted as virtual drives and there's a reasonably well supported piece of freeware code around to facilitate this.

Bob.

riredale wrote on 2/1/2013, 4:28 PM

Ah, I remember the days not too many years ago when I backed up my important files to 3 1/2" floppy. 1.3 MILLION Bytes, that's a lot.

Funny thing is how time is warped depending on the industry. About the time I would have used floppies for backup, "Groundhog Day" came out. I think of that as a contemporary movie. Guess I'm getting old.

DeadRadioStar wrote on 2/1/2013, 6:47 PM

@farss: very important, keep the original hard drives untouched so you can always try other methods. Your idea to virtualise them is good, but you may need to seek help to be able to reconstruct the RAID array under Linux -- unless Thecus tell you how . Even if you get the Thecus box repaired, it's still worth the investment to buy new drives, clone the origiinal disks onto them, and then see how far you get.

Byron K wrote on 2/2/2013, 3:28 PM

Yes, the raid controller is the weakest link on a raid system.

If you loose your RAID card you're stuck. I don't do RAID just for this reason. I just lost my P4HT motherboard which was providing RAID 0+1 array and now cannot recover anything from that machine. If it was just a standard drive or just a RAID1 I could at least remove the drives from the machine and recover.

Loosing my ASUS P4 motherboard was the last time I'll RAID my workstations.

I've also lost an adaptec RAID cards and a Netcell RAID card both of which have corrupted arrays that had to be rebuilt. The only card that I've used and recommend, that provided excellent stable RAID arrays is 3WARE. I've built a few server systems with these and none of them has failed. I've since move on from the desktop/workstation/server tech where I was always experimenting with this stuff, I just don't have the time to deal with that stuff anymore. (;

farss wrote on 2/2/2013, 4:17 PM

"I just don't have the time to deal with that stuff anymore"

You said it right there. I did find what looks like excellent recovery software: http://www.r-tt.com/

It isn't the cost of the licence that puts me off so much, it'd be just another thing I'd have to spend more time on dealing with learning, thanks but no thanks. Better to pass the problem to someone whose already been there and see the cost as a "s-t-u-p-i-d fine".

Bob.

Soniclight wrote on 2/2/2013, 5:14 PM

I don't use RAID and don't have the pressure of backing up commercial/business work, however I always have at least two system backups -- the last, and the second to last.

And since Bob brought up backup software and stupidity...

I started using True Image way back when and used to upgrade as I went along. BUT... at one point the "let's make this prog be more consumer friendly" cuteness and visual slickness ended up being too bloated and too many un-usable backups ensued. I now use the no-cost Acronis Disk Wizard (actually TI v.9--and that's really old) bundled with my last Seagate hard drive purchase BUT IT WORKS great.

And here is the s.t.u.p.i.d. aspect for me...

One of the reasons I felt compelled to upgrade from TI v.11 of TI Home 2011 at the time was that the Acronis site stated that my v.11 wouldn't be 64-bit compatible (I had just replaced my old 32-bit system). Then why is it that the 2-versions-

Sounds like a (rather dishonest) marketing hustle to me...

fldave wrote on 2/2/2013, 9:05 PM

I just have a basic "copy main data/work project folders" from 3 PCs to two HDD. Automatically every morning. Those HDDs get copied to two other external HDD automatically. I compare the drives a couple of times a month. About twice a year there is a discrepancy, that I correct. So three live copies of my important files. Yeah, it's about 12TB, but I haven't lost much the past few years.

farss wrote on 2/2/2013, 9:13 PM

Dave,
What software do you use to do that with?

Bob.

JohnnyRoy wrote on 2/3/2013, 9:33 AM

Sorry but I feel compelled to say this (not trying to "school" anyone... just sharing a best practice) ;-)

RAID is NOT a BACKUP all by ITSELF!!!

I can't stress this enough. RAID is meant to protect you from down-time by allowing a drive to fail and it keeps on going while you replace the drive and rebuild the array. It is not to be used as a backup of anything all by itself. It is robust on-line storage but you still need an off-line backup.

I have a 12TB RAID 5 that holds all of my old source files and projects in one place. I also have several off-line backup drives with contents from the RAID because RAID is not a backup. After every shoot, I capture the footage to the RAID and immediately back it up to another hard drive so that I have 2 copies. Then I delete the files from my camera storage. When I'm ready to edit, I copy the source from the RAID onto my project drive. When I'm done editing, the project gets copied back to the RAID and another off-line project backup drive. This way I always have 2 copies of old or future work and 3 copies of current work. (my project drive only contains current work)

BTW, I use Beyond Compare 3 to keep all of the backups in sync since I like to do this manually. It is fully integrated into the Windows Explorer and it will copy just what has changed so that I don't waste time copying files that haven't been touched (which are usually huge video files). This is also how I keep my current work backed up to the RAID while I'm working each day. I know I said RAID is not a backup but in this case my project drives has the master copy and the RAID drive has the backup copy so it really is a backup because it's a second copy. At the end of the project I back it up to both the RAID 5 and the off-line drive and delete it from my project drive so there are still two copies, one on the RAID 5 and another on an off-line backup.

Never have your only copy of something be on a redundant RAID alone. You still need a second backup.

~jr

monoparadox wrote on 2/3/2013, 9:53 AM

I regularly use Microsoft's "synctoy" and back up to external drives. The name is a misnomer. It's not a toy. Simple and easy to see what's happening. I learned long ago I want to see my files under all circumstances, copy/move them, and get out of Dodge. The greatest point of failure will be my mind and at that point I won't care.

At least the next person to come along, as long as DOS is existent will see the stuff.

tom

fldave wrote on 2/3/2013, 10:06 AM

Ha, Bob. I use DOS batch file! The switches cause it to copy only the things changed since the last copy. The .txt file contains a list of all of the files copied. This doesn't work with file names longer than 256, so be careful.

XCOPY I:\Projects\*.* \\xxx\yyyy\Projects\*.* /Y /c /e /k /r /m >>\\xxx\yyy\backupAComputer-last.txt

Barry W. Hull wrote on 2/3/2013, 11:01 AM

Just yesterday, I had four large video files, raw .M2T footage, corrupted as I was using Vegas. Vegas greyed out, next thing I know I have four corrupted files.

Thankfully I had them backed up. I use NovaBackup and it is fantastic, differential, incremental, and copy options.

I have a main "working" external hard drive and three additional external backups. If you are looking for a dependable, easy to use system, NovaBackup has saved my bacon on several occasions.

JR is right, if a file is corrupted, simply because you have it located within a RAID drive, it's gone unless you have it backed up somewhere else.

UlfLaursen wrote on 2/3/2013, 12:04 PM

Well, my NAS on 3TB is backed up every night to an external USB drive (3 TB) connected directly to the NAS, a Netgear pro NAS. It is syncing, so only new orchanged files are copied.

I have an HP microserver too with windows server 2008 too for all important files not video. These fies are backed up once a day to the NAS and synced to on-line storage at godaddy.com, they are quite cheap and seems really good.

Ulf

farss wrote on 2/3/2013, 2:35 PM

ftadce said:
"Ha, Bob. I use DOS batch file!"

Thanks Dave,
I used to have one like that a long time ago, you'd think we' have moved on a bit. Yes, the 256 char limit can be a pain, I think that applies to the full path+name, not just the name, I recall trying to extricate a bunch of files for someone and having a lot of grief because very deep nesting with long folder and file names.

JR said:
RAID is NOT a BACKUP all by ITSELF!!!

Indeed. All my static data e.g. camera original file are backed up three ways.
The advent of the affordable SxS card meant we could keep the camera original on the media it was recorded on until the job was out the door plus two other backups.

The issue is with dynamic data e.g. email files and databases. These can be quite large, my email file is around 3GB. Trying to devise a set and forget stratergy that is reasonably secure is not easy.

What is alarming in this digital age is how the public I think is being missled.
For example there's quite a few reasonably priced two disk NAS boxes that'll run RAID 1. These are promoted as being pretty secure, what's not disclosed is that yes, there's two copies of the data on two disks but if the box fails good luck getting the data off either disk easily.

The other amusing aspect to my current minor grief has been finding out that the most common cause of failure in these NAS boxes is the flash memory. The Linux system is held on a Disk On Memory module (DOM). Kind of ironic that in an effort to guard against the inevitable failure of mechanical spinning disks there's a single point of failure in a piece of technology that's supposedly more reliable.

Trying to go forward and come up with a simple, secure, easy way to store data is not proving to be a simple task either. For example these NAS boxes have USB 2/3 ports and an eSATA port. One would think it'd be a trivial task to plug a NTFS formatted disk into one of those ports and have the NAS do a backup directly to it, nope. Instead I'm offered features such as a HMDI port so the NAS can connect to a TV and I can play movies directly from it, argh :(

Bob.

TheRhino wrote on 2/4/2013, 6:11 AM

IMO there are (2) types of loss your backup plan should account for:
1) Theft, Fire, or Other Catastrophe
2) Hardware Failure

CATASTROPHE:
I keep a copy of all source video on hot swap drives which I store in a different location and use a script to regularly copy all work files (VEG, etc.) to a MediaSmart home server located in a hidden location in the studio and another I keep at home. Hopefully the common grab & go criminal will not find the hidden server, but if a fire gets it, I have my important stuff backed-up at home and I could work from home until the insurance replaced my studio gear...

HARDWARE FAILURE:
My fastest workstation has a massive RAID0 for source video (since a copy is on the hot swap drives...), a RAID10 for output files, and a RAID1 for work files (VEG, etc.). All drives are in hotswap bays connected to 3Ware hardware raid card. In the last 7 years I have had (3) hard drives fail and I had little down time.

When a RAID 10 drive failed, I just plugged in the spare and went to lunch. When I came back the RAID10 was restored. When my RAID0 failed, I simply copied the source video from the backup hot-swap drive & continued to edit.

I also have more than one workstation so that if one fails, I can just reload the VEG from the server, install the hot-swap drive with the source video, and continue editing on another workstation until I have time to repair the failed system.

All of my RAID cards, RAID drives & motherboards match, so if something in my key workstation fails, I can also rob parts from one of the lesser-used systems until I have a chance to order new parts. Fortunately my ASUS workstation class motherboards & 3Ware RAID cards have never failed. I am very pleased with my setup and it helps me to sleep at night...

TheRhino wrote on 2/4/2013, 6:38 AM

In addition to the above plan, I use Paragon to backup the OS drive about once per month. Each workstation used to multi-boot between XP, Vista 32 & Windows 7 64 but now they just have (3) instances of Win 7 64. The first boot option is just barebones Vegas 8, 9, 10e, the second has Vegas 8-12 plus all of my key programs installed, and the third has all of the above plus the Internet & all updates.

Currently I boot into the 3rd option, with the Internet, but prefer to begin all work in Vegas 10e. The only time I switch to V11 is when I want to test-out the Color Match FX or render a project that renders-to-black in V10e... I have not used V12 for paid work & will probably hold-off until the next generation of GPUs surpasses my CPU rendering speeds.

TeetimeNC wrote on 2/4/2013, 8:57 AM

Bob, I use Windows Home Server 2011 for my backups. I have a three client PC network, one of which is my Vegas editing PC. The WHS contains all of my shared data (music, pictures, video resources, etc.) and has a removable 2TB drive for backups that I rotate off-site each week. The client PCs are automatically backed up to the server each night, and the server (OS/, program files and shares) is backed up to the removable drive twice a day.

What I like about this is it just works with very little intervention from me. Basically all I do is rotate the backup drive weekly. WHS is pretty smart in its backups in that if there are duplicate files on two or more clients it only gets backed up once. It also automatically spools the oldest backups off the rotating drives as they become full. I do not use RAID anywhere in my network.

I also use WHS to host my rendered videos which can be played via my network connected devices (WD Live and a Blu-ray player).

Here is an article I posted with details of the WHS build: http://www.takeonesolutions.com/blog/windows-home-server-for-videographers/

One area where I am vulnerable - I don't have backups of archived projects. I have a set of bare drives I fill with archived projects that set on my shelf and I wish I had redundant copies of those off site or in the cloud.

/jerry

farss wrote on 2/20/2013, 5:37 PM

A local professional data recovery business got all my data back however their quite reasonable bill would have bought me a quite nice video camera :(

I passed the job to them because:
1) It's a lot of work, just creating mirrors of 5x 400GB drives takes a lot of time.
2) RAID recovery software isn't terribly expensive however it's not simple to use and determining the drive and RAID parameters is only solvable by brute force.
3) Every other alternative *might* have worked. Some Thecus users report being able to put the drives from old systems into new ones and all is good. Thecus themselves advise this probably will not work and it can trash the data on the drives. Even doing a firmware upgrade has the potential to trash your data. The problem here is that RAID 5 can only do single error recovery and doing a RAID migration with an exisitng data error is going to cause serious problems.

I'm now building a new NAS with 1TB WD Red NAS drives and then putting into place a functional backup plan.

Speaking with the data recovery people was a bit of an eye opener. No matter how good your data security might seem to be any single point of failure can and will fail. One recent job thay had involved 120TB of data stored on mirrored RAIDs connected a server with dual power supplies each run from its own UPS. A simple failure in the mobo of the server rendered all data unaccesible.

Another current job was was from a wedding photog. New camera, new CF card. Near the end of the wedding the camera reported it couldn't write to the card. They'll get the data back but that involves painstaking work and no doubt a hefty bill.

The most promising solution I've found is ZFS, developed by Sun. It solves the problem of "silent errors", designed to work with commodity drives and is open source. One can roll your own and some local lads have 1 Petabyte on home servers. Not quite backing up all of the Internet, just the "blue" parts :)

Bob.