[taoster]

Martinos Center for Biomedical Imaging - Backups

try a backup solution instead, even if it is external to external.

 

[OverlordXenu]

RAID is for redundancy and/or speed, not backups. It is far from fail-safe.

+1 for offsite and/or external HDD backup.

 

[Chu]

I'm currently working on a server for many collections of mine including music.

The biggest problem I have with most RAID systems, besides the fundamental problems associated with RAID5, are that they are not easily upgradeable.

The best solution right now in my opinion is ZFS. You essentially create arbitrary storage pools, and those pools can be further used to create new pools. Because of this completely abstract approach to physical storage you can emulate a huge variety of RAID structures and create new ones to meet specific demands.

It also allows upgrade by replacement and rebuilding with larger drives, something that RAID5 should theoretically be able to do but noone seems to support.

One important note about RAID systems though. Try to get all your hard drives from different manufacturers, or at the very least different batches. Drives in the same batch and the same manafacture have similar life expectancies.

More about ZFS here.

 

[Chu]

Quote:

Originally Posted by OverlordXenu /img/forum/go_quote.gif RAID is for redundancy and/or speed, not backups. It is far from fail-safe. +1 for offsite and/or external HDD backup.

I fully agree with this, but the big problem right now is there really is while it's dirt cheap to build 10+ Tb file servers at the consumer level, there isn't any consumer solution out there that will let you back it up.

Where are my 100Tb tape drives

 

[hempcamp]

Another vote for manual or automatic hard disk backups. If something gets corrupted in a RAID, the corruption can perpetuate to all involved disks.

--Chris

 

[HFat]

No one mentioned the first cause of data loss: user error.

RAID is not a backup. It's an availability solution.

Quote:

Originally Posted by mofonyx /img/forum/go_quote.gif Backing up to an external source? I'm already using an ext HDD, should I buy another one just to use for backup?

Yes. If you care about the data, you need at least two independent drives, preferably more. If you could recreate your data by re-ripping your CDs or something, it's less of an issue.

Quote:

Originally Posted by mofonyx /img/forum/go_quote.gif How do you do it automatically for each music file.

You use software that's designed to do that. I usually roll my own with rsync, robocopy or some such but there are innumerable tools out there.
rsync is cool if you only have two drives in that you can copy the modifications from a drive to the other without having both connected at the same time (it's called batch mode or some such)... and if you only have two drives, you don't want to connect them at the same time!

EDIT: oh yeah, and with rsync+hardlinks you can keep old mirrors around at minimal disk space cost if there's little turnover in your data (as with the typical music library).

 

[HFat]

Quote:

Originally Posted by Chu /img/forum/go_quote.gif the big problem right now is there really is while it's dirt cheap to build 10+ Tb file servers at the consumer level, there isn't any consumer solution out there that will let you back it up.

Consumers with 10T+ servers?!?

How about a couple more servers? Or maybe you want remote backups, can't do diffs for some reason or can't afford the bandwidth for them, and don't want to lug drives around? Yeah, there's still a use for tapes... but people without those requirements can simply back up to another server.

 

[Chu]

I think you're hinging too much on the word "external."

If your backup solution is an external drive and software to synch them on the fly (i.e. the external drive for all intents and purposes is part of the system, it's never actuially unmounted, just physically external), you might as well just use raid 1 instead. It's essentially the same thing.

 

[HFat]

Quote:

Originally Posted by Chu /img/forum/go_quote.gif you might as well just use raid 1 instead. It's essentially the same thing.

Not unmounting is silly but even then it's a lot better than RAID because you can control when the replication happens which means that you can roll back frackups.
Now, if you've got a reliable solution for keeping old mirrors on your main drive(s) and restoring from them, then hardware failure might be your main concern and RAID might start to look good as long as you don't end up relying on proprietary hardware or some such to access your data.

 

[Chu]

Quote:

Originally Posted by HFat /img/forum/go_quote.gif Consumers with 10T+ servers?!? How about a couple more servers? Or maybe you want remote backups, can't do diffs for some reason or can't afford the bandwidth for them, and don't want to lug drives around? Yeah, there's still a use for tapes... but people without those requirements can simply back up to another server.

I guess "consumer" isn't the right word, but I can't think of an equivalent to the word "prosumer" that you see in other geek circles.

Right now for about $5k you can put together a 5 Tb file server with 5:2 redundancy (it's two 7 drive arrays with 2 parity drives, you essentialy need to lose 3 hard drives before there is a chance your data is in danger per array). In a year it'll probably be double that since Segate and WD are making some pretty wild claims about how far Perpendicular Recording can go without any new technological breakthroughs (10Tb hard drives in a decade!). It's actuially probably a lot more space now -- I priced this out completly before 1Tb drives for $250 became common, PCI-E controller sistation was pretty grim in terms of options, and 1KW power supplies were rare since 750W power draw from a consumer system was unheard of before triple-SLI.

You really can't do any real disaster recovery though. Tape Drives are lagging behind hard drive capacity by so much it's sad. The cheapest storage library I can find that could do a true backup of my system is approaching $10K with tapes! Might as well just build a 2nd complete server and mirror it.

Really I miss the days where you could just go out, buy a $300 tape drive, and just dump your entire system to tape every once in a while as a backup solution. It's been so long I don't think people even realize this was at one point possible.

I know your average consumer doesn't really care about this now, but I think the lack of an easy way to do durable backups easily is going to be a big problem in the future. More and more a hard drive failure for an individual (much less a small buisiness!) is a disaster and a backup hard drive really doesn't cut it. It's better then nothing but it's simply too fragile.

 

[Chu]

Quote:

Originally Posted by HFat /img/forum/go_quote.gif Not unmounting is silly but even then it's a lot better than RAID because you can control when the replication happens which means that you can roll back frackups. Now, if you've got a reliable solution for keeping old mirrors on your main drive(s) and restoring from them, then hardware failure might be your main concern and RAID might start to look good as long as you don't end up relying on proprietary hardware or some such to access your data.

I might be misunderstanding what he wants, but if he truly wants an on-the-fly automated backup solution then really he can't control when replication happens and the backup storage is pretty much forced to remain mounted.

I personally would rather have an external drive and just robocopy it every once in a while though. I don't think a permanently logicially and physically connected really qualifes as a "backup" either.

 

[HFat]

Quote:

Originally Posted by Chu /img/forum/go_quote.gif a backup hard drive really doesn't cut it. It's better then nothing but it's simply too fragile.

That's true, you need several drives.

Hard drives last longer than tape drives in my experience because you don't use backup HDs much while tape drives have to work hard. Not everyone is willing to keep expensive spares or to rely on eBay for parts. IDE lasted a whole lot longer than any tape "standard".

 

[HFat]

Quote:

Originally Posted by Chu /img/forum/go_quote.gif I might be misunderstanding what he wants, but if he truly wants an on-the-fly automated backup solution then really he can't control when replication happens and the backup storage is pretty much forced to remain mounted.

Well, it depends on what's automated. It could mean you only have to click on a shortcut after connecting the media to backup or something.
If it was truely automated, you wouldn't get actual backups as you say. You'd need a backup server which protects the old backups or something.

 

[mofonyx]

Thanks for all that discussion. I'm really bogged down with work right now, but I am watching this thread.

Too much to read at the moment, but I will get back on this thread soon enough!

Thankss again guys.

 

[mofonyx]

Okay, I still feel that RAID 1 configuration would be best.

How would it be different than running 2 external drives with scheduled copying and backing up of data?

I doubt that there would be mess-ups in the drive because really all I'll be storing in it would be music files (*.flac) and nothing else.

Any other nonsense (series, movies, games, downloads etc.) will be on my other external drive that I have now.

I'm still concerned about the part where the RAID controller might fail.

If it does, can I just buy another (from the same make) and plug and play?