Sup guize! 
It's been quite a while since I've perused Sneap-land, and today I make my triumphant return with an absolutely priceless tutorial in data integrity, in lieu of my own personal experience (and disasters) so that none of you should have to feel the same pain I did.
Back story:
I had a very thorough method in place to ensure I never lost a single take to drive failure / data loss. In addition to regular external drive backups, my studio desktop rig had two 640 GB Western Digital Blacks in a RAID1 mirror array. For those not-so-tech savy, a RAID1 array is when you have two identical hard drives that the OS recognizes as just one drive, and every write gets mirrored to both of them, so that the contents of both drives is always completely the same. The idea is that if one of them fails, Windows doesn't even skip a beat and continues to use the other drive as if nothing happened. The user is quietly notified, the bad drive is replaced, and the array rebuilds itself and you never lose a single byte.
Sounds like a no-brainer for any studio engineer with the know-how to do some PC configuration, right? Desktop drives are so cheap these days that it's hard to find an excuse not to. I'd recommend anyone with a desktop rig (Windows or OSX) to set up a RAID1 array. There's other more complicated RAID arrays using many drives to get even more speed/data protection, but for the sake of pro audio, two desktop drives mirrored is usually plenty.
But I digress, if you want to learn more about RAID and other RAID configurations (and you definitely should if you're interested in editing HD video) you can google or wiki it.
The Problem:
Notice that throughout the description of the aforementioned data protection solution, I was speaking in the past tense. That is because what I thought was a bulletproof plan, was in fact, not. Here's what happened:
Some of my files, in this case guitar takes, became corrupted. A session that I hadn't opened in several weeks suddenly had several blank spots where there was previously guitar. I took a look at the take files, and found that a few of them were now 0 bytes. Peculiar, I thought, since I was reading these files from my mirror array. If one of them corrupted on one drive, shouldn't the good drive rebuild the missing data on the bad drive? As it turns out, one of the drives had some bad sectors, but was still partially functioning. Evidently, the corruption got mirrored to the good drive as well. I confirmed this by removing the drives and checking the contents of each of them individually. So in a RAID1 array, sometimes the corruption gets mirrored... sounds ridiculously counterproductive. Apparently, the system can only intelligently recover from and notify me of a total drive failure. I think, possibly, because I had S.M.A.R.T disabled in my BIOS settings (defaults, I have no idea why) the system was unable to foresee the failure of just a few bad sectors. The corruption was subsequently mirrored to the good drive.
So I think to myself, no worries, surely even if both drives in my mirror array have failed me, I always have my external drive backup, and I am always sure to back up to that after every tracking session.
Nope.
Since the basic copying system in Windows is not very intuitive, what I used to do is just drag and drop an entire session folder over to the external drive in Windows Explorer, and tell it to over-write everything. This means that somewhere along the way, I replaced the original takes on the backup drive, with the corrupted 0 byte ones. Totally fucked.
I made certain to always have my priceless studio data in at LEAST three places at all times, and still lost some.
When you have a thousand dollar job with a (really good) band that drives four hours at a time to track with you, realizing that you lost some takes is an absolutely nauseating discovery. And it is so that some of you may be spared this anguish, that I have shared this tutorial with you today. Here's how I changed up my process to ensure that such a catastrophe never occurs again.
The Solution:
First of all, a RAID1 mirror array is still a no-brainer for any desktop user. There is simply no reason to not have it set up because of how cheap drives are these days. Just make sure you BACK UP before setting it up! And if you have priceless studio material on there, which given the context of this tutorial I'm assuming you do, BACK IT UP TWICE! (SERIOUSLY!)
Secondly, and this is the real priceless tip, make differential backups to an external disk, using a tiny piece of freeware known as http://codesector.com/teracopy. TeraCopy is an absolutely genius, tiny, efficient piece of code. What it does is replace the shitty counter-intuitive file copy system in Windows with one that is absolutely delightful to use. And when I say external disk, I don't mean a portable hard drive, I mean a powered desktop external drive with its own power supply that isn't specifically meant to be portable, per-se. One of these should live on your desk permanently and should not be touched, because a drive that doesn't go anywhere is a drive that is less likely to fail. Use a fast interface like USB 3.0, eSata, or 1394.
What is a differential backup and why should I give a shit?
A differential backup is one that doesn't touch the shit that's already backed up. So say you already have your main hard drive backed up to an external. Then, you work on a session tracking all day. You don't want to completely copy over your entire main drive again, you just want to UPDATE the backup with the new stuff. But, at the same time, for a studio engineer there could be thousands of new files and an assortment of project files that have been updated, and it can of course be tricky to know specifically what needs to be backed up. With Windows alone, the only way to make sure you haven't missed anything is just to plop the whole thing over there again and have it over-write what was already there. With TeraCopy, when you drag everything over in Explorer, the first thing that happens is you are presented with a much more intuitive dialogue in regards to how you want the copy procedure to be handled.
And all we need to do, is smack the "Older Only" button. What this does is skips the files that are already there, UNLESS you've modified them since the last backup. This means session files and things like that will be updated, but takes will not. This ALSO means, that if a file (such as one of your takes) becomes CORRUPTED on your main hard drive, the ORIGINAL doesn't get replaced with the corrupted file. This is because the file system only updates the "date modified" flag on a successful write. If a file is modified due to corruption or bad sectors, the "date modified" flag does not get updated, and thus the bad file doesn't replace the original.
Additionally, TeraCopy offers another priceless and ingenious feature, the "Verify" button. This will run a CRC (cyclic redundancy check) on all of the files on your backup versus the originals. It checks all of them, even if you're doing a differential backup. This means if a single byte in a single file doesn't match up with your originals, you'll spot it quick, thus identifying corruption on the spot before it has a chance to spread and ruin your life. This feature helped me spot a few other files in other places that had been messed up from my mirror array. More often, hopefully, the verification will provide peace-of-mind when you're reminded that everything is definitely intact.
So that's it! Pretty bulletproof!
To re-cap:
1. Get TeraCopy - http://codesector.com/teracopy - (it's freeware.)
2. Use it to keep a backup of all of your crucial studio data on an external desktop drive. When updating your back up, use the "Older Only" button.
3. If you aren't confined to a laptop, set up a RAID1 array. It's a fun project and good IT learning experience.
Mazel tov! :Spin:
It's been quite a while since I've perused Sneap-land, and today I make my triumphant return with an absolutely priceless tutorial in data integrity, in lieu of my own personal experience (and disasters) so that none of you should have to feel the same pain I did.
Back story:
I had a very thorough method in place to ensure I never lost a single take to drive failure / data loss. In addition to regular external drive backups, my studio desktop rig had two 640 GB Western Digital Blacks in a RAID1 mirror array. For those not-so-tech savy, a RAID1 array is when you have two identical hard drives that the OS recognizes as just one drive, and every write gets mirrored to both of them, so that the contents of both drives is always completely the same. The idea is that if one of them fails, Windows doesn't even skip a beat and continues to use the other drive as if nothing happened. The user is quietly notified, the bad drive is replaced, and the array rebuilds itself and you never lose a single byte.
Sounds like a no-brainer for any studio engineer with the know-how to do some PC configuration, right? Desktop drives are so cheap these days that it's hard to find an excuse not to. I'd recommend anyone with a desktop rig (Windows or OSX) to set up a RAID1 array. There's other more complicated RAID arrays using many drives to get even more speed/data protection, but for the sake of pro audio, two desktop drives mirrored is usually plenty.
But I digress, if you want to learn more about RAID and other RAID configurations (and you definitely should if you're interested in editing HD video) you can google or wiki it.
The Problem:
Notice that throughout the description of the aforementioned data protection solution, I was speaking in the past tense. That is because what I thought was a bulletproof plan, was in fact, not. Here's what happened:
Some of my files, in this case guitar takes, became corrupted. A session that I hadn't opened in several weeks suddenly had several blank spots where there was previously guitar. I took a look at the take files, and found that a few of them were now 0 bytes. Peculiar, I thought, since I was reading these files from my mirror array. If one of them corrupted on one drive, shouldn't the good drive rebuild the missing data on the bad drive? As it turns out, one of the drives had some bad sectors, but was still partially functioning. Evidently, the corruption got mirrored to the good drive as well. I confirmed this by removing the drives and checking the contents of each of them individually. So in a RAID1 array, sometimes the corruption gets mirrored... sounds ridiculously counterproductive. Apparently, the system can only intelligently recover from and notify me of a total drive failure. I think, possibly, because I had S.M.A.R.T disabled in my BIOS settings (defaults, I have no idea why) the system was unable to foresee the failure of just a few bad sectors. The corruption was subsequently mirrored to the good drive.
So I think to myself, no worries, surely even if both drives in my mirror array have failed me, I always have my external drive backup, and I am always sure to back up to that after every tracking session.
Nope.
Since the basic copying system in Windows is not very intuitive, what I used to do is just drag and drop an entire session folder over to the external drive in Windows Explorer, and tell it to over-write everything. This means that somewhere along the way, I replaced the original takes on the backup drive, with the corrupted 0 byte ones. Totally fucked.
I made certain to always have my priceless studio data in at LEAST three places at all times, and still lost some.
When you have a thousand dollar job with a (really good) band that drives four hours at a time to track with you, realizing that you lost some takes is an absolutely nauseating discovery. And it is so that some of you may be spared this anguish, that I have shared this tutorial with you today. Here's how I changed up my process to ensure that such a catastrophe never occurs again.
The Solution:
First of all, a RAID1 mirror array is still a no-brainer for any desktop user. There is simply no reason to not have it set up because of how cheap drives are these days. Just make sure you BACK UP before setting it up! And if you have priceless studio material on there, which given the context of this tutorial I'm assuming you do, BACK IT UP TWICE! (SERIOUSLY!)
Secondly, and this is the real priceless tip, make differential backups to an external disk, using a tiny piece of freeware known as http://codesector.com/teracopy. TeraCopy is an absolutely genius, tiny, efficient piece of code. What it does is replace the shitty counter-intuitive file copy system in Windows with one that is absolutely delightful to use. And when I say external disk, I don't mean a portable hard drive, I mean a powered desktop external drive with its own power supply that isn't specifically meant to be portable, per-se. One of these should live on your desk permanently and should not be touched, because a drive that doesn't go anywhere is a drive that is less likely to fail. Use a fast interface like USB 3.0, eSata, or 1394.
What is a differential backup and why should I give a shit?
A differential backup is one that doesn't touch the shit that's already backed up. So say you already have your main hard drive backed up to an external. Then, you work on a session tracking all day. You don't want to completely copy over your entire main drive again, you just want to UPDATE the backup with the new stuff. But, at the same time, for a studio engineer there could be thousands of new files and an assortment of project files that have been updated, and it can of course be tricky to know specifically what needs to be backed up. With Windows alone, the only way to make sure you haven't missed anything is just to plop the whole thing over there again and have it over-write what was already there. With TeraCopy, when you drag everything over in Explorer, the first thing that happens is you are presented with a much more intuitive dialogue in regards to how you want the copy procedure to be handled.
And all we need to do, is smack the "Older Only" button. What this does is skips the files that are already there, UNLESS you've modified them since the last backup. This means session files and things like that will be updated, but takes will not. This ALSO means, that if a file (such as one of your takes) becomes CORRUPTED on your main hard drive, the ORIGINAL doesn't get replaced with the corrupted file. This is because the file system only updates the "date modified" flag on a successful write. If a file is modified due to corruption or bad sectors, the "date modified" flag does not get updated, and thus the bad file doesn't replace the original.
Additionally, TeraCopy offers another priceless and ingenious feature, the "Verify" button. This will run a CRC (cyclic redundancy check) on all of the files on your backup versus the originals. It checks all of them, even if you're doing a differential backup. This means if a single byte in a single file doesn't match up with your originals, you'll spot it quick, thus identifying corruption on the spot before it has a chance to spread and ruin your life. This feature helped me spot a few other files in other places that had been messed up from my mirror array. More often, hopefully, the verification will provide peace-of-mind when you're reminded that everything is definitely intact.
So that's it! Pretty bulletproof!
To re-cap:
1. Get TeraCopy - http://codesector.com/teracopy - (it's freeware.)
2. Use it to keep a backup of all of your crucial studio data on an external desktop drive. When updating your back up, use the "Older Only" button.
3. If you aren't confined to a laptop, set up a RAID1 array. It's a fun project and good IT learning experience.
Mazel tov! :Spin:
) have overlooked:
