Well, it only took the better part of a week, but the backup process is complete. We've gone from having data on 1 out of 3 drive to having it on all 3. The RAID is restored, we swapped out the funky RAID card for a much improved one, and we got it booting off the RAID again. This is all pretty non trivial when you consider how screwed up everything was to begin with- apparently, some confluence of the old hardware RAID and the software RAID after I upgraded the server to FreeBSD 8.0 (the last time we had any significant downtime), got all screwed up- it went from a RAID-1 (mirrored), to running as a RAID-0 striped with 1 drive. That's pretty scary stuff! I don't like to play on the edge of disaster like that. First thing we had to do was simply figure out a way to pull a good backup from that busted configuration. Even that took several days. Then we had to wrestle with the new RAID card; unfortunately it does some things to the drive that really complicated the issue of restoring all the data from the backup onto the new mirror. We had to write a partition table for the new drive by hand, of all things! It was seriously nuts. After all was said and done, and Bill was about ready to do a victory lap, he discovered that not all is perfectly well, even now. And it gets better- it shouldn't even be possible. However, everything seems to be working perfectly- I haven't found any problems with anything yet. So we're going to sleep on it for a bit.
A thousand thanks to Bill for getting us through this. Bill is very much a mentor figure for me, a very good friend, and an overall brilliant guy. I waited to do this project until I could get his assistance, and man am I ever greatful I bet against myself. He kept the downtime to below a week. If I had been doing it alone, I'm not sure it would ever have come back. I've been living with my curse long enough now that I have a pretty good idea what kinds of situations it likes to kick in for. This morning I was working on my MisterHouse installation for fun, and trying to get the server to play music. As I expected, it doesn't work. I put in a new sound card, loaded the drivers, and everything looks good, but I'll be damned if I can get it to make any sound. cat /dev/random > /dev/dsp0.0 doesn't even make a peep. This is supposed to be really easy- sound is not supposed to be a big problem. Yet here we are- the curse in full effect.
Next step is to get the local backup system in place, and to get all the data on the Datesim.org server backed up- in particular, the database for Datesim. I'm thinking of just mirrororing it on the local server, but we'll see. That always runs the risk of corruption hitting everything.