Forums Archive -> VectorStar News

Data corruption last night... 2002-01-03 09:44:16
by lmbedore
last night, about 10AM GMT, something went haywire with our disk subsystem in apathy, corrupting some data on the main drive. The really bad news is that this happened DURING our nightly backup, thus corrupting the backup as well.

So far, two users' home directories and one documentation directory on our main site are the only things coming up as corrupt. If you experience ANY peculiar behavior, let me know.

Sad thing is, the data that was lost is unrecoverable. Not even a backup solution is foolproof.



Re: Data corruption last night... 2002-01-03 16:40:20
by superfly
I think my site. superfly.vectorstar.net got hit pretty hard by it. Luckily i have full backups of everything here. let me know when everything is cool. Its all good
Re: Data corruption last night... 2002-01-03 16:51:43
by superfly
hey i just checked the server again. seem that a project in part of my site is ok. just the portfolio and main sections are gone.. Thanks for saving the brisgeek data Laird http://superfly.vectorstar.net/bga/v1/index.htm
Re: Data corruption last night... 2002-01-03 17:12:56
by lmbedore
Wow. wierd. when manually going to your home directory, i see problems, but my scan doesn't pick it up. Hopefully I'll be able to recover your data (and others') tonight when I'm physically on-site.

We haven't been able to determine if our problem was caused by a failing hard drive, failing SCSI controller, or a software freakout. The errors logged aren't incredibly helpful.
Re: Data corruption last night... 2002-01-03 20:09:28
by lmbedore
Good News! We took the server down, replaced the SCSI cable (just in case it was bad), scanned the drives, got 'em fixed, and it looks like all user data was recovered. Don't take my word for it 100%, since I don't know exactly what data should or should not be there.

So, if anyone sees anything funky (like a file which contains the wrong data), yell at me or post up a comment here.
Re: Data corruption last night... 2002-01-03 21:56:05
by Superfly
nothing funky here. the site is 100% ok. no need to re-upload for anything but updates
Re: Data corruption last night... 2002-01-04 18:27:27
by stale
Heh, well if we did some incremental backup type stuff we would've had a copy that was valid from the night before. Did you notice that the voting booth is gone?
Re: Data corruption last night... 2002-01-04 19:38:01
by lmbedore
If we had 5 grand to drop on a robot tape system and software that works with it, i'd be so happy to fill up some tape.

oh shit. the entire cgi-bin directory and all of its data is gone. I have a REALLLLLLY old backup of it (september 2000), but there wasn't much of importance in cgi-bin anyways. The only real thing I lost was the User websites script, but it's easy to rewrite. I think i have a separate copy of the voting booth code backed up somewhere else as well.

Heh... this www backup is so old, it has the MessageBoard 1.1 code and flat-file DB's :) We were only at message 402 back then!

Well damn. we did lose some data after all. At least we didn't lose the new user signup, any of the admin tools, or the database server.
Re: Data corruption last night... 2002-01-04 21:17:48
by lmbedore
Correction:

It appears that 26 users have completely lost their home directories. I'll be working tonight to restore those users' home directories and services, but sadly their data is gone. :(
Re: Data corruption last night... 2002-01-05 16:24:12
by stale
Yeah, I guess swapping tapes on servers that are SO far away would require some kind of mechanical intervention. My bad :)