Jan 17 2007

When your life comes crashing down

Category: Insanity,Life Stories,Things That Get Me Maderm @ 9:40 am

It happens, people die, hard drives crash, files get erased.

I really feel guilty for using that analogy. After all a bunch of deleted files can’t possibly compare to the grief of loosing a loved one.

Recently – within the last 3 days- I somehow managed to recode a mission critical

I don’t know if I should share with you how my hard drive crashed … or partitions got erased, but I might as well. So you won’t do the same thing.

To start our story we must go back to the root of the problem.

When I was installing linux I thought it would be a good idea to have a /var folder for my www data, and a / partition. I neglected to create a /usr partition. After a while / started getting full.

My /tmp directory caused mysql to wine about no space. So I moved it to /var/tmp

It kinda bothered me that I didn’t have enough space, so I started uninstalling various programs I didn’t need. I only had about 300-500 meg free by the end of it. This kinda bothered me.

So I decided that there was plenty of room on my “backup” drive that I could make a nice hefty 10 gig partition for /usr and use some of that empty space.

Keep in mind this is my backup drive. It’s a separate drive I back everything up to.

I figured worst case scenario I would have to reinstall /boot.

I backed up /etc /boot and just to be safe a copy of my latest mysql dump that’s ran every day, and then zipped.

I really don’t know why I was worried about those files, but I did. I guess it’s because of all the work I did on the dns recently, and well … /etc is pretty vital. Mysql is equally vital to me.

I don’t know what i was thinking not backing up /var/www/ It’s one of the most important folders on my computer. I guess I was thinking … I’m not playing with /var so everything should be ok.

In preparing for my trip to the unknown insanity which my life was about to become I also downloaded the latest kubuntu linux cd. A live cd … I love live cds. A full os on a CD that is really quite functional.

I popped in the live disk to reparation my drive.

All went well /var was there, the old /usr was there, and the new /usr was there as I expected /boot needed to be reinstalled.

Just to make sure, kubutu gives you the option of booting to the hard disk.

No problems there. The server was there, and intact.

Everything was exactly how it should be minus /boot.

So I copied over the backup of /boot and … rebooted.

Nothing … hmmm still couldn’t boot from the drive without the CD.

I didn’t want to have to have a cd in the drive just to boot the hard drive, so I loaded up the live cd.

Looking back … having it installed the way it was dumb.

I first had windows on /dev/hda and linux on /dev/hdb

The windows drive later became my backup drive … just so you know why I had hdb as my boot drive.

Long story short I some how erased /var. I guess I saw that free 10 gig on the drive, and said why don’t I use that as /usr instead.

I guess I clicked the wrong line, and when I caught it /var was gone gone gone.

What is kept in var …. /var/mysql *I care about this* and /var/www *I care about this*

At first I panicked. Three years of my life … no you don’t understand three years of my life was gone .gone gone ….

I laughed and laughed and laughed. Just gone like that.

Backup backup backup. I thought to myself. Always backup. But it should be ok … it was ok why did i do this I could have just booted from the CD for the rest of my life. Why did I have to play with the drive like a loose tooth that was waiting to be pulled out. Like a scab that needs to be ripped off. Like a hangnail. WHY. Why did I play with it like pealing paint on a bedroom wall that you know if you pull it you’ll just end up painting the whole room.

ARGH hahahaha

So I called the parties involved. It’s bad enough when you delete your drive, but when you delete your clients /bosses data … it’s bad really bad.

At first I wanted to tell him to F-off and just dump on him every little thing I ever thought of saying to him, but I didn’t.

In fact he asked me if I was quitting. I said … Actually I was wondering if I still had a job, he asked me if he wanted me to have him get an external usb drive, and I told him no, this is my f-up, I’ll get it.

So it was recoup time. At this point in time I thought I had lost all of /var Most but 1 key area was hidden and intact.

I got off the phone, and went straight to Sam’s Club. Pretty cheap drives. I grabbed a 250 GB drive, so I wouldn’t have to mess with the other drive I just goofed up, just in case I could find some sort of data recovery tool. It was worth $100 to me to get a drive that I didn’t always have to wonder about maybe I can retrieve the data ….

I also grabbed an external 100 GB usb drive.

At this point in time my head is spinning, and I’m just asking myself what do I need to do first, what first?

What’s the first step?

Set up http, no first step is … first step is.
Well obviously it was put in the drive, but that wasn’t what I was thinking about.

For the record I went the speed limit the whole way 🙂

I kept thinking to myself … no I’m not going to kill myself over a bunch of pictures, it’s just too stupid…

It’s a good thing that my boss doesn’t care about pictures that were already sent to the client.

There were quite a few “Oh thank God” moments, and “thank you Jesus” while I was plugging away that night.

I thought I lost my database … I didn’t I forgot I even backed it up … Thank God I had the presence of mind to back that up.

So the night kept getting better.

Damage control was … I just lost /var, and not *all the pictures* just the ones that were of my children … that were backed up on my wife’s computer … I hope.

I still had my precious phpMUR, and xphMUR because I kept them in /home/erm/ 🙂

My friends website was unharmed as well, just one small area was affected … which I don’t think he cared about, but I’m not sure.

On the first day I set up the following servers:
DNS, HTTP, SSH, SMTP, POP3, and MYSQL

Most of those were pretty easy since I had the backup of /etc. If your BIND9 dns is complaining about permissions, just type killall bind9, and restart. It wasn’t letting me kill it using /etc/init.d/bind9 restart.

My second third and fourth days were spent re-developing a mission critical site … well what I get paid for anyways.

It’s weird deciding … should I use the same open template I did before when I make this, or try something new. I’m boring I used the old template.

Granted the css isn’t quite the way I like it on the new site, because I just haven’t had the time to tweak it, but overall I have a site that is 110% better than the old one in some areas, and 85% in others.

Overall I’d say that the site is 90% functional in comparison to the old site.

It’s weird when you recode something, and you think to yourself … why didn’t I do it this way last time, this is SOO much easier. I’m glad I had my db, and html classes. Those 2 saved so much time.

In other areas, like my calendar class which I lost I was thinking to myself … now how did I do that?

I don’t know if this was a testament to my coding/server_admin skills, or a testament to how dumb 1 person can be, but now that I’m almost done with the mission critical site … I can see a light at the end of the tunnel, and realize I must be pretty good at it.

Erm

Leave a Reply

You must be logged in to post a comment. Login now.