We became so dependant on the server over the last month that when it crashed it completely shut our dev work down for a day.  We are able to use one central account for project management (Daylite), manage our graphics properly (Version Cue), host dev work and even some non-mission critical, live websites.

I decided to upgrade the server over the weekend but before I did, I upgraded my local machine (which is also a Leopard Server) to make sure everything was fine with the update.  The local upgrade went without a flaw so I assumed (did you get that – assumed) it would be ok to upgrade the server as well.

I proceeded to install the update using Apple’s auto-updater.  Everything went fine until a big ugly dialog popped up “Unable to install update”…… <system resets> …… <flashing folder with a big x>….. <flashing apple icon> ….. <alternating between the two>.

So I immediately contacted Brian, who runs macminicolo.net, to see what he said.  His immediate reply was mine:  “Uh oh”.

To make a long story short: Apple’s update was a partial update, leaving system files damaged.  And to make matters worse, when trying to restore from a Time Machine backup, it couldn’t even recognize the hard drive!  At this point, I was in a panic.  I had several projects that needed to get done ASAP, a graphic designer that needed to access files, and several live sites that weren’t live anymore.

With the excellent help of Brian Stucki, we installed a fresh copy of Leopard Server.  I then looked at the Time Machine backup and surfed the files, mentally creating a backup strategy.  Surprisingly, it was very easy!  I opted not to use Time Machine’s GUI to restore files because it was too time consuming and more of a delicate situation than just restoring a file or two.  After inspecting the files, I fired up Server Admin, setup the ftp, web, open directory, and MySQL services.  I then just copied everything back to where they were and it worked!  I was leery on how to restore MySQL because I am used to just importing SQL files but it turns out copying over the actual MySQL files (.myd files, etc.) did the trick.  I was amazed at how simple it was to restore everything.  From the second I logged into the fresh, new server install, it took me an hour to get everything up and going again.

Though it was easy, I did run in to some snags along the way.  Due to copying the files from Time Machine, the old permissions carried over.  This wasn’t much of a hassle as it took very little time to change them.  Two of the MySQL tables became corrupt as well.  Fortunately, it was just a matter of recreating the tables.

Once the server, files, graphics, etc. were all in place I ran to the web browser to test all the sites and everything was fine.  The only caveat was Time Machine’s old backup.  The last backup Time Machine performed was missing about 45 minutes of work.  It wouldn’t have been a big deal to recreate it but that’s where my second backup system came in:  SugarSync.  I was able to download the latest files from SugarSync and all was fine.

Lesson learned:  Backup all the time!

On top of that lesson, I’ve learned another:  use multiple backups!  Because of this incident, I’m looking for a third way to backup.

Many thanks to Apple for Time Machine, Brian for his excellent support at macminicolo.net, and SugarSync for a great sync-backup tool!