Last week I got the call all developers fear. Our excellent colocation provider, Hosted Solutions, called to say one of our servers was not responding.

After a few minutes of testing, I hopped in the car and headed down to see what was up.

One of our Xserves had fallen and couldn’t get up. After an hour of trying everything, the last rites were read, and I moved into recovery mode.

In our data center we have a number of Xserves and Mac Minis.

The Xserves do all the heavy lifting (database, mail servers and business applications, and minis handle trivial tasks, like serving static content and running low traffic applications (like this blog).

The hard choice was what to sacrifice in order to restore essential services.

The mail server with all the anti-spam, virus and archival functions would never run on a Mac Mini.

One of the Xserves was running a mix of legacy and recently released applications.

The new applications, having been built using Ruby on Rails not WebObjects, I knew could run anywhere. In a matter of minutes, with a a few swapped Xserve drives, the mail server was back in full swing.

The other Xserve drive was placed in a separate drive bay to allow access the application files just in case they were needed. It mounted perfectly and a few tests later I was satisfied stability had been achieved.

I headed home to strategize the restoration of the business applications.

Our old creaky WebObjects applications would not run on anything less than the Xserve running Mac OS X Server without some major web foo. So we decommissioned those for now.

That left four applications, two of which were critical, to get back online.

Within a couple hours all four applications were running on one of the minis under Lighttpd and FastCGI, with the databases still hosted on Xserves.

Surprisingly, the applications are performing better on the Mac Mini than on the Xserve.

We had been planning a major restructuring of our infrastructure. The Xserves are all the first models Apple released and are growing long in the tooth.

The dead Xserve is now being dissected to see if it can be resurrected. But it will have a whole new purpose if it makes back into the line up.

This is providing the opportunity to try out a day dream of mine.