Jun 3, 2008 : The Planet Reports on Server Outage

📅 June 3, 2008 - As it attempts to recover from an explosion that cut off power to 9,000 servers at its H1 data center in Houston Saturday, The Planet (theplanet.com 👉 Total Reviews: 3
🙌 Average Rating: 1 / 10
👍 Good Reviews: 0
👎 Bad Reviews: 3
👈 Official Responses: 0 -> softlayer.com -> ibm.com) has been working to keep customers in the loop on a situation that certainly has at least some of the makings of a "pr nightmare."


Since Saturday, the company has been battling to bring servers back online, after an explosion reportedly destroyed much of the electrical equipment room in the facility and a significant amount of underground cabling.

While repairing the damage and getting servers back online has obviously been the top priority in responding to a situation that has apparently affected 7,500 customers, keeping those customers informed has also been a major concern.

The Planet has been posting hourly updates to its forums (even issuing regular "no new information to report at this time" notices).

But the company, which for the moment may be relying on a certain amount of hard-earned benefit of the doubt from customers, also knows the "time is money" adage applies as directly in the web hosting business as anywhere else. In particular, "down" time.

Late Monday, CEO Douglas Erwin reported on the forums that Phase 2 of the facility had been online and powered up since just after sunrise Monday morning. For the more directly-affected Phase 1, the company set up an alternative means of restoring power, running off a generator while it rebuilt the electrical room.

As of about 5:00 p.m. Central Time Monday, The Planet was powering up Phase 1 and beginning to bring servers online.

Monday evening, Erwin issued an audio message to customers, explaining some of the details gleaned from the company's own investigation of the explosion. He says the explosion appears to have occurred in the underground conduit that brings utility power in to the facility, and that the explosion "damaged beyond repair both the electrical gear where the utility service enters the building and the transfer switch and main distribution panel that feed the first floor of the data center." 

On a somber-but-fortunate note, Erwin says a technician might have been patrolling in the neighborhood of the very destruction was thankfully elsewhere working with a customer when all four walls of the room were destroyed.

Erwin's message also describes the work many of the company's suppliers did, pitching in to bring the power back online.

Finally, he explains the company's immediate priorities, having brought the power back to Phase 1 of the building. First, he says, The Planet will work on stabilizing the environment and working with customers who are still having problems. Then it will turn its attention to rebuilding the electrical room and getting everything back up to standard. Third, it will develop a plan for minimizing the impact on the servers in Phase 1 when the company switches back to the rebuilt facility power source.

"After that," he says, "I owe you an answer as to how we will handle the SLA credits we need to distribute - when it will get done, and how it will get done. I owe you those answers after I finish the aforementioned priorities."

Further reports from The Planet's forums said the generator power to Phase 1 experienced some failure at 2:20 a.m., which required the company to bring in a replacement generator, and replace faulty breakers on the generator.

Later this morning, the company reported that efforts to replace the faulty breaker were unsuccessful and that the replacement generator was on its way to the facility. As of The Planet's last posting to its forums, the new generator was "expected to arrive this afternoon."

The Planet says it will issue more information about the new generator when it arrives.