June 2, 2008 -- (WEB HOST INDUSTRY REVIEW) -- Dedicated hosting provider The Planet (theplanet.com) suffered an explosion, 4:55 pm CST on Saturday at its H1 data center in Houston, Texas, which led to an outage that affected approximately 9,000 servers and 7,500 customers.
The explosion and accompanying fire knocked down three walls surrounding the facility's electrical equipment room, as well as destroyed the underground cabling that powers the first floor of the facility. Fortunately, there were no injuries, nor were any servers damaged or lost.
The company has been in constant communication with its customers -- an essential aspect of any disaster recovery plan -- through its forum (forums.theplanet.com) and customer portal, as well as other popular forums like Web Hosting talk (webhostingtalk.com).
In an earlier post yesterday on The Planet's forums, Kevin Hazard said the "short was in a high-volume wire conduit" and that the fire department instructed the company to not "activate [its] backup generator plan."
After the fire marshall inspected the facility The Planet was allowed to restore power to the facility. The company immediately turned back on the generators to receive power on the second floor.
Then, earlier this morning, the company successfully brought back 90 percent of the second floor servers after more than 40 hours of troubleshooting. Meanwhile, support technicians are currently working on location to manually restore the remaining 10 percent on the second floor, as well as restore power to the first floor.
Despite this, many customers, namely those companies who have lost sales, customers, and other revenues, remain upset by how long the recovery process is taking and have actively voiced these criticisms on forums.
One WHT member C0bra writes: "I'm fed up waiting for more information about these terrifying issues. Communication is poor and every minute my site is down is costing me $1000s in ad revenues. This is probably going to cost me a $10,000,000 sponsorship deal with a major brand advertiser."
Douglas Erwin, The Planet chairman and CEO, responded to these complaints late Sunday evening onThe Planet forum.
He writes: "I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service.
"We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed."
COMMENTS
Regarding that c0bra person.. if he's really raking in $10,000,000 advertisers, he should invest some of that money in round-robin DNS, redundancy in terms of multiple data center locations, etc. Idiot.
posted by: Whir reader | June 02, 2008 12:26PM
c0bra, yeah. Sweet. WHT is such a font of useful information. Not.
Everyone is losing "thousands of dollars an hour" every time there is a hosting outage. Doesn't matter if they are a $4.99 a year site that gets 10 visits a month. You know, that next visit could be the MILLION DOLLAR DEAL!
What a tired lie. People really need to think of something new to say during an outage..."Hey, my brain surgeon can't access my brain scan images! He's in the middle of surgery on my brain now! Get the server back up!"
That's a good one. Try that next time c0bra.
posted by: David Fink | June 02, 2008 12:39PM
I think it goes without saying at this point that multi-million dollar businesses that rely on a web server to be online are exactly what disaster recovery is for. This Cobra guy's claims seem, at the least, to be an obvious exaggeration. However, I'm sure the lost-ad-revenue impact of this outage, if it could be calculated, would be astonishingly high.
posted by: Liam Eagle | June 02, 2008 12:51PM
Liam I agree with you especially since some ad exchange businesses are hosted there. I've learned a lot from this: from now on our TOS and a checkbox at signup will make it clear our clients are NOT getting redundant hosting unless they order it. Some will still blame us, but you can't please every bargain hunter.
posted by: Jeff Lasman | June 02, 2008 01:17PM
I feel that c0bra needs to take a reality check and assess how he operates his business. If he is telling the truth, I think he needs to invest in a guru that will bring his site(s) up on multiple servers in seperate data centers running redundant round-robin DNS. But then again, he's probably full of crap.
posted by: ThePlanet Customer | June 02, 2008 01:33PM
c0bra guy is an idiot and the typical kid that runs off to WHT whenever their server dies for X reason and says the same crap. What kind of person would run their entire operation out of 1 DC. He claims to have his operations running on multiple servers and has backups blah blah blah but he keeps them all in the same DC on the same floor in the same state. If he truly is making the kind of rev's he is claiming then it would be pocket change to run out of multiple sites across the US or world. I really feel sorry for his clients as he must have a great scam going on them to allow this type of very poor design to host their ad campaigns and I would hope at least one of them will read this or his WHT posts and call him on his BS claims.
10K deal not a 10M deal should be setup this way.
posted by: Very Old TP Client | June 02, 2008 07:55PM
I guess next time people go spouting off about loosing millions of dollars they will think twice, your complaint could end up on WHIR for everyone to see what a fool you are.
posted by: Loyal Reader | June 02, 2008 08:06PM
This suck bad, what the hell happened, we lost the db on www.outaouaisweb.com Hurry and fix this please ... a.s.a.p.
posted by: XploreR | June 02, 2008 09:31PM
i have 2 servers hosted with them that have been down since this started saturday. one came up for about an hour or so this morning as some mail trickled in, then back down again. with loss of sales i hope the compensation offered is decent...
posted by: primo | June 02, 2008 09:45PM
On s'en Calice
merci
bonsoir !
posted by: Jee | June 03, 2008 02:53AM
Il y a-t-il quelqu'un qui peut traduire en francais un résumé et quand ca va être arrangé?
posted by: Martin | June 03, 2008 07:11AM
Do we have a time-frame on when this unfortunate problem will be resolved?
Thank you......
posted by: B.C...:) | June 03, 2008 07:43AM
Une partie du cablage electrique des serveurs a été detruit par une explosion... Les techniciens travaillent 24h/24h pour remettre en ligne leur serveurs. Presque tout les serveurs du 2ieme étage ont été remis en ligne et ils travaillent maintenant sur ceux du premier étage. Il demandent d'etre patient jusqu'a ce que tout soit rentré dans l'ordre... C'est a peu pres ce que ca dit.
posted by: TheGearHead | June 03, 2008 07:46AM
Pis c'est quand que ça va revenir dans l'Ordre? Parce que moi je commence à être pas mal tannée!
posted by: Stéphanie.. | June 03, 2008 07:59AM
C'est arrivé samedi a 4h55... Ca peux encore prendre un petit bout! Je peux essayer de les contacter pour avoir un update sur la situation et je l'écrirai ici...
posted by: TheGearHead | June 03, 2008 08:04AM
C'est arrivé samedi a 4h55... Ca peux encore prendre un petit bout! Je peux essayer de les contacter pour avoir un update sur la situation et je l'écrirai ici...
posted by: TheGearHead | June 03, 2008 08:18AM
Je viens tout juste de leur ecrire un mail... Je vous tiens au courant...
posted by: TheGearHead | June 03, 2008 08:29AM
Does anyone knows the estimated time left for a 100% server recovery? Can it possibly take 3 more days?
posted by: Patrick26 | June 03, 2008 10:15AM
http://forums.theplanet.com/index.php?s=999beeb8acce402a0790e9dd08fba95f&showtopic=90185&st=40
posted by: | June 03, 2008 10:25AM
I am a small time operation running 1 quad core box and they are basicly ignoring my issues at the planet. My ticket got closed saying my box has power. That is fine and dandy, but everything is inaccessable. They are ignoring the little guys here.
posted by: Chris | June 03, 2008 11:18AM
It's just torture being down, then up, then down again, then up again, then down again...I have to question whether or not the place is salvagable, or they should just start hauling the servers elsewhere...
posted by: RickyT | June 03, 2008 11:49AM
They are trying to fix the problem. The time they take to answer everyones little question is time that they don't put on the repartation. If everyone ask questions like you do, they will keep answering and not reparing.
Be patiente, you are not the only one waiting.
posted by: Sandra | June 03, 2008 11:52AM
Sandra : To be honest with you, I don't think the technicians nor the ingenieurs are the only doing the clicks and pressing the spacebar.
posted by: Jean-Pierre | June 03, 2008 12:04PM
"with loss of sales i hope the compensation offered is decent...posted by: primo" -- Oh, this is another good one. "I lost SALES and REVENUE! You owe me a million dollars!" Good luck with that.
posted by: david fink | June 03, 2008 02:49PM
Got to love this beautiful world of technology. To many people whining.
posted by: Litrum | June 03, 2008 03:04PM
Given all the response to the quote from the person called C0bra, I thought I'd point out down here also that we added a note to the story about a previous WHT post where that user claimed to have no servers at The Planet. Seems like they might just be somebody who enjoys piling on.
posted by: Liam Eagle | June 03, 2008 03:38PM
I think that the planet did a great job posting updates to their forum through the day. I checked in also via phone to see what all was going on, they did a good job IMO.
We have most of our servers at the Dallas NOC but our backups run from their Houston H1 DataCenter.
I know that the "frequent updates" didn't add a whole lot of comfort to those that were affected, BUT it could have been a lot worse if they were completely shut out as to what was happening.
Bottom line, The Planet is a good company and I don't think that they could have done a whole lot else to keep everyone up to date except maybe do "twitters" which are kind of handy...
posted by: Cotton Rohrscheib | June 03, 2008 04:43PM
Ditto to the "ignoring the little guy" comment above - I'm still down, & the tech support guy on the phone said no end in site. They are working reboot tickets in the order received. I didn't open a ticket until Tuesday - figured what's the point, they are aware of the problem, right? Kind of restating the obvious... Now I am still down - 300 sites offline - with supposedly 9000 tickets ahead of me. They are just too big and too few people to physically push all the buttons.
posted by: mocroft | June 05, 2008 01:07PM
[POST COMMENT]
