Hostway's Enright Speaks on Server Outages

As you may already be aware, Hostway suffered a series of hardware failures that turned a planned Monday migration of servers into a serious downtime issue this week. A large number of dedicated server customers acquired in the company’s purchase of Affinity Internet were offline well in excess of the planned 12 to 15 hours – some of them for days, some still offline as of right now.

Hostway’s vice president of marketing and business development, John Enright, understandably a busy man at the moment, took a few minutes to talk to me today and offered some insight into the situation.

Enright says the move brought 4,000 servers – representing about 3,000 individual customers – from Miami to Tampa.

“Our goal in moving the customers from one facility to another was to improve the quality of service in the long run due to the fact that our Tampa facility has better connectivity and more capacity.”

Of course, the irony of a protracted outage in that case was not lost on Enright, who seemed genuinely contrite and took pains to impress upon me that Hostway’s resources, including its data center and executive staff, are all focused on getting servers back online at the moment.

“The majority of customers and the majority of servers were actually back online within that time frame. However, for reasons that we have yet to determine, there were an abnormally high amount of hardware failures among the servers that we migrated. I don’t have a definitive number, but it was approximately 500, give or take 100 servers, that actually experienced the hardware failure.”

He says the company anticipated some hardware failure, a predictable result of moving any server, or just turning it off and back on. Hostway had spare parts waiting at the Tampa facility. But the number of hardware failures exceeded the company’s most pessimistic estimates.

“We’ve been working around the clock since then to get all of those customers back up and running. We have additional staff in place in our Tampa data center, and in our call center to respond to customer inquiries. At this point we’re down to about 50 servers that are still affected by a hardware outage of some form or another. We’re diligently and feverishly working through those to bring every customer back online.”

Enright says Hostway is compensating affected customers on an individual basis with credits, and reviewing its next steps. But the company is understandably focused on getting things back online at the moment.

“The priority here is to make sure every customer is back up online and running. That’s 100 percent of our focus. And as soon as we get to that state where every customer is back up, we’re going to review what our next steps are.”

Hostway is obviously hoping that state arrives sooner than later, but Enright couldn’t offer a firm out-of-the-woods ETA.

“As we get down to the last few customers, the hardware problems are getting a little bit more complex. In some cases it’s more than one component that has failed. At the rate that we’re going, we do expect everybody to be online shortly.”

Customers have complained of being unable to contact the company when the outage first occurred, which is usually the case with a major outage. It is unlikely that Hostway has the capacity to handle 500 customer calls at once. But Enright says the company is reaching out to each customer as it is able, and providing them with the status of their servers.

“Right after the migration took place, communication was a challenge. And we did have an enormous amount of phone calls coming into the call center. We added additional staff to both the call center and the data center immediately. And since then, service levels have improved. They’re still nowhere near what we would consider acceptable in an ordinary case, but customers are able to reach us of an average wait time of 20 minutes, give or take.

“We acknowledge that communication was a challenge in the first part of the outage, but we’re doing everything we can to rectify that by reaching out to each and every customer.”