Hosting Forums Offer Info During Alabanza Blackout

by Jay Lyman

September 23, 2004 -- (WEB HOST INDUSTRY REVIEW) -- Web hosts and resellers all over the world were hungry for information when hosting wholesaler and automater Alabanza suffered a loss of power after a fire near its data center in Baltimore, Maryland this week.

With servers and sites down and a VOIP phone support system, Web hosting forums turned out to be the best place to find out about the outage.

Alabanza took to the forums as well, informing users of the reason for the downtime -- a sub-street fire that was not actually in Alabanza's facilities had nonetheless prompted fire and electric officials to cut power to the entire building -- and signaling that reserve power was on the way.

But word from an Alabanza systems admin that the major hosting provider was "pulling in an emergency generator to power up the NOC" was not necessarily good news for some, who posted complaints on unofficial Alabanza forum Alahosts.com (alahosts.com), saying, "We pay enough -- an emergency generator should already be up and running!"

A lengthy discussion of both the incident and the company on Web Hosting Talk (webhostingtalk.com), contained both harsh criticisms of Alabanza and understanding that official, binding orders to cut power are both common in fires and out of Alabanza's control.

According to a company correspondence posted by an Alabanza customer, the actual order apparently came from Baltimore's electric company, which shut down all power to the building to repair damages from the fire.

"BG&E had at one point brought power back up, however it was greatly fluctuating," said the email from Alabanza. "Our Sys team made the decision to not bring the servers back up until the power surges stopped in order to protect our client's harward [sic] and data. The power has been stablized [sic] and all servers are coming online at this time."

Despite the eventual return of servers and service, it appears the downtime - logged in fearful, frustrated, and refund-themed posts throughout the forums - may have led some Alabanza customers to seek alternative partners.

It may not have helped the company that a similar incident had similar effects one year ago, when fire was again the culprit in what amounted to downtime and lost dollars for many of Alabanza's hosts.

Some posters resented a second outage with similar circumstances, arguing it was something Alabanza said it was planning to avoid.

"I clearly remember an almos [sic] identical scenario months ago when there was a fire in the basement of the building where the Data Center is located and a very formal email afterwards in which Alabanza laid out the plan for preventing it from happening again..." said one poster on the Alahosts.com thread.

In a separate letter to clients, posted to the Web Hosting Talk thread, Alabanza Chairman and CEO Thomas Cunningham wrote that the fire affected the power switching equipment within his company's building, not the power source. All main and auxiliary power was mandated off by the fire department, he said, adding that Alabanza's disaster recovery plan was triggered and "worked nearly flawlessly."

"The only thing that went wrong in the plan is the alabanza.info (alabanza.info) site which is supposed to be giving you constant updates during a disaster is being overhauled and wasn't available," Cunningham wrote, referring to a "system monitor" that indicated both power and network were both fine during the downtime.

There was little disagreement on the significance of the outage, which lasted more than 10 hours for some and, according to the Web Hosting Talk thread, might have impacted nearly half a million sites.

"If the entire data center in Baltimore is down -- nearly a million websites could be without service," said one post.

Despite complaints that the busy phone lines and site outage were leaving Alabanza customers in the dark, there were also compliments for the forums and the information posted there by both customers and Alabanza itself, which included an emailed apology from Cunningham, who conceded the company's reliance on a redundant power grid with three stations.

"The idea as it was presented to me was quite practical -- instead of having every company in the downtown area have its own generator, why not share three power sources on a redundant grid. It's more practical and it's better for the environment. It didn't only make sense to me -- Qwest, ATT, and MFS all depend on the same redundant system," Cunningham wrote. "It's my fault I was enamored with the technology. I believed the engineers that told me to depend on their system."

Alabanza, which did not respond for comment, indicated it was already working on bringing up two separate power feeds from the triple-generator source it sits on in case the one internal feed fails again. The company said with the generator brought in for this week's outage, it now has four total power generators -- three on the redundant grid with two separate feeds into the building and the fourth being installed in the data center, on the second power feed, Alabanza said.