More than 7 million people are without power in the US on Tuesday, and New York City has been hit with massive flooding, and fires in the wake of Hurricane Sandy.
New York is home to many web hosts and data center providers, including Internap, whose 75 Broad Street building experienced sub-basement flooding as a result of the heavy rains on Monday.
In a notice to customers posted on Pastebin on Monday, Matt Price, director, hosting operations and support, Internap, said the flooding had destroyed the site’s diesel pumps, preventing fuel from being pumped to the generators.
The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now.
According to a blog post by Internap customer and live chat software provider Zopim, Internap was up and running on Tuesday morning. Zopim apologized to its customers, many of which are web hosting providers, for “the way the situation was dealt with.”
In the event of scheduled downtime, we actually have contingency plans to keep the service running. Typically, our data center has available fuel reserves to go on for at least 5 more hours without power, giving us sufficient time to react. However, this time round the flooding caused by Hurricane Sandy has submerged and destroyed the fuel pumps, rendering the fuel reserves useless. All services affected by the power outage were suddenly taken offline without notice. There was nothing we could do about this.
PEER 1 has used its forum to keep customers aware of the state of its data center at 75 Broad which until early Tuesday morning was running on emergency generator power. As of 4:20 am, it only had four hours for the fuel left on its generators, and by 5:30 am had made the decision to shut down its New York data center.
We are going to implement a controlled shutdown of NY data center at 10:45 ET. Customer communications is being prepped.
Manhattan web host and ISP DataGram was hit hard by flooding, which took down customer sites including Huffington Post, Gawker and Gizmodo. A Tumblr post by Buzzfeed, another affected Datagram customer, says some of its site is accessible thanks to its CDN.
Datagram has lost power, an official there told us via text this evening. “Basement flooded, fuel pump off line – we got people working on it now. 5 feet of water now,” the official wrote.
Elements of BuzzFeed’s site and many story pages are back online, thanks to a content delivery network, Akamai, which hosts the content at servers distributed around the world.
On web hosting forum WebHostingTalk, members reported multiple network and data center outages including Cogent, InterServer, HoneLive, ColoCrossing, XO Communications, Atlantic Metro and SteadFast. SteadFast said in a post on its website that it has no ETA for service restoration.
We have just been informed that due to an auxiliary electrical failure, power systems have failed at our nyc16 facility at 121 Varick St in New York City. All New York servers and services are now offline and we have no ETA for service restoration. We will provide updates as we have them.
Most systems should be configured to power on when power service resumes and we can assist via ticket as needed when power becomes available. We will, of course, honor full SLA credits for the period(s) in which power has been unavailable in the facility. We apologize for any inconvenience this has caused you or your business.
Services like CDN or disaster recovery are extremely important as entire data centers have been shut down due to the storm. Web hosting customers should have geographically dispersed data in the event that their primary data center is impacted.
The full damage of Hurricane Sandy is still unknown, but a report by CBC News says that storm damage is already projected to cost anywhere between $10 billion to $20 billion US.
Talk back: How were you impacted by Sandy? Did you notice any disruptions to your services coming out of NYC data centers? Let us know in a comment.