Many areas of NYC have experienced flooding, those in the green, and yellow shaded parts of this Google map, specifically, and residents in the dark yellow corridors have been forced to evacuate due to Hurricane Sandy floods

Noise Filter: Hurricane Sandy Floods NYC Data Center, Impacts Hosts, Colocation Providers

1 comment

More than 7 million people are without power in the US on Tuesday, and New York City has been hit with massive flooding, and fires in the wake of Hurricane Sandy.

New York is home to many web hosts and data center providers, including Internap, whose 75 Broad Street building experienced sub-basement flooding as a result of the heavy rains on Monday.

In a notice to customers posted on Pastebin on Monday, Matt Price, director, hosting operations and support, Internap, said the flooding had destroyed the site’s diesel pumps, preventing fuel from being pumped to the generators.

The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now.

According to a blog post by Internap customer and live chat software provider Zopim, Internap was up and running on Tuesday morning. Zopim apologized to its customers, many of which are web hosting providers, for “the way the situation was dealt with.”

In the event of scheduled downtime, we actually have contingency plans to keep the service running. Typically, our data center has available fuel reserves to go on for at least 5 more hours without power, giving us sufficient time to react. However, this time round the flooding caused by Hurricane Sandy has submerged and destroyed the fuel pumps, rendering the fuel reserves useless. All services affected by the power outage were suddenly taken offline without notice. There was nothing we could do about this.

PEER 1 has used its forum to keep customers aware of the state of its data center at 75 Broad which until early Tuesday morning was running on emergency generator power. As of 4:20 am, it only had four hours for the fuel left on its generators, and by 5:30 am had made the decision to shut down its New York data center.

We are going to implement a controlled shutdown of NY data center at 10:45 ET. Customer communications is being prepped.

Manhattan web host and ISP DataGram was hit hard by flooding, which took down customer sites including Huffington Post, Gawker and Gizmodo. A Tumblr post by Buzzfeed, another affected Datagram customer, says some of its site is accessible thanks to its CDN.

Datagram has lost power, an official there told us via text this evening. “Basement flooded, fuel pump off line – we got people working on it now. 5 feet of water now,” the official wrote.

Elements of BuzzFeed’s site and many story pages are back online, thanks to a content delivery network, Akamai, which hosts the content at servers distributed around the world.

On web hosting forum WebHostingTalk, members reported multiple network and data center outages including Cogent, InterServer, HoneLive, ColoCrossing, XO Communications, Atlantic Metro and SteadFast. SteadFast said in a post on its website that it has no ETA for service restoration.

We have just been informed that due to an auxiliary electrical failure, power systems have failed at our nyc16 facility at 121 Varick St in New York City.  All New York servers and services are now offline and we have no ETA for service restoration.  We will provide updates as we have them.

Most systems should be configured to power on when power service resumes and we can assist via ticket as needed when power becomes available. We will, of course, honor full SLA credits for the period(s) in which power has been unavailable in the facility. We apologize for any inconvenience this has caused you or your business.

Services like CDN or disaster recovery are extremely important as entire data centers have been shut down due to the storm. Web hosting customers should have geographically dispersed data in the event that their primary data center is impacted.

The full damage of Hurricane Sandy is still unknown, but a report by CBC News says that storm damage is already projected to cost anywhere between $10 billion to $20 billion US.

Talk back: How were you impacted by Sandy? Did you notice any disruptions to your services coming out of NYC data centers? Let us know in a comment.

Newsletters

Subscribe Now and Get Our Exclusive Report on "The Hosting Infrastructure Ecosystem"

Enter your email to receive messages about offerings by Penton, its brands, affiliates and/or third-party partners, consistent with Penton's Privacy Policy.

Related Forum Threads

Add Your Comments

  • (will not be published)

One Comment

  1. In response to your request for updates from NYC after the storm, we do have this statement from our client, TELEHOUSE America (www.telehouse.com) that was just released a short while ago: Since Friday, 10/26, TELEHOUSE New York initiated plans to sustain operational readiness should Hurricane Sandy strike the greater New York City area. Since Sunday, 10/28, all three TELEHOUSE New York City facilities (25B, Chelsea and Teleport) have remained fully fueled and staffed, with extra Security and Engineering personnel. Last night, 10/29, between 7PM and 8PM, all TELEHOUSE New York City facilities proactively engaged backup diesel power (off-grid) in anticipation of lower Manhattan’s utility power shutdown. Since that time, these facilities have remained operational without any critical system (power, cooling) outages. These facilities will remain fully staffed and off-grid (independent power generation mode), with periodic refueling until utility power is restored and stable. During this period, both Manhattan sites (25B, Chelsea) were operationally unaffected by local flooding. Building engineers and equipment, at both Manhattan facilities, are engaged in mitigating this threat. The Teleport facility, on Staten Island, had no flood threat during this time. All TELEHOUSE NYIIX extensions have remained operational without any outages. Apparently some 111 8th Ave. NYIIX Customers did experience their own power problems there. Please email our NYIIX support team at, iieg@telehouse.com, for the latest status or questions. TELEHOUSE has received reports of sporadic Carrier/ISP circuit outages, which have already been restored or corrective actions underway. If you are experiencing any problems, please open a trouble ticket with TELEHOUSE America’s Customer Assistance Center at 718 355 2525, or via email, cac@telehouse.com. We will be happy to help you obtain responses from Telehouse for any questions or comments you may have. Please email us at jsa_telehouse@jaymiescotto.com. Thank you!

    Reply