Colocation provider Internap has restored power and connectivity in its lower Manhattan data center on Wednesday, after Hurricane Sandy knocked it offline on Monday night.
Heavy rains, treacherous winds and swelling seawater spelled disaster for the 75 Broad St. building when flooding destroyed the site’s diesel pumps late Monday night, preventing fuel from being pumped up to the generators.
In a blog post, Internap says its customers at 75 Broad Street “should be up and operational at this time, working as expected on generator power. We have over 40 hours of fuel onsite with a second truck waiting to refuel the current tank as necessary.”
“At our 111 8th data center, we experienced an outage last night due to building-fed fuel system malfunctioning. When the issue occurred, the fuel pumps could not provide diesel to the rooftop generators, causing them to stop supplying power to our UPS system. Once battery backup was exhausted, our infrastructure lost power,” Steve Orchard, senior vice president, operations and support, Internap Network Services said in the blog post. “The incident caused a loss of IP connectivity for several service points until power to our P-NAP was restored. At this point, we have connectivity restored for all customers and power restored to the majority of our data center customers. We continue to work with vendors to bring the entire site back online.”
According to a post on Data Center Knowledge, web hosts PEER 1, SquareSpace and Fog Creek Software are using their own hands and ingenuity to fix the fuel problem at 75 Broad St. About 25 employees, friends and customers are carrying 5-gallon buckets of diesel fuel up 17 floors to refuel the generator — a short-term solution, but an admirable one.
SquareSpace calls the current situation is untenable since it can’t keep manpower going for 24/7 for days, and it is currently “in a wait-and-see posture” as it waits for an alternate pump.
“The basement is not draining at all either, despite the large pumps that were brought in late last night. DEP and ConEd have been here for a few hours. They fear a water main has ruptured somewhere and is pushing water (and other stuff) into our basement as we pump it out. This is pure speculation at this point,” according to a Squarespace status blog post on Wednesday afternoon.
Talk back: What do you think of the efforts made by these hosting providers to keep their customers sites operational? How would you deal with the situation if it was your data center? Let us know in a comment.