November 12, 2007 -- (WEB HOST INDUSTRY REVIEW) -- Last week, theWHIR reported on an extended outage that at times affected as many as 165,000 sites hosted by Web hosting provider NaviSite (navisite.com) - a situation that set off a significant storm of customer complaints on hosting message boards, blogs and in the comment sections of news posts.
Level 1 PCI DSS Certified Service Provider! DataPipe delivers the best network & support; top tier data centers; New York metro, Silicon Valley, London, Hong Kong, Shanghai. DataPipe - Personal Touch, Global Reach.
The outage was caused by unexpected difficulties in a planned migration of Alabanza customers acquired by NaviSite to new systems. A planned 12 hours of downtime that began Saturday, November 3 stretched into several days, then almost a week. By Thursday afternoon, NaviSite issued a statement saying it had 90 percent of the equipment back online and expected everything to be up on Friday.
On Friday afternoon, the company posted a notice "from the desk of CEO Arthur Becker" on its Website, thanking customers for their forbearance, patience and understanding.
"I understand that this has affected your business and your customers," says Becker, "and I apologize for the inconvenience and disruption. This is not the standard for NaviSite. While we had planned this migration for months, we did not anticipate a number of scenarios that became major issues during the execution of the migration. Despite the fact that we have acquired a number of companies during the past few years and both consolidated data centers and migrated data, we had never encountered the series of problems that we saw earlier this week."
As of Monday, a few complaints were still trickling in from customers reporting that their servers were offline.
According to Rathin Sinha, senior vice president and chief marketing officer at NaviSite, the migration was officially completed early Friday evening, and any downtime a customer may have encountered since then would instead be considered a matter of normal maintenance and dealt with in the normal course of support.
"There are some customers whose service might have gone down or up," says Sinha, "but those are normal production or maintenance related issues, like rebooting the servers. So what we have done at the conclusion of the migration is we are treating these issues as our normal support issues."
Although NaviSite has undertaken other such integration and migration activities in the past, says Sinha, three separate issues arose during the Alabanza migration that combined to slow the process of bringing customers back online so significantly.
"There were really three unanticipated issues" he says. "One was - as we planned to do it by bringing the data over a dedicated line - we had data transfer issues. It was much slower than anticipated. Second was a major issue around the nameserver, where we could bring up the IP address, but the URL was not having a one-to-one relationship with the IP address. And the third problem was the ARP request, which came when many IP addresses were back live. A lot of people got into the network and there was slowness."
Troubling for NaviSite is the fact that the downtime that came with the migration has overshadowed the fact that the company set out to move the Alabanza customers on to a better platform.
"We made investments on creating a better infrastructure," says Sinha. "The goal was to move all of the Alabanza customers to a state of the art infrastructure with Sun machines and virtualization from VMware. This was really a migration for something better. Obviously during the process, the outage became much longer. And obviously many customers were distressed. It was a major disruption for them and it created a major inconvenience of them. And we got many, many calls."
According to Sinha, NaviSite feels it did the best it could with a bad situation, with all the company's technical resources working round the clock to get equipment online and creating extra support lines to handle the greatly increased volume of customer calls. He says the company issued regular updates via its Web site, though the messages have since been replaced by the CEO's, and NaviSite says it doesn't have a file of the messages it can provide at this time.
There are certainly some customers who feel the company didn't handle the situation well. Message board, blog and comment posts from as late as Monday, November 12 suggest that at least some NaviSite customers were still waiting for their sites to come back, and for an explanation from the company.
Sinha, however, paints a rosier picture of customers response to the outage.
"In fact," he says, "a lot of customers - and I have quite a few examples here - when their services came back, they thanked us for working around the clock and thanked the support personnel and the people who took their phone calls. They complimented them in every possible way."
NaviSite seems content to consider the online outcry the work of a vocal minority of customers, though the sheer volume of activity on message boards (185 posts on a Webhostingtalk thread at last count) would, at least by that measure, make the incident one of the more talked-about outages in recent Web hosting history.
However it chooses to characterize the criticisms, NaviSite's next step is the same. Sinha says the company is looking at its terms for issuing amends to customers whose guaranteed services levels weren't met. He says the mode of calculating compensation is in almost every case a condition of the service contract.
"Now we're definitely looking at what are the appropriate things to do," he says, "in terms of measuring the inconvenience and the disruption and quantifying that and seeing what customers really need. We definitely will do the appropriate things to serve these customers."
It may prove difficult at the moment to measure the precise impact of the outage on NaviSite's business, but there will most certainly be a monetary cost in the form of compensation and, it is more than likely, a loss of customers.
I think that everyone was grateful for the efforts of the Navisite employees fix all the problems (let's not spin that into customer satisfaction with the outage!). Something is better than nothing when clients are screaming to see their sites. Still, the line from management prior to the move was cocky and dismissive of may of the concerns laid out before hand from Alabanza employees and customers. A majority of servers are still NOT on the new platform and further migrations are still needed. After 5 failed attempts, Navisite would have been wise to reevaluate their plan. The sad truth is that much of this could have been avoided. As for the loss to customers, after building their business from scratch over years, many are out of business in 7 days thanks to this huge blunder. posted by: nobody special | November 13, 2007 01:51PM
This is spin, and bull.
They claim they had 300 clients move. There are 508 registered users on an Alabanza Board, and no one is complimenting Navisite.
This is corporate lies and spin at it's finest - they promised that those clients would have an explanation.
The press got it first. Despicable people. posted by: Ex-Navisite Client | November 13, 2007 02:04PM
This gentleman (Sinha), knows something of webhosting or he is a simple salesperson, or talkative of stupidities. posted by: ex-idiot | November 13, 2007 02:20PM
To anyone who is reading this, and thinks that any of NaviSite's customers are satisfied as of 11/13/07 with what has transpired over the past week, you are sadly mistaken. As of today, clients can not contact support, are having frequent network outages, due to multiple Sun VMWARE blades crashing regulary, not to mention the damage to their individual businesses and reputations after a seven (7) day outage.
Much of what NaviSite has spewed to news organisations, are blatant lies, and they are still scrambling to stablise their network, even today.
It remains to be seen, how many of their clients will be around this time next week. posted by: Another NaviSite Victim | November 13, 2007 02:24PM
NaviSite was incompetent at every stage of this move, and their attempts to paint a rosey picture in the aftermath doesn't help the hundreds of their customers whose livelihoods were destroyed by their ineptitude. Personally, I lost business in this mess, but others suffered much greater losses. I hope that this compensation of which they speak will provide for the businesses they have destroyed. posted by: Jack | November 13, 2007 02:30PM
Who wouldn't be thankful when their servers have been down for 9 days.. came back up? Being thankful at the end of frustration doesn't mean the frustration didn't exist.
They can lie to the media all they want, but the more they lie, the more their clients will see them as liars. posted by: ARP | November 13, 2007 03:01PM
I havee lost customers that have been with me for 5 and 6 years due to this fiasco. Calls and emails to Navisite Management go unanswered. It took 6 days for my server to be back online. It took an additional 6 days for a support ticket to be worked. While their network may run faster when it is online, if you don't have any customers left it really doesn't make any difference. They are no Alabanza for sure. As far as I'm concerned, they need to let us know what they intend to do this week to reimburse us for our losses and expenses or all I can see in the future is legal actions. It's really sad it has to come to this. posted by: yorweb | November 14, 2007 10:59AM
We were not informed of any take over. We had our sites hosted at Linux Webhost for years and there was no talk about any down time, take over or even what the name of the new company was. Contacting the old company did nothing as everything was shut down. If it wasn't for finding out via forums or via Google news we would still be lost and wondering. This is not a good first impression of this company and already I have clients that want to move hosts. Their service was not good, not informative, and utterly useless! posted by: G. | November 14, 2007 12:02PM
Exuses are for dumb people (or dumb companies). We had a server offline not just the migration process but for a whole week! NaviSite is not compensating clients. I would advise anyone to stay off from NaviSite/Alabanza from now on.
posted by: Charles | November 19, 2007 11:12PM