By now you've probably read a thing or two about the typing error at DreamHost that caused the company to accidentally bill customers for roughly $7.5 million earlier this week.

DreamHost, characteristically, was quick to own up to the mistake, and let customers know what happened, through its blog. Of course, with a mistake of this magnitude there's really no alternative to the honest-apology approach. I can only imagine that trying to keep something like this from getting out would at best cost the company its customers' trust, and at worst make the whole incident indistinguishable from credit card fraud.

But I wonder how much damage DreamHost might have done to its reputation with this particular gaffe, and more specifically with the form and tone of its apology.

This is particularly interesting to me because of DreamHost's successful past work in the areas of apologies and their tones. I've held up DreamHost in the past as an example of how to go about apologizing for mistakes. After a 2005 outage, it was precisely the way the company blogged about the outage that earned it, in many cases, the appreciation and respect of its customers.

(We ran a story about that situation and its outcome here)

This time, though, the company might have taken the wrong tone in its apology.

It's a tricky thing, being glib. Particularly in the context of an apology. And there's a world of difference between informal/forthright, and jokey and (as I've seen it described in several places) condescending.

Specifically (and I'll keep this brief, since this point has already been made elsewhere) the offenses here are: the title "Um, Whoops" and tone ("Ha, the joke is on you! I guess. Um, okay, no, not really, I'm sorry."), as well as the picture of Homer Simpson accompanying the post seem destined to be taken badly by customers who are understandably upset to find their bank accounts missing considerable sums of money.

This time, it seems, DreamHost's personal and transparent mode of interacting with customers appears to have misfired.

Customer reaction, even the reaction revealed in the blog's comment section, served to illustrate the affection DreamHost's customers had for the company and its methods following its previous problems. The more than 600 comments on Monday's apology post paint a different picture. While not every comment is negative, and certainly some are supportive and appreciative, the usual DreamHost feel-good atmosphere is decidedly absent. And a good portion of the posts are of the "Jokes are NOT APPROPRIATE in this situation" variety.

It remains to be seen just how significant the impact of this error is on DreamHost's business, and how many of its customers actually set off in search of another service provider. But it's certainly a change for me to be pointing out DreamHost as an example of a company doing a bad job of handling a volatile situation.

I can't even think, off the top of my head, of who I'm going to use next time I need an example of "here's how you should have done it" from a customer service standpoint. Will I really have to look outside the Web hosting business.

I hate to be glib myself, given the subject matter, but is it really that hard to get this right? Customers appreciate your candor. But they're not your buddies. Especially when you're messing with their money.

[Note: and here we're just talking about the effectiveness of the apology. Of course, there's the entirely separate issue of just how many major catastrophes necessitating large-scale public apologies are acceptable from a single service provider, and in what span of time.]

To someone who covers Web hosting news for a living, the of-the-minute issue of last week was the level of customer outrage stirring around the incidents at C I Host (a break-in and theft) and NaviSite (a week-long server outage).

Though the incidents themselves were very different, their impact on each company's relationship with its customers was nearly identical - in a nutshell "my website is down and I want it back up."

Outrage is an interesting thing. It sometimes seems that a certain percentage of people are inclined to be outraged about just about anything. And I have no doubt that a certain percentage of the customers a Web host deals with in a given day is going to be angry, regardless of how good the company's service is.

This hardly falls into that category, however. In each of these cases, messageboards, blogs and - most interestingly, from a publishing-a-website perspective - the comment sections of the stories we've run on both issues have been spilling over with customers venting their frustration and recounting their own personal tales of lost service, lost business or unresponsive hosts.

And in both cases, the complaints seem to have a lot to do with the confusion surrounding the problems. Customers of both companies - in some cases, people whose sites had been offline for as much as a week - seem most concerned with the fact that they're not getting the kind of explanation they want from their service providers.

There are two sides to this kind of situation (or maybe there are more than two, but lets say for the sake of the next couple paragraphs that there are two).

On the one hand, obviously neither of these companies wanted their services to go down. Obviously, both companies want to get those customers back online as fast as possible. I can only imagine that in both cases, the company threw everything it had at the problem of getting things back online (we can probably all agree - priority number one for a Web host).

And yes, that's priority number one for customers too. But customers also want to know they're being treated honestly.

The other side of these situations is that hosts too often seem to want to cover up or downplay the mistakes they might have made. I wouldn't implicate any host in particular here. It seems that almost every time there's an outage, customers report having to fight to get the cause of the disruption out of their hosts. Sometimes they point to explanations they believe to be outright lies. Those claims may or may not be true, but they illustrate the point - confusion is a problem.

A major outage isn't exactly a find-the-silver-lining sort of situation. You'd probably spend a lot of time looking for it and not come up with much. But a problem is always, at least, an opportunity for a company to demonstrate to customers that it's ready to handle a problem.

In a conversation with theWHIR, C I Host's chief corporate counsel James Eckles said "we're just as victimized as our customers."

NaviSite's chief marketing officer Rathin Sinha said "If we look at this issue as something where unanticipated things happened and the company did everything it could to resolve that and restore services, I think that is where the focus ought to be. And that's where most of our customers focused."

Neither of these is an outright offensive point of view, or blatant buck-passing per se, but I'd bet neither one is exactly what their customers are waiting to hear.

A little free PR advice from somebody who sees a lot of PR: if it sounds like spin, it's not very good spin.

Better than any spin, in my opinion, is transparency.

A great example of that is the major outage DreamHost suffered about 16 months ago (you can read our article about it here). The company used its blog to keep customers appraised of the situation at every turn, and assure them that it was doing everything it could to fix things. It left posts and comments up, creating a forum for customers to discuss things. And it took responsibility for the mistake.

Perhaps surprisingly, the majority of customers who offered up opinions on the incident were understanding and even grateful.

At the time, Seth Godin summed it up really well by writing:

"Lesson one: when things get messed up, being clear, self-critical and apologetic is really the only way to deal with customers if you expect them to give you another chance."

It's important to understand that if you fail to deliver on your services, you could end up backing up your relationships with customers to the point where you're basically selling them on your services again. A good start would be giving them a reason to trust you.

This past Sunday and Monday, separate incidents at Rackspace caused significant outages. Now, Rackspace is quite a bit bigger than DreamHost, but it took much the same tack, keeping customers appraised of its repair process, and most importantly, remaining accountable.

In one of its posts (which are still up on the company's site), the company said another thing I'd like to borrow to help make my point:

"We cannot promise that hardware won't break, that software won't fail or that we will always be perfect. What we can promise is that if something goes wrong we will rise to the occasion, take action, resolve the issue and accept responsibility. If you are a Rackspace customer and don't think we've lived up to this promise at anytime during the outage, please let your Account Manager know."

I'm not really interested in making any distinctions between the quality of the services Rackspace provides and those provided by C I Host or NaviSite, or any other host for that matter.

But I do think that even though the company is living through a very comparable situation, I'd be surprised if Rackspace lost many customers over this outage.

An unusually accusing news item caught my attention today. Not that it’s particularly unusual to see an accusing news item, that is. But in this case, the content of the accusation was certainly out of the ordinary.

 

A news post on the (apparently French) website of Reporters Without Borders revealed that hosting giant Go Daddy had pulled down 10 sites dealing with human rights in China.

 

The story seems to suggest that Go Daddy failed to offer an explanation – a situation that prompted Reporters Without Borders to voice the perfectly reasonable opinion that the “situation of online free expression in China is difficult enough already, so foreign companies need to act with the utmost transparency.”

 

The article suggests that the takedowns could conceivably be the result of the Chinese government pressuring the company to take down sites that are blocked in China. This, also, is not a completely improbable notion, although it already seems like the least likely of a dozen possibilities.

 

The article cites as its source the account of a site called Boxun News (a site with which I am not personally well acquainted), which offers a report (attributed to “boxun”) on the interaction with Go Daddy that led to the shutdown.

 

I have my doubts about this story too. A less-than-fluent English rendering of a roughly-disguised firsthand account doesn’t strike me as journalism in the sense I’ve come to understand it. And it’s not precisely what I’d consider news either. But it’s out there. And, in evidence of the Internet’s playing-field-leveling effect, it is a sort of “issue” simply by virtue of its existence.

 

It’s probably only old hat if you’re a regular reader of news or forums that deal specifically with Web hosting (an assumption I’m not ready to make about the Reporters Without Borders folks), but Web hosts take down customer sites all the time, for all kinds of reasons. And one of the unifying characteristics of Web hosting customers is they make websites.

 

Rarely does a customer like having his or her site taken down, right or wrong. And quite often those jilted customers respond by, well, making a website.

 

It’s always a war of words. But very rarely, in my experience, does the Web hosting company pass up the opportunity to tell its side of the story. So I was curious, mostly, about the fact that Go Daddy hadn’t responded to the request from Reporters Without Borders.

 

According to Go Daddy (which got back to me almost immediately, it’s worth noting) that matter is in the process of being worked out right now.

 

Elizabeth Driscoll, VP of public relations says:

 

“A reporter posted a story on this topic today without discussing the accusations directly with Go Daddy to find out what happened. We are working with her now. While we can not reveal detail about the complaint, in accordance with our privacy policies, we can assure you it was not a complaint or request made by or on behalf of the Chinese government and has nothing to do with politics or content.”

 

Thankfully, it seems Go Daddy is not operating under the influence of the Chinese government at the moment. In fact, Driscoll also offered a more specific comment on the complaint that triggered the situation:

 

“Go Daddy received a complaint regarding the server in question and allegations of ‘attacks’ involving that server. Upon conclusion of our investigation, Go Daddy found the server was being used in a manner which seriously violates our terms of service.  We contacted the client and asked their site administrator to move from our servers.”

As you may already be aware, Hostway suffered a series of hardware failures that turned a planned Monday migration of servers into a serious downtime issue this week. A large number of dedicated server customers acquired in the company’s purchase of Affinity Internet were offline well in excess of the planned 12 to 15 hours – some of them for days, some still offline as of right now.

 

Hostway’s vice president of marketing and business development, John Enright, understandably a busy man at the moment, took a few minutes to talk to me today and offered some insight into the situation.

 

Enright says the move brought 4,000 servers – representing about 3,000 individual customers – from Miami to Tampa.

 

“Our goal in moving the customers from one facility to another was to improve the quality of service in the long run due to the fact that our Tampa facility has better connectivity and more capacity.”

 

Of course, the irony of a protracted outage in that case was not lost on Enright, who seemed genuinely contrite and took pains to impress upon me that Hostway’s resources, including its data center and executive staff, are all focused on getting servers back online at the moment.

 

“The majority of customers and the majority of servers were actually back online within that time frame. However, for reasons that we have yet to determine, there were an abnormally high amount of hardware failures among the servers that we migrated. I don’t have a definitive number, but it was approximately 500, give or take 100 servers, that actually experienced the hardware failure.”

 

He says the company anticipated some hardware failure, a predictable result of moving any server, or just turning it off and back on. Hostway had spare parts waiting at the Tampa facility. But the number of hardware failures exceeded the company’s most pessimistic estimates.

 

“We’ve been working around the clock since then to get all of those customers back up and running. We have additional staff in place in our Tampa data center, and in our call center to respond to customer inquiries. At this point we’re down to about 50 servers that are still affected by a hardware outage of some form or another. We’re diligently and feverishly working through those to bring every customer back online.”

 

Enright says Hostway is compensating affected customers on an individual basis with credits, and reviewing its next steps. But the company is understandably focused on getting things back online at the moment.

 

“The priority here is to make sure every customer is back up online and running. That’s 100 percent of our focus. And as soon as we get to that state where every customer is back up, we’re going to review what our next steps are.”

 

Hostway is obviously hoping that state arrives sooner than later, but Enright couldn’t offer a firm out-of-the-woods ETA.

 

“As we get down to the last few customers, the hardware problems are getting a little bit more complex. In some cases it’s more than one component that has failed. At the rate that we’re going, we do expect everybody to be online shortly.”

 

Customers have complained of being unable to contact the company when the outage first occurred, which is usually the case with a major outage. It is unlikely that Hostway has the capacity to handle 500 customer calls at once. But Enright says the company is reaching out to each customer as it is able, and providing them with the status of their servers.

 

“Right after the migration took place, communication was a challenge. And we did have an enormous amount of phone calls coming into the call center. We added additional staff to both the call center and the data center immediately. And since then, service levels have improved. They’re still nowhere near what we would consider acceptable in an ordinary case, but customers are able to reach us of an average wait time of 20 minutes, give or take.

 

“We acknowledge that communication was a challenge in the first part of the outage, but we’re doing everything we can to rectify that by reaching out to each and every customer.”

Here’s something to be upset about.

 

We’ve mentioned it before, but it popped up in the news again this week, in The Globe and Mail and ABC News.

 

I won’t go on at great length describing what the articles I linked already have, but there appears to be a trend of self-described pedophiles operating websites that, while not technically illegal, cross all kinds of ethical boundaries by actually providing a resource for pedophiles.

 

Parents, Internet citizens, and generally anyone with an intact sense of right-and-wrong is understandably outraged at the existence of this type of site. But fighting these sites seems to be more difficult than it ought to be.

 

The site mentioned in the links is hosted in Amsterdam, outside the jurisdiction of American law enforcement. And, though the site has been shut down before by its host, other items I’ve read have said the operator is hosting now with a provider known for turning a blind eye to unwholesome content.

 

Incidentally, I had some conversations with colleagues and readers when we last covered this site about whether talking about this type of site did any good at all. Is it worth trying to “expose” something objectionable if the coverage could also be considered publicizing it? Well, that debate is still open, I suppose. I personally think it’s worth trying to do something about.

 

What can we do?

 

To start, there’s making sure it can’t happen to you. You have an acceptable use policy, and a terms of service. You can craft those documents in such a way that you are legally able to remove anything you consider patently objectionable from your servers. There are, after all, hosts that refuse to host pornographic images. It is worth mentioning that my “legal advice” comes backed by no credentials whatsoever. So don’t plan on holding up a printout of this blog post in a courtroom at any point. However, it is worth looking into these means of protecting yourself from this kind of content.

 

As for protecting children from this kind of predator, the more important issue, hopefully bringing attention to this kind of site can help put a stop to it.

 

I mentioned last week that I felt more confident about the existence of a hosting community, and impressed by the potential power of that community. I think this is exactly the sort of situation that an organized community would have the power to affect. As an industry, Web hosts ought to be able to exert their influence, to define what is acceptable ethically, whether that is an accepted definition for “carbon neutral” or shutting down a website that caters to sexual predators.

 

Websites have Web hosts, and Web hosts have network providers and hardware suppliers. Even if one host doesn’t mind hosting a site like this, I’d imagine the industry at large would be quite unanimous in its desire to exert its will in this situation.

One more thing: the lawyer quoted in the Globe and Mail article, Perry Aftab, posted a comment on the online story pointing out that his organization wiredsafety.org, can always use volunteers to help stop sexual exploitation of children and other abuses online.

This blog post by David A. Utter in WebProNews points to a post published by blogger Devindra Hardawar, that appears to have uncovered the questionable purpose behind a site that was a big hit in recent weeks with social bookmarking sites.

 

The site, known as “The Web 2.0 Effect” purported to be an experiment testing how a selection of Web hosting providers fared at weathering the storm of traffic that can come from an appearance on something like Digg.

 

As planned, the site apparently made a splash on those very sites in the last few weeks, and appeared to be a somewhat useful tool for evaluating low-cost hosting.

 

After a bit of investigation, Hardawar posted that something fishy appeared to be up at the site. The site, it seems is affiliated with the hosting provider finished the “test” with the best score. A company called Burton Hosting.

 

There is a long list of red flags on the site, not the least of which is the fact that the person behind the experiment (who, incidentally, writes in the first person but does not identify himself) inexplicably chose to include the almost-unheard-of Burton Hosting while leaving out certain other obvious low-cost considerations, such as Go Daddy.

 

What’s more, it would have been a fairly unreliable – arguably almost useless – experiment even if it was legit. Hardawar says:

 

“This wasn’t a very realistic experiment: The test was simply a paragraph of “Lorem ipsum” text on a colored background. If they really wanted to test real-world hosting capabilities they should have included post-1995 web features such as images, dynamic content through a database connection, and scripting of some sort.”

 

I’m not going to run through the whole list of bogus details, which are listed quite nicely in Hardawar’s post. I’d suggest reading the whole thing.

 

There is a bit of a history now of Web hosts trying unusual Web-based marketing projects. There’s the troubling (Globat’s maybe-unknowing work with amateur pornographer Dustin Diamond). And there’s the simply dull (the Web “games” produced by companies like Affinity, WestHost and Hostway).

 

But for all their faults, none of these efforts crossed the not-so-imaginary line that I have to imagine exists somewhere in the grey area of questionable marketing efforts. If The Web 2.0 Effect is indeed affiliated with, or produced by, Burton hosting, then it incorporates at the very least a bit of misdirection by omission. And using misdirection to sell at all costs is a spammer’s tactic.

 

Is Web hosting still far enough under the radar that nobody’s watching to hold this sort of effort accountable? I’m sure the old “Pepsi Challenge” taste-test TV ad routine was held to a certain standard of accountability in terms of truth - if not by the FCC then probably by Coke.

 

Maybe it’s up to the Web hosting community to do something about this kind of dubious tactic.

I think it would be safe to assume that everyone's quite aware of the Viacom/Google/YouTube lawsuit by this point, given the tendency of $1 billion legal actions to be reported in the news.

There's an interesting Reuters story here, which includes input from a couple of analysts on the impact the lawsuit might have on the popular interpretation of the Digital Millenium Copyright Act and how changes to that interpretation may affect service providers as a result of this particular dispute.

(There's a similarly interesting article on The Register)

This, of course, is where we begin to see the impact on Web hosts - and let me be clear here that I'm talking about anybody that provides users with a means for posting (potentially copyrighted) content online, be they an old-fashioned Web host, or the provider of some social networking service or user-generated content site.

Basically (bearing in mind that I'm not technically a lawyer), the interpretation up to this point of the DMCA's "safe harbor" provisions has protected hosts against liability in exchange for responsibility. That is, there's an understanding that a host can't necessarily prevent its customers from putting copyrighted materials online, but it bears a certain responsibility for responding to complaints.

Up to this point, it has been enough for a site like YouTube to respond quickly when notified of a possibly infringement. But that understanding of the DMCA may change with the legal action.

Part of the premise for the Viacom lawsuit seems to be that YouTube isn't doing enough to prevent users from uploading copyrighted content in the first place. The company isn't taking advantages of some of the technologies that could identify potentially copyrighted material when a user tried to post it.

This may surprise you, but when reading about a $1 billion legal dispute between two gigantic media corporations, I tend to view both companies' official lines as just slightly less than 100 percent true.

Again, I suspect I'm not telling you anything you don't already know here, but YouTube does see a tremendous amount of traffic from people searching for copyrighted clips of, say, the Colbert Report. And Viacom may not be quite as interested in the sanctity of its copyrights (I think it is patently impossible that Viacom believes a copyright-infringing 30-second clip of one of its shows on YouTube is anything other than highly effective free advertising) as it is in stalling the progress of YouTube while its own Internet video investments play catch-up.

Ultimately, getting a look at anybody's real feelings about truth or justice becomes awfully difficult when there are billions of dollars involved.

The important thing, for you and I, is that this case could have a tangible impact on how you treat the possibility of copyright infringement among your customers, and what the court is willing to offer you in its interpretation of the DMCA's safe harbor provisions.

(By the way, this seems like a perfect subject for a couple other bloggers to weigh in on, so I'm going to go so far as to email some people looking for input if they happen to miss this post - expect some interesting follow-up commentary)

TAGS: google, youtube, viacom, dmca