Last Thursday night, I came across this SearchStorage.com article via the Storagezilla blog. Beth Pariseau wrote that Amazon's Simple Storage Service (S3) has had "performance and reliability issues serious enough" to prompt second thoughts among early adopters. In particular, SmugMug CEO Don MacAskill recently decided to move hot storage back in-house.

The instant I finishing reading the article, my RSS reader lit up with Don's response. He still loves Amazon, even if S3 hasn't solved the "speed of light problem". It takes at least 60-80ms for bytes of data to travel the distance between SmugMug's west coast location and Amazon's east coast data center. There's no getting around that. He moved hot storage closer to his web servers NOT to solve Amazon's performance problems, but to reduce those thousands of miles to inches. Don also tracked down the Storagezilla post and added a comment there.

Fast forward to this morning, when someone sent me a snippet from a Tier 1 Research news brief in which Dan Golding wrote about Amazon's disillusioned users. I gave Dan a hard time for basing his article on the same two customers Beth interviewed without giving her credit. Dan argued that attribution isn't customary in the analyst world. Besides, we shouldn't even be having this conversation. As a non-subscriber, I should have deleted any T1R content that came my way upon receipt.

Ironically, during his HostingCon presentation last year, T1R founder Andy Schoepfer's key message was "don't be an island". It's important for web hosting providers to connect customers to external ecosystems like eBay and Amazon, because no e-business can thrive in isolation. Given T1R's Hosting 2.0 advocacy, Dan's reaction seemed... Analyst 1.0-ish. But towards the end of our conversation, he did promise that an upgrade is on the way. As a point of reference, Burton Group, Dan's former employer, has a great blog that links to external sources. Same goes for Forrester. And at least 220 other research firms, including T1R parent company The 451 Group. Raven Zachary, who leads 451's open source practice, is even on Twitter!

Anyway, I'll get off my soapbox now and back to Amazon. I think every web hosting exec needs to read Don's blog post - along with Robert Cichon's post on customer satisfaction metrics. Robert said a hosting provider has done a good job if (a) the company gets written testimonials, (b) customers refer other customers because they're happy with service quality, and (c) customers defend the company against negative remarks. Amazon gets three points based on Don's reaction. What's your score?

There are so many things I like about FastServers, starting from its product packages. The lowest-cost SKU on its front page is a $198 Celeron 2.4. I can't think of many dedicated server providers who can match its ARPU.

Among the several thousand EV1 customers I came in direct contact with, folks who clamored for more support *FAR* out-numbered those who wished their hosting plans included more bandwidth. You'd never guess this is the case by looking at bandwidth allocation trends across the industry, but FastServer recognizes such preferences. The server above comes with just 1500 GB, but customers have access to 5 DEFCON service levels.

Some time ago, the CEO at a competitor of FastServers' shared his concern that customers were gravitating towards a handful of rock star data center techs and network engineers. He worried that this kind of dependence isn't scalable. Yesterday I got to spend some time with FastServers VP Sales and Marketing Aaron Phillips; he has a completely different take. When he asks customers for testimonials, many of them mention how much they enjoy working with specific employees. In addition, prospects often sign up because they like what they see on FastServers' staff profiles. (On a related note, check out Michael Cote's post on the value of leveraging non-executive employees' influence.)

FastServers also gives individual employees exposure through its blog and monthly newsletter - which Aaron publishes with *amazing* regularity. Over the past three years, the only month he missed was December 2005.

I think FastServers is awesome - especially since Aaron confirmed that another HostingCon party is in the works :) And I wish more hosting providers shared their openness and consistency in customer communications, not to mention their ability to steer clear of race-to-the-bottom commoditization.

I keep reading about Twitter everywhere. It's a service that lets users post 140-character-or-shorter updates on what they're up to. If you scroll through the public timeline, you might not immediately conclude that Twitter is a must have business app. (A few minutes ago, xboxbetty was eating chocolate and BrianWarren was making coffee. And as you can see, I am writing a blog post about Twitter.) But as LifeHacker points out, it can be a nifty communication tool.

1. You can use Twitter as a shorthand newsletter. The example that LifeHacker gave was a video store, whose employees can post new movies now available for rental, holiday business hours, limited-time sales promo, etc. Likewise, web hosting providers can keep customers posted on network status ("some DC2 circuits scheduled for maintenance @ 2-3am") or announce new products ("this just in: quad core servers for $199!"). As Amazon's Jeff Barr puts it, the 140 character limit forces you to be brief without wasting time on formatting. Customers can stay up to date via your RSS feed, which is both more reliable and less intrusive than email.

2. Twitter could also be useful for managers who are responsible for scheduling sales or tech support coverage, especially if your reps are scattered across multiple locations. You can subscribe to a feed that aggregates your entire team's real time updates ("running 15 minutes late", "leaving for meeting, back @ 3pm", etc), which they can post via SMS (or a web browser, or IM or an internal app that you integrate with Twitter's API).

3. Lastly, LifeHacker recommends Twitter as a to-do list, but Anne Zelenka says it might be better as a "already done" list. For instance, if you and your two partners are traveling to three different cities, you can use your aggregated Twitter stream to keep each other informed on important events that don't require follow-up action ("contract signed with Company X", "new batch of switches will be delivered next Tuesday", etc). It reduces email clutter and is more flexible than IM; each member of the group can catch up at his convenience.

Technorati, CNN, BBC Video and Google News all have Twitter channels; shouldn't you? It's free, it's super convenient - and according to Kathy Sierra's "Asymptotic Twitter Curve" (which I came across via David Terrar's blog), it might help you capture a little more of your audience's attention.

Michael Arrington says the single hardest thing about running Techcrunch is keeping the site live. Some weeks more hours are spent on the site's infrastructure than its content. But thank god for Media Temple - and for Chris Lea for getting out of bed at 4am to get stuff fixed.

Dennis Howlett says he, like Mike, has had difficulties with the new version of Wordpress. Most hosting providers won't have anything to do with third party apps, but Chris is helping Mike out. Of course, Techcruch does have 189K RSS subscribers, not to mention millions of monthly visitors - many, many of whom are developers. Still, Mike's Media Temple experience made me think of RedMonk analyst Stephen O'Grady's recent post on "triangulating for success".

Stephen points out that PR/marketing is not about reaching prospects as separate individuals. When someone hears about your product or service from a single source (especially if it's you - but even if it's someone they know and trust), it probably won't have much impact. "But the second notice, from a trusted party, triggers a little click of recognition, and is far more likely to register. Further mentions only escalate this, until the interest to skepticism ratio tilts in favor of a trial."

What this mean, Stephen says, is what you already know: every user counts. Anyone - not just prominent influencers like Mike - can have an impact on adoption. Have you triangulated your way to any new business lately? Chris has. Rackspace is a winner as well. Not one but two happy customers commented on Mike's post that they love being able to reach specialists - people who know and care - instead of someone who isn't empowered to help. I'm sure word on the street was one of the reasons why they were able to sign up 1,092 new customers last quarter. So think twice before you feed customers that "unsupported" line!

PS - If you need help identifying potential influencers, maybe RedMonk can help. James Governor's "distribution first, dollars second" pitch is mainly directed at the IBMs and Suns of the world ("Why not give some zSeries mainframe capacity to Linden Labs, say, but before it got famous?"). But I think his message is equally applicable to The Planet, say, or Verio. He says you need to "capture the customer at the point of desire, but then charge them at the point of value." Doesn't that make sense?

My trip to Taipei began and ended with two very different customer service experiences. (The photo above was taken from Taipei 101, the world's-tallest-for-now skyscraper at 1,671 feet. The building has a 10 GigE fiber optic trunk line.)

(a) Upon my arrival, I got into a taxi whose driver offered a choice of sparking or still mineral water and Chinese, Japanese or English newspapers. He explained that a cab ride is not just a cab ride. Customers might be thirsty if they've come off a long flight. And they might want to catch up on local news if they're on their way to a meeting. He tries to think about what goes on in the customer's life before and after each ride, because people are more likely to give repeat business to vendors who create a seamless experience.

(b) On the morning I was scheduled to fly back, I heard on the radio that all flights were temporarily grounded because of heavy fog above the airport. United told me that my flight would be delayed by 6 hours. Nonetheless, I should arrive at the airport 2-3 hours before my originally scheduled departure time because it's "more convenient" to check in "on time". And how would I get back to DC if I missed my connecting flight in Tokyo? The rep confessed he hadn't thought of that.

During my looong wait, I tried not to be mad at United for circumstances beyond their control. Somehow the situation reminded me of my EV1 days; I'm embarrassed to say that the experience I delivered was closer to (b) than (a). If a customer's server died, he was supposed to create a ticket for "easier tracking" - even if he'd already reported the outage by phone. And once the machine had been replaced, any problems associated with restoring data from backup were "beyond our scope of service".

Of course, policies and procedures are essential for keeping costs and work loads under control, but is "we don't support third party scripts" the best way to deliver a seamless experience? Where are hosting customers coming from? And where are they headed with the bandwidth and storage space they've purchased? Are there manageable equivalents of newspapers and bottled water for surprising them with care?

PS - I've always really liked DataPipe's "operational empathy" tagline: they say it's about anticipating customers' needs and taking a personal interest in their goals. While I'm sure they've trademarked the phrase, they don't have a monopoly on thoughtfulness!

1. Jan from HosTest sent me some very interesting data from his latest research. During November 2006, Jan submitted a variety of tech support requests to 50 hosting providers. He rated responses on a 5-point scale based on accuracy and completeness. MochaHost, Intermedia.NET and NetRegistry got perfect scores. Jan was particularly pleased with MochaHost's 24-minute response time.

In contrast, the wait time at Verio was 62 hours and 59 minutes. This is particularly disappointing considering the "select the right hosting provider" report that Verio recently commissioned from Tier 1 Research. It advised customers to monitor their hosting providers closely. "The company's response to issues early in the relationship will be an indicator of things to come".

2. Declan McCullagh from CNet surveyed 12 domain registrars on their policies for handling abuse complaints. DirectNIC and French registrar Gandi.net, he said, offer the most extensive guarantees against unnecessary domain name suspension. DirectNIC requires a court order unless a clear case of child porn or phishing is involved, and Gandi.net takes extensive steps to contact domain owners.

I was surprised to read that Demand Media subsidiary eNom chose not to respond to Declan's survey. CEO Richard Rosenblatt told Business 2.0 some months ago that he's planning to bring a sprinkling of Web 2.0 stardust into the domains market. One would hope there's some transparency in the mix?

3. Barry Abrahamson from WordPress writes that he's in the market for 37 servers. It looks like ServerBeach will get the business because of their "super-simple and painless sales process", their "everything we asked for at a very fair price" proposal and the fact that Peer 1 VP Marketing Rajan Sodhi blogs on WordPress. Rajan was also first to respond to Barry's blog post. Very cool!

SoftLayer, on the other hand, misses out on a possible opportunity. James Byers from WikiSpaces comments that SoftLayer is fantastic, but no follow up from SL so far. Considering that 4 out of the company's last 5 press releases mentions Web 2.0, maybe SL should start a WordPress blog?

PS - BTW, Matt Mullenweg pointed out earlier this week that you can blog on WordPress using MS Word 2007. You might want to pass along these instructions and these screen shots to your customers.

A few months ago, Bambi Francisco from MarketWatch wrote a really interesting article on Google's success. Google beat Yahoo, Lycos, Excite, Netscape, AltaVista, etc because its search-based approach to monetization is better aligned with end users' natural behavior. The other search engines focused on funneling search traffic into portals and selling banner ads on those portals. It's much tougher to make money from an audience whose behavior you first have to modify.

I think Bambi might approve of SixApart's latest announcement that its blogging platform is integrated with Windows Vista and Office 2007. (I read about it on Larry Dignan's ZDNet post.) Users can post to their blogs from Microsoft Word, set up RSS feeds so that new blog posts appear in certain Outlook folders, download blog archives into Word documents, etc. SixApart VP Anil Dash says he won't be happy until every company can have a blog. To reach that goal, he's bringing blogs into an environment that prospective new bloggers are already familiar with.

I feel like this is a significant challenge that web hosting providers need to tackle. During the mid-1990s, the consensus within the industry was that we've only seen the tip of the iceberg in terms of web hosting demand. And as recently as three weeks ago, Lou from HostMySite commented on one of my posts that vast opportunities lie ahead still. After all, most consumers still don't know what web hosting is. And every other small business still doesn't have a website. And 90% of all web servers are hosted in-house. And 50% of enterprise data centers will become obsolete in the near future....

These are encouraging stats, but during his HostingCon CEO panel last year, David asked whether we're counting unicorns. What will it take to bring these folks into the web hosting market? Some of the panelists said it's about trust: we've got to convince the public that our infrastructure is reliable and secure! Other said it's about customer service: we need to have friendly reps on hand to patiently explain how our services work! But how come nobody mentioned the importance of better aligning our way of doing business with customers' natural behavior?

The average consumer probably doesn't know how to install scripts. The average small business owner most likely isn't an expert on securing/patching servers. The average enterprise IT manager might think of his operating expenses in different terms than your colo packages. The question is, should the goal be for customers to adapt to what web hosting providers have to offer - or could web hosting be adapted for their worlds?

Robert Cringely says Google is on its way to become the biggest CDN ever. It controls more network fiber than any other company in the world - and it's building two huge data centers in South Carolina alone. Google's enormous capacity, coupled with end users' increasingly voracious appetite for all things web-based, will lead to a future in which Google will act as a giant proxy server for the Internet. It will be "our phone company, our cable company, our stereo system and our digital video recorder"...

Greg Linden disagrees. He thinks that instead of cornering the market on bandwidth, Google is trying to build a world of infinite storage and CPU power. Well, *and* bandwidth.

No web hosting provider will have access to Google's economy of scale. Or out-of-this-world local government incentives. (Unless you're a state-licensed operator in China, but that's a different story.) Sorry, there isn't and won't be any way around this. BUT one resource that Google won't be dishing out in buckets is... knowledge.

I came to this realization while reading this article on MySpace's infrastructure. Once again, I spotted it on Data Center Knowledge!

MySpace was originally built on Perl + Apache + MySQL, but before the site went live, its developers switched to Windows + Cold Fusion + MS SQL. Later on everything was rewritten in C# + ASP.NET. The site started out with just one database, then write vs read transactions were split between one master DB and two slaves. Soon this evolved into vertical partitioning, or a separate database for each and every feature. Finally its technical team settled on running separate database instances for each block of 1 million accounts. All logins come through one single front-end DB, which redirects each user to the database containing files associated with his account.

In the beginning MySpace had server-based storage, then it built a storage area network with room for more disk drives. At one point the company had two full time technicians (!) manually distributing SAN resources between database instances. Finally it moved on to a 3PARData's virtualized storage solution, on which all disk drives can be accessed as one single pool of capacity. In 2007, MySpace plans to replicate its SAN (which is currently in LA) in two other locations, to eliminate its dependency on one single data center location.

Other major changes MySpace has made to its infrastructure include a caching layer between its web and database servers. In addition to minimizing DB lookups, machines on the caching tier are used to store temporary session data, which aren't given permanent database space. Also, Myspace was among the earliest adopters of MS SQL Server 2005, the better to take advantage of its 64 bit support. The 32-bit SQL 2000 limited MySpace to 4GB RAM per server. Now its standard config is 64 GB RAM.

Throughout endless trials and errors, MySpace's developers were under enormous pressure to keep its service online. Could they have benefited from having a team of Internet infrastructure experts to bounce ideas off of? Comparing their experience with benchmarking data from other fast-growing sites?

At this point, MySpace has grown beyond any hosting provider's past knowledge. On the other hand, right now this minute, the founder of tomorrow's MySpace could be ordering his first server at your data center. Ask him what he's up to. Do some research on whether any of your current customers are building similar applications, and what kind of growing pains they've gone through. Turn your hosting company into a repository of knowledge. Because it's the one card you can to play against infinite storage, CPU power and bandwidth.

Matt Mullenweg from WordPress caused quite a stir last week when he vented on his blog about his frustrating experience with Sun's Startup Essentials Program. He wrapped up the post by saying he's "far more excited about what Amazon is doing these days."

Within 23 minutes, Amazon evangelist Jeff Barr - who was recovering from surgery - commented on Matt's blog that he would love to talk. And about a day later, Sun CEO Jonathan Schwartz posted an apology on his blog:

"All I can say is... I'm really sorry, Matt. That's not the way Startup Essentials is supposed to work. We screwed up, and you're completely right to suggest if that's the norm, we should kiss goodbye our aspirations of reestablishing our business in the startup community. If there's anything I can do to win a second chance, I'd like to know."

I totally agree with Red Monk analyst James Governor's assessment of the situation: "the sales battle for contracts, from major to minor, is increasingly being fought out in the blogosphere. If you don't play then you may miss out on a pay day... Go read Matt's blog again. As an RFP." (RFP = request for proposal.)

Now that Amazon and Sun have responded to Matt's RFP, another company that I wish Matt had heard from is LayeredTech. Back in November, Matt mentioned that WordPress has 50+ servers at Layered. Considering (a) the existing relationship, (b) Matt's interest in Amazon's S3 and EC2 services, and (c) Layered's new on-demand, pay-as-you-grow computing service, I'm disappointed that Todd from Layered hasn't joined in the conversation. Especially since both Matt and Stephen O'Grady, James' Red Monk colleague, are both getting interested in Media Temple's GRID platform

Speaking of which, Media Temple's recent blog post really helped spread the world. Along with Matt and Stephen and dozens of others, Scott Yang over at HostingFu is talking about it too. Unfortunately, Media Temple hasn't followed up on these threads either - at least not publicly.

I know that every web hosting CEO is overwhelmingly busy - but I'm sure Jonathan Schwartz' to do list isn't any shorter than yours?? PLEASE... do a Google Blogsearch for your company - then subscribe to the RSS feed for new search results, and keep on top of these discussions. Publicly.

The last company I'm going to pick on in this post is ServerBeach. Kevin Burton wrote last November that "while I like Serverbeach they need to get their act together fast or I might just switch to EC2." I've forwarded this post to at least 3 different people at Peer 1, but the only public response Kevin's gotten has been from... Jeff Barr over at Amazon.

Allan Leinwand, who used to be Digital Island's CTO, says he's intrigued with Dell's upcoming data migration service. Later this year, customers will be able to securely transfer not just their documents/photos/videos/songs - but also programs, drivers and settings from their current computers - to Dell's storage portal. Dell will install everything on their new machines during the manufacturing process, reducing migration hassles to opening a box.

Allan thinks other vendors should follow Dell's lead. Wouldn't it be cool to get a new Tivo that's pre-configured with your current settings? A new GPS that's pre-loaded with your frequently used addresses? A new cellphone with your contacts and calendar items ready to go?

Allan's post reminded me of the same conversation I had over and over while I worked at EV1Servers. At least once a day, a customer will say he wants to upgrade to a different server. We can move all of his stuff over, right? Right? With no down time??

I hated dashing their hopes. And if Dell's data migration service takes off, you will no longer be able to get away with saying no. Customers will increasingly take for granted that contents of Machine A can auto-magically be transported to Machine B without requiring them to lift a finger.

That's another disadvantage of standalone dedicated servers. Think about it... Is it easier to resize someone's entire infrastructure with a few clicks - or spend forever troubleshooting why code from their ancient Celeron won't run on a Quad Xeon? Because thanks to Dell, you - rather than the customers - will be doing the work.