During an early 2004 customer gathering, former EV1 CEO Robert Marsh asked attendees whether they'd be interested in putting multiple servers on the same private network. I still remember how their eyes lit up. 100Mbps server-to-server connectivity? Oooh yeah!

I had a similar reaction to this announcement on SoftLayer's new "customer exchange". SoftLayer says every single one of its dedicated servers is now deployed with a secure gigabit connection into its private Meet Me Room. Imagine scenarios like Zoho's integration with OmniDrive, or tools from Flickr, del.icio.us, etc that automatically output users' content on their TypePad blogs. SoftLayer customers can now jointly develop these sorts of cross-site interactivity without incurring bandwidth charges for data transfer between their systems. How cool is that?? (Better yet, SoftLayer actually saves money by keeping traffic between customers within its private network instead of paying for packets to travel in and out of its data center.)

Earlier this week, AlertLogic CTO Misha Govshteyn and I had a discussion on customer service via his blog. I think we came to the agreement that while investments on delivering an exceptional customer experience can pay off big time, your ROI is contingent upon having an outstanding product (which itself is a key part of the customer experience). SoftLayer is a great example of how hosting providers can combine both.

By the way, SoftLayer's mobile portal is another customer-friendly new feature. In addition, the company recently launched a new data center with 500 watts psf of power capacity. It also upgraded its Verio and Internap uplinks to 10GigE.

I hope SoftLayer's upcoming projects include an application directory where customers can find partners for creating mashups with. (I think this could become a powerful sales tool as well.)

I'm on my way to Taipei, so this post will be the last one until I recover from the 24 hour trip and 13 hour time difference.

I wanted to mention Tom Offenbach's very interesting blog post about peering at the edge. Tom says:

1. Internet access is becoming increasingly ubiquitous.

2. P2P can improve download efficiency and save costs, but it works best when sharing happens locally. The closer your peers, the fewer hops the data has to travel.

3. It's crazy that last mile service providers aren't peering with one another. If Tom has Comcast and his next door neighbor has Pac Bell DSL, a P2P request from the neighbor would have to travel through Pac Bell's Central Office, Pac Bell's regional hub, Pac Bell's upstream provider, some public peering point, Level 3, Comcast's regional data center, Comcast's headend and Comcast's neighborhood node before it reaches Tom.

(BTW, this is the kind of inefficiency that CoralCDN is trying to quantify through its Illuminati network measurement project.)

Instead of this madness, Tom wants to know why Comcast and Pac Bell can't peer at the Central Office or headend level?

Let's say this does happen down the line. With increasing P2P efficiency, more and more of your customers' content will be shared at the edge, rather than downloaded from your data center. The good news is, you'd be under less pressure to deploy additional GigEs (or 10GigEs - and sooner or later, 100GigEs). On the other hand, the high capacity network that you've invested heavily in may become less of a value proposition.

A few months ago, Jon Udell (who recently became a Microsoft evangelist) wrote in an InfoWorld article that:

"We've already seen how open source software projects harness collective effort to produce quality results. We're now seeing how open content projects such as Wikipedia do the same. Can open infrastructure be far behind?"

I was intrigued by the idea and dug up a bit of info on other open infrastructure projects besides CoralCDN. All are academic, and none have been commercialized. Lots of cool technologies are out there though. Makes me wonder what the web hosting business will look like 5 years from now...

I read about it on CNet. What you see above is Lucasfilm's data center. Its network features 300 10-gig ports and 1,500 1-gig ports. Every digital artist within the company has a desktop with gigabit connectivity.

Lucasfilm uses Verari servers with dual core, dual Opteron processors and 16 GB RAM as well as bunches of legacy machines. IT director Kevin Clark says it can take as little as 6-7 months before new equipment turns legacy. For storage, Kevin has 300 TB on a NetApp NAS. (BTW, check out this Infoworld article about NetApp's Data Ontap GX virtualization software, which allowed Lucasfilm to maintain 200 TB across 20 servers while having storage capacity appear as one single 200 TB disk. Lucasfilm has been testing the software for the past year, and will be moving to a GX cluster soon.)

As with Google, Lucasfilms says power utilization is an important factor in its data center equipment selection. The company's website also points out that its facility is LEED certified by the US Green Building Council. I've been hearing a lot about LEED these days. It's mentioned in this SearchDataCenters article, and it will be on the agenda at Data Center World. If green IT is a priority for corporate data center managers, it will likely become an increasingly important topic in your sales discussions with enterprise customers.

That's what this very funny blog post from RedS Swoosh says. I came across it through Read/Write Web, where Richard McManus points out that Mark Cuban of the Dallas Mavericks is a Red Swoosh investor.

Red Swoosh is a free P2P CDN. Content owners "swoosh" their content by adding "http://edn.redswoosh.net" in front of URLs. Visitors must install a desktop client in order to access swooshed data. Files are downloaded or streamed from up to 30 peers who previously requested the same content.

Johnny Cakes, Red Swoosh's probably non-existent "Downloader in Chief", read on the BBC that Apple has sold 1.5 billion songs and tens of millions of TV shows and movies through iTunes. He calculates that...

* 1.5 billion songs x 5 MB = 7,500 TB
* 25 million TV shows x 700 MB = 17,500 TB
* 25 million movies x 2 GB = 50,000 TB
* 75,000 TB x $200 per TB in Akamai bandwidth = $15 million, which Apple could use to:

* Buy the world a Coke
* Give away 100,000 iPhones
* Buy 3,750 Segways and a lifetime supply of hot dogs
* Buy Sealand

Red Swoosh is not the only P2P CDN. The New York University-operated CoralCDN is free as well (and doesn't require a desktop client); Fark, Slashdot and Digg are all occasional users. The Coral team has been collecting data on the distance between web users and their DNS resolvers to determine how efficient it is for CDNs to make IP-address-based matches between end users and cache servers.

There's also Metalink, an open standard that bundles P2P files AND HTTP/FTP files on multiple mirrors into one single format. If a server becomes inaccessible during a download, Metalinker will automatically switch to a different location. It's even able to download segments from different sources at the same time. OpenOffice and openSUSE (among many other applications) are available via Metalink.

If you have hosting customers who need more bandwidth than they can afford, work with them to evaluate these options instead of canceling their accounts. If enough high-bandwidth users start offering P2P content, it might buy you some time before you have to fire up the next batch of 10-GigEs.

Dan Golding from Tier 1 Research sent me the link to this recent blog post by Josh Jones from DreamHost. Josh says overselling leads to more new customers and more revenue - which can be used to expand staff and improve data centers. Therefore, contrary to popular opinion, overselling not only doesn't degrade service levels, it actually makes it possible to provide a better hosting experience.

Josh made the same point back in May 2006. The law of large numbers, he said, makes overselling feasible. Some customers actually do use all of their allocated resources, but most don't. In aggregate, DreamHost is able to "live on the edge and make a profit". It keeps its file servers 90% full and peak traffic at 85% of capacity.

Over the past few months, DreamHost started giving away 8x more storage space and bandwidth. Remember how they were already close to capacity *before* this upgrade? Thankfully, ALL the quota increases did was increase signups. The new customers made it possible for DreamHost to spend over $1 million in November alone on hardware - *without* using up any more resources. Sweet! BUT... will DreamHost be able to stay ahead of the game?

For some perspective, let's take a look at the Wayback Machine. Remember 1999? I do. The average sub-$10 shared hosting plan included 20 MB of storage and less than 1 GB of data transfer.

HostMySite offered "ultra fast T-3 connectivity" (that's only 45 Mbps!) at the time; many other hosting companies ran on T-1s (1.54 Mbps). Since most Internet users were on 56k or slower dialup connections, hardly any customers came close to maxing out their super generous hosting plans.

But times are different now. Rich Miller reports that *each* viewer on the Venice Project, an IPTV start-up, downloads as much as 320MB/hour. Josh said last year that relatively few DreamHost customers "stream porn and use a TB of bandwidth". But what if more folks decide to fill their shared hosting accounts with videos? Josh said it's easy enough to add more disk shelves and install more GigEs - but that's only if demand doesn't exceed the supply of cash in your bank account...

BTW, I'm less worried for DreamHost than shared hosting resellers who run their businesses on dedicated servers. In 1999, a $500 server from digitalNATION came with 50 GB of bandwidth. Today, $500 will buy you 5 low-end machines with 1000 GB each. That means 100x more bandwidth at the same price! But to remain competitive with DreamHost, 1&1, GoDaddy, BlueHost and others, a reseller must now increase his customers' data transfer limit by 2000x. The math just doesn't add up, and the growing discrepancy could put small resellers in an ugly position.

For several billing cycles during 2005, I manually looked up the URLs associated with each and every EV1 account with a bandwidth overage balance. Can you guess what kind of business the hardest hit customers were in?

Just saw the news on CNet , GigaOm and GigaGamez. The International Association of Virtual Reality Technologies has announced that it will form a new public network that's "capable of meeting the data transmission requirements of emerging cinematic and immersive virtual reality technologies." Neuronet, as it will be called, will be "separate and distinct from the Internet, and used for everything from gaming to entertainment to virtual business."

What will Neuronet's infrastructure look like? Well, "the massive overcapacity of fiber optic cable left over from the dot-com era makes the new network feasible with minimal investment." Also, "much of the infrastructure and programming utilized to facilitate the Neuronet will be outsourced to telecommunications and virtual reality innovators." Neuronet is already talking about a domain naming system, forming committees on standards, regulatory affairs, etc, and selling memberships.

I'm a little skeptical, because never once does the IAVRT identify any company or individual who's already involved with the non-profit association. If its members include "market leaders in telecommunications, entertainment, gaming, business, several Fortune 500 companies and leading research institutions around the globe", why aren't they lending credibility by being officially represented on the site - or publicly announcing their participation?

Speaking of the site, the IAVRT.org domain (under private registration at GoDaddy) traceroutes to VerveHosting of Pontiac, Michigan. It seems Verve resells for or colos at Net@ccess, whose data center is in New Jersey. The IAVRT itself appears to be located in Vancouver, Canada. If it were a big enough operation to bring together a whole new separate public network, I'd expected its website to be hosted with a more prominent provider, or one that's closer to its home base?

Anyway, if my skepticism is unfounded, the game server teams at Peer 1, LayeredTech and The Planet ought to look into this. If there's going to be a separate network for gaming traffic, you won't be able to stay in the game hosting business without getting involved.

P.S. On a not entirely related note, speaking of virtual business, of which there is already plenty in Second Life, maybe VeriSign and Comodo should start offering virtual SSL certs? Shouldn't virtual shoppers have the ability to verify the legitimacy of their virtual vendors? You wouldn't want to lose Lindens to an unscrupulous avatar - or get your virtual identity stolen by a virtual phisher?

Just saw this graph on GigaOm. Higher capacity + lower prices in the Internet access market means increased bandwidth consumption. Unless your bandwidth prices are falling even faster, the profitability of oversold hosting plans is going to be squeezed.

Silicon.com quotes Cisco CEO John Chambers' presentation at ITU Telecom World in Hong Kong that by 2015, the monthly volume of Internet traffic will reach 15 Exabytes (that's 15 million TB). Wikipedia says as of the end of 1999, the sum of all human knowledge (including all text as well as audio/video recordings) only amounted to 12.6 million TB. The blogosphere and YouTube might have increased that figure somewhat...

Back in 2003, when I first joined EV1Servers (or Rackshack, at the time), I was impressed to learn that the company's network was responsible for carrying 2% of all Internet traffic. If we assume that the % was and will continue to be accurate, the same network will be hit with 114 GB/second in data transfer within less than a decade (and that's just average rather than peak usage). 114 GB/sec = full utilization of 912 GigEs, isn't it? Yikes.

Help me double check my math, please. 15 million TB divided by ~2.6 million seconds per month = 5.7 TB per second of worldwide usage. 2% of 5.7 TB is 114 GB/sec. 114 GB x8 is 912 gigabits/sec. Does that sound right?

And by the way, John Chambers says his estimate could turn out to be conservative. Which could be the case:

ThePlanet-EV1Servers recently announced plans to boost network capacity to 100 Gbps, and Dan Golding from Tier 1 Research says he expects Internet traffic to continue doubling each year. This GigaOm write-up last year suggests that bandwidth capacity should be 2x peak traffic. And based on EV1's network graphs, does 3/4 seem like a reasonable guess-timate of average to peak ratio? If we put all these figures together, the company's network could have 4800 Gbps of average utilization by 2015. That's crazy!

Just spotted this on Newsvine... Amazon ran a special offer for 1000 units of XBOX 360 at $99 each, and eager shoppers brought the site down! Amazon.com was offline for about 20 minutes. I guess that goes to show there are limits to scalability...

Update: the comments on Digg were surprisingly supportive: "Yeah, servers slow down during heavy traffic. Welcome to the Internet." Also: "Obviously there was overwhelming response and perhaps millions of people were clicking reload. That's enough to slow down any server, even Google. Not Amazon's fault."

By the way, those 1000 XBOX 360s sold out in less than 7 seconds.

Update 2: eWeek calls Amazon's Thanksgiving performance "virtually perfect" compared with 6-8x slowdowns and prolonged website outages at Wal-Mart and Macy's.

A little over a year ago, when I worked at EV1Servers, I got a call from an event producer who needed 200 Mbps of bandwidth capacity for a three-day webcast. I told him I could set up a dedicated gigabit switch with unmetered connectivity, but he'd have to place his order a week ahead of time, and keep the system for at least 30 days. He said I was out of my mind, and I thought he was being unrealistic. What I offered was a highly flexible solution that had served hundreds of other customers well.

But I was wrong.

Earlier this week Second Life (SL)(which traceroutes to Internap) came out with a new version of its client software. Half way through release day, the file was moved to Amazon's Simple Storage Service (S3). According to SL's official blog:

"For the tail 8 hours of the download rush, we averaged roughly 70 gigabytes of viewer download per hour. [That's about 160 Mbps] Then it settled down to a relatively steady stream of about 20 to 30 gigabytes per hour. In the last 23 hours we've transferred a total of ~900 gigabytes so far - which I'd estimate to be around 30,000 to 38,000 downloads."

S3 costs $0.15 per GB for distributed, redundant storage and $0.20 per GB for bandwidth. Second Life now owes Amazon $180.15. In contrast, the solution I quoted the would-be customer cost at least 20x more. And that's for one single server on Cogent-only bandwidth. Yikes.

By the way, SL had also considered Akamai, but...

"It just turned out that the S3 solution was ready for deployment immediately, where akamai requires more negotiation. In other words, we already had an amazon S3 account where I was test something out, and then when we noticed the bandwidth was pegged, we made a fast decision to speed up our plans to put our viewer elsewhere, and chose S3."

Here's what I think is the moral of the story: the web hosting industry is evolving very, very quickly. If you can't meet a customer's needs - no matter how crazy it sounds - someone else will.

Which reminds me of a conversation I had with my friend Patrick. He's an attorney now, but I'm expecting him to become a Web 2.0 CEO any day now.

Him: If I order a web hosting plan, they'll back up my data continuously and automatically, right?

Me: Which planet are you from?

But based on what David Wartell at Righteous Software has been telling me, quite a few web hosting companies will be able to deliver what Patrick's looking for very soon. The question is, will you be one of them?