October 27, 2006 | Posted By : Isabel Wang at 8:46 PM PERMALINK | Blog Comments (6)

30,000+ Users x 30 MB downloads in 23 Hours - Guess How Much It Cost?

A little over a year ago, when I worked at EV1Servers, I got a call from an event producer who needed 200 Mbps of bandwidth capacity for a three-day webcast. I told him I could set up a dedicated gigabit switch with unmetered connectivity, but he'd have to place his order a week ahead of time, and keep the system for at least 30 days. He said I was out of my mind, and I thought he was being unrealistic. What I offered was a highly flexible solution that had served hundreds of other customers well.

But I was wrong.

Earlier this week Second Life (SL)(which traceroutes to Internap) came out with a new version of its client software. Half way through release day, the file was moved to Amazon's Simple Storage Service (S3). According to SL's official blog:

"For the tail 8 hours of the download rush, we averaged roughly 70 gigabytes of viewer download per hour. [That's about 160 Mbps] Then it settled down to a relatively steady stream of about 20 to 30 gigabytes per hour. In the last 23 hours we've transferred a total of ~900 gigabytes so far - which I'd estimate to be around 30,000 to 38,000 downloads."

S3 costs $0.15 per GB for distributed, redundant storage and $0.20 per GB for bandwidth. Second Life now owes Amazon $180.15. In contrast, the solution I quoted the would-be customer cost at least 20x more. And that's for one single server on Cogent-only bandwidth. Yikes.

By the way, SL had also considered Akamai, but...

"It just turned out that the S3 solution was ready for deployment immediately, where akamai requires more negotiation. In other words, we already had an amazon S3 account where I was test something out, and then when we noticed the bandwidth was pegged, we made a fast decision to speed up our plans to put our viewer elsewhere, and chose S3."

Here's what I think is the moral of the story: the web hosting industry is evolving very, very quickly. If you can't meet a customer's needs - no matter how crazy it sounds - someone else will.

Which reminds me of a conversation I had with my friend Patrick. He's an attorney now, but I'm expecting him to become a Web 2.0 CEO any day now.

Him: If I order a web hosting plan, they'll back up my data continuously and automatically, right?

Me: Which planet are you from?

But based on what David Wartell at Righteous Software has been telling me, quite a few web hosting companies will be able to deliver what Patrick's looking for very soon. The question is, will you be one of them?

Print | Send | del.icio.us | Linking Blogs

Comments

[Add Comment]

Today's expectations are very demanding, or are they? What once were features in some datacenters are now industry standards.

Just take cell phones for example I remember when getting one that would make calls was sufficient now people want them to be able to do everything from take pictures, email, text, video, etc., now that is pretty standard.

# Posted By Mario Rodriguez | 10/28/06 12:26 AM

You weren't wrong. Things change rapidly in this industry. A year ago, you were doing the best you could for that customer. You've observed that this same offering today wouldn't cut it against other providers, but you're being a bit unfair to yourself with your comparison, don't you think?

As for Wartell's stand, I can see this being possible. In fact, I'm pretty sure companies are doing this now, sending an incremental image to a series of remote hard drives, each in a RAID array. Every hour, a different drive gets the incremental backup. Because it is incremental, the load and time is relatively small and quick, and you always have a "save point" available up to X# of hours back. This would be overkill for a 1-to-1 server/backup setup (unless your data was really that mission-critical), but for a cluster, I can see it, and I think it's already taking place.

# Posted By Paul Hirsch | 10/28/06 11:23 AM

I didn't mean wrong as in I treated the customer badly, but in retrospect I was short-sighted. EV1's private rack was a really popular solution, and people who bought it seemed happy. So I thought of it as a finished product; it didn't occur to me that customers requirements might evolve beyond its capabilities. It's not easy to innovate and standardize at the same time, but it looks like that's what it takes to stay in this business!

As for backup, you should look at Wartell's product :) (http://www.r1soft.com) It does almost continuous incremental backups, even on open files.

# Posted By Isabel Wang | 10/28/06 12:24 PM

I was reading through his site yesterday. I looked at product overviews, but there was little about the products, just information about what's wrong with current methodologies. So I stopped looking. I just now looked again, and I did find the product information. One of the big questions when looking at backups is the time windows. What is the time window to correct a data error before you're stuck with that error in backup? I saw that Wartell's product does continuous backups while keeping a journal of changes, presumably so you can return to a very exact restore point. I'm curious as to how incremental those restore points really are (is it RAID with a journal??), and what the overhead is like to set up a system whereby you have a few days or a week or more to make your restore.

Or am I looking at this whole thing from the wrong perspective? Maybe my brain is conditioned to think about backups one way, and I need a different viewpoint to understand Righteous. Either way, I think I'm taking this all a bit off-track. Sorry! :-)

# Posted By Paul Hirsch | 10/29/06 8:36 AM

These are very good questions. How frequently you can backup depends on how write intensive your environment is. For example, some of our customers with heavily loaded virtual hosting servers are able to take incremental backups every 5 or 10 minutes. The snapshots are stored on a backup server. Typically one backup server can handle backups for between 50 and 300 servers depending on how often they are backed up and how much data is changing. The journal is used only to drastically reduce the time it takes to complete a backup operation and is stored in memory. The change tracking introduces no measurable overhead when enabled. Only the incrementals taken during a backup operation and stored on the backup server are used for restores.

If compression is enabled its not unusual to be able to store 10 complete disk images in less space than it takes to store one copy of your data. Rotation policies can be setup to automatically rotate out old backups based on your defined requirements. For example some users keep 24 hourly backups, 7 daily, 4 weekly, and 12 monthly backup images.

You are welcome to try the software for free by joining our beta test program. Contact us at www.r1soft.com for more information.

# Posted By David Wartell | 10/29/06 10:50 PM

Thanks for the clarifications Dave. Your explanations make a lot of sense. I know a few people who would be interested in looking at this, and I'll point them to this blog post :-)

# Posted By Paul Hirsch | 10/30/06 10:52 AM

[Add Comment]