Dave Young from Joyent recently blogged about Twitter's use of Joyent Accelerators. Accelerators are Solaris Containers on Sun Fire X4100s with Sun Fire X4500s (also known as "Thumpers") for storage. Joyent promises on-demand, no-leash computing and offers virtual servers for as little as $45/month (includes 256 MB RAM, 5 GB storage, 15 GB bandwidth). It sounds pretty cool - and check out the video of Dave and Jason on Sun's website!
The problem is, after reading Dave's post, I think of him every time Twitter is down. Which, as many of his readers pointed out, happens often. Dave says us complainers are missing the point. Twitter is growing like crazy! It serves 4,000+ requests per second! That's a lot - and Joyent helped get them there! Unfortunately (or fortunately?), Twitter users' demand seems to exceed its already-substantial capacity.
If I were Dave, I'd move Twitter to as many XXL Accelerator Sparcs as it takes. Having come a long way just doesn't make good enough PR fodder when you've got John Edwards live blogging from the campaign trail ("About to make remarks at the Int'l Assoc. of Firefighters. Then remarks at the Boilermakers conference.").
A few months ago, I was telling Steve Kahan over at The Planet that he ought to turn a couple of his sales reps into venture capitalists, of sorts. These folks would scan the customer database for major brand names as well as up and coming influencers. They'd proactively monitor these VIPs' infrastructure and offer free scalability advice and migration assistance. They'd set up an invitation-only beta program and strong arm Dell into providing test units of its latest gear. They'd research these customers' industries and make introductions if they come across people in similar markets...
More recently, RedMonk analyst James Governor suggested something much more radical. Forget that beta program; how about long-term loans for future movers and shakers? And instead of my idea of creating case studies out of The Planet's great working relationships with today's news-makers, take a great leap forward to the open source hardware business model. Put your tools in the hands of tomorrow's innovators. You need to do this quickly, because you're competing with Jeff Barr. In Joyent's case, I have no doubt that last part is true...
PS - It just occurred to me that SoftLayer, in particular, might have much to gain from being a patron to soon-to-be influencers. Softlayer announced a private meet me room a few weeks ago, where developers of different SoftLayer-hosted applications can interconnect without incurring bandwidth charges. So if someone's created a community that many others are eager to extend and/or leverage, wouldn't it be worthwhile for SoftLayer to make itself that community's home base?
PPS - Hosted Solutions, too! It's cool that they're spearheading the Carolina SaaS User Group, but I think what would really enhance their appeal is if they hosted the most-mash-upped apps.
The down-time with Twitter wouldn't necessarily be solved by giving them more Accelerators. We're putting together a complete "behind-the-scenes" post that will be on http://joyeur.com next week. The outages have been a combination of application architecture (mistakes in said architecture can often *only* be discovered as things scale), the challenges of getting Rails (Ruby) achieve the performance required by the Twitter user volume, and some issues pertaining to operating system tuning (for which we're receiving outstanding support from Sun).
Having said this, Twitter is getting all the capacity they need from us, on-demand, to meet their growth needs. And we're working with them re: the other issues mentioned. I'm pretty sure Twitter would say they wouldn't have been able to achieve the growth they've realized to-date and hope to achieve in the future without Joyent's cooperation and help. You'll have that in writing next week. Thanks.
David S: No, not scanning customers' servers. Just scanning their order forms. And calling/emailing to find out what their businesses are about, what they plan to use their servers for, etc. That's allowed, right??
It's a troublesome thing to blame a hosting provider for what is really a customer software issue, but it happens all the time. Typically, it's becuase people write something (poorly) which works just fine under low amounts of traffic. Then they get Dugg, or something, and the site has problems. The external perception is "it was working before and now it's not, and I didn't change anything, so it must be the hosting provider's fault." In reality, it's would not take me very long to write a Wordpress plugin that would take down about any hardware you put your blog on with even a moderate hit rate. And, if I were to do such a thing, there's nothing any hosting provider could do to bail me out, short of fixing my code, which really is not their responsibility.
Please don't misconstrue this post as my saying that Twitter is written badly. Given how big it's gotten, I in fact imagine that it must be written really very well. Rails is a platform that makes many things very convenient for developers, but it does so at the expense of a lot of machine resources. When your traffic is skyrocketing like theirs is, and you are faced with the reality of finite machine resources, it's a likely scenario for growing pains. But, said pains are really an issue of software engineering. Just adding more hardware, which Joyent seems to be doing a great job of, will help to mitigate the situation, but ultimately the road to smooth sailing will almost certainly be paved more by Twitter's software engineers than Joyent's hardware support.
Congrats to both companies for coming together to make such a cool application.
Too Expensive!