I came across Uptime Institute founder Ken Brill's CIO Magazine article via 3tera VP Marketing Bert Armijo's blog.

Ken says while hardware prices are falling, total cost of data center ownership is headed through the roof. 5 years from now, the purchase price for a rack of servers will drop 27.5% from $138K today to just $103K. But while it only takes 15 kilowatts to power that rack right now, the energy requirement will rise to 22 - 170 kilowatts by 2012. It could cost as much as $2.3 million to power/cool $103K worth of gear throughout its 3-year lifespan.

(I'm not sure if this figure includes switches and routers and such. A recent Cisco/APC/Emerson study shows that servers/storage/cooling consume 76% of data center power, with 11% going to networking equipment, 3% lighting, and 10% power conversion losses. If Uptime's calculations didn't take the other 24% into account, Ken's $2.3M becomes over $3M!)

I've been thinking about Ken's stats and trying to understand what they mean. As a point of reference, I was looking at Dell's website, which advertises the 4U PowerEdge 6950 dual core, dual processor Opteron server for about $9K. Is Ken saying that:

(a) This particular machine will cost 27.5% less 5 years from now?

(b) 2012's late model machines will sell for 27.5% less than what's on the market today?

(c) The amount of server hardware that fills up 4U of space will be available for $6500 in 2012?

If we assume he means (c), and we accept Sun's claim that "server performance, power and space efficiencies are improving at up to 40% annually on average, and could double every 2 years", then 4U of space may be able to accommodate not one but 4 servers that each feature 4x more processing power and 4x greater energy efficiency.

In other words, $6,500 could buy you 16x more computing resources than that dual Opteron! If that's the case, you might even be able to afford $1M per rack per year in electricity. But only if you virtualize like crazy. No more leasing data center space per square foot or per rack. No more dedicated servers, either. The average customer won't need 4x more processing power in 5 years, which means you won't be able to justify turning on a whole entire server just for them.

You'd also have to replace hardware early and often. Sun recently announced a refresh service for swapping out your servers at least 3 times over 42 months. At first I thought that sounded wasteful, but if server power efficiency is improving at 40% per year, holding on to old gear might end up costing you more. Again, virtualization would be a must. You wouldn't want customer apps to become attached to machines that will be phased out before long.

Bert from 3tera says changes in data center economics will make it increasingly difficult for enterprise CIOs to justify operating their own facilities. But they won't outsource to traditional colo or dedicated server providers. Instead, he agrees with Cassatt CEO Bill Coleman that in the near-ish future, you'll be "paying for data center horsepower the same way you pay for electricity or gas". I think so too. How about you?

PS - On a somewhat related note, eWeek says Intel will release its "Clovertown" chips today. The quad core processors have a 50 watt thermal envelope, versus 80-120 watts on earlier models. That's a 38-60% drop.

PPS - Also, speaking of the Uptime Institute, check out this SearchDataCenter.com interview on how they've helped The Planet save $10K/month on electricity. The Planet, the article says, is looking to expand beyond Texas into the Midwest.

Amazon CTO Werner Vogels mentioned it first. Now there are discussion threads on Slashdot , Engadget and Digg.

Google Research's disk failure trends report reminds me of Ronnie, a former customer whose 10-server population suffered FOUR hard drive failures over the course of three weeks. At least a couple of the failed drives drives were brand new. Other folks with his server config experienced no such disasters. Were Ronnie's machines located in a particularly hot section of the data center? Was his utilization much higher than anyone else's?

Believe or not, after analyzing over 100,000 hard drives (parallel or serial ATA, 5400 to 7200 RPM, 80GB to 400GB) between Dec 2005 and Aug 2006, Google found that there's no significant correlation between either temperature or activity level and failure probability. Age isn't necessarily a good predictor, either. The average failure rate among 1 year old drives was ~2%, rising to ~8% in years 2 and 3, but declining to ~6% in year 4 (see page 4 of this PDF).

Also importantly, while certain SMART parameters (especially scan errors, reallocation count, offline reallocation, probational count) correlate strongly with higher failure rates, over 56% of the failed drives had zero counts on any of these variables. In other words, it's not possible to pinpoint impending failure based on SMART data.

On the other hand, drive models, manufacturers and vintages do make a huge difference. The report doesn't show breakdowns, but Amazon's Vogels says you pretty much get what you pay for. In addition to investing in high quality disks, you can also improve average longevity of your hard drive population with a longer burn-in period, during which bad disks will be weeded out.

Still, hard drive survival is very much a game of numbers, which isn't reassuring news for anyone who's running mission critical apps on standalone servers. Vogels says this is a good reason to store data on S3 and let Amazon worry about the problem; many users agree. If you're in the hosting business and would like to hold on to your customers, a virtualized storage platform is increasingly a must-have. "Fast hardware replacement" offers just aren't as compelling as freedom from hassles associated with hard drive failure.

Between May and August 2006, Forrester Research surveyed 1,770 companies on their adoption of virtualization technologies. 40% of the respondents said they've virtualized production servers (up from 29% in 2005). 11% are in evals. More impressively, as InfoWorld reports:

"Awareness of virtualization by Global 2000 large enterprises grew to 92% in 2006 from 87% in 2005. Awareness by medium-to-large businesses jumped to 86% from 60% and to 83% from 62% by SMBs."

And which virtualization vendors are these companies familiar with? VMWare (53%). I was surprised that only one respondent mentioned Xen, but the upcoming Xen-integrated RHEL5 might change that. 6 mentioned Solaris Containers, which Joyent offers. SWSoft's Virtuozzo was notably absent from the list; I think its focus on web hosting providers as a distribution channel might be at least partially to blame.

Within the web hosting industry, the predominant virtualization use case has been VPS hosting, where the goal is to put multiple customers on a physical server. Customers typically choose between pre-packaged service plans with fix resource allocations; they have no control over VPS density, and no ability to create additional virtual environments with excess capacity.

But out in the enterprise IT world, people find virtualization appealing because it gives them beyond the box access to computing resources. Imagine the difference between opening a 2-liter soda bottle every time you're thirsty, versus being able to pour any amount of its content into a glass.

Unfortunately, this architecture isn't all that compatible with the web hosting business model. Dedicated hosting providers want customers to deploy as many servers as possible, instead of piling multiple operating systems and software applications onto the same machine until it reaches full utilization. Aside from the obvious objective of maximizing revenue by increasing customer server count, there are also logistical considerations such as software license and IP address management. And shared hosting providers are heavily dependent on overselling capacity.

The question, can web hosting providers hold on to the "soda only comes in 2 liter bottles (which, in our hands, can be stretch out to a couple dozen 1 liter servings)" paradigm forever when prospective customers have near-total awareness of more flexible alternatives?

Allan Leinwand, who used to be Digital Island's CTO, says he's intrigued with Dell's upcoming data migration service. Later this year, customers will be able to securely transfer not just their documents/photos/videos/songs - but also programs, drivers and settings from their current computers - to Dell's storage portal. Dell will install everything on their new machines during the manufacturing process, reducing migration hassles to opening a box.

Allan thinks other vendors should follow Dell's lead. Wouldn't it be cool to get a new Tivo that's pre-configured with your current settings? A new GPS that's pre-loaded with your frequently used addresses? A new cellphone with your contacts and calendar items ready to go?

Allan's post reminded me of the same conversation I had over and over while I worked at EV1Servers. At least once a day, a customer will say he wants to upgrade to a different server. We can move all of his stuff over, right? Right? With no down time??

I hated dashing their hopes. And if Dell's data migration service takes off, you will no longer be able to get away with saying no. Customers will increasingly take for granted that contents of Machine A can auto-magically be transported to Machine B without requiring them to lift a finger.

That's another disadvantage of standalone dedicated servers. Think about it... Is it easier to resize someone's entire infrastructure with a few clicks - or spend forever troubleshooting why code from their ancient Celeron won't run on a Quad Xeon? Because thanks to Dell, you - rather than the customers - will be doing the work.

I've been having a lot of conversations - with the 3tera folks, the Tier 1 Research guys and several hosting providers - about how interested enterprise IT managers are in outsourcing their data center requirements.

3tera CEO Vlad Miloushev thinks as many as 90% of all servers are hosted in-house, and research from AFCOM and Gartner both show that a significant proportion of corporate data centers will become obsolete in the very near future. In which case, an attractive market should be opening up for external hosting providers, right? Because surely outsourcing fees will be easier on enterprise CFOs' eyes than construction costs for new data centers?

But blade consultant Martin MacLeod offers some interesting perspective on why enterprises may find outsourcing less affordable than we think. (A server manager he recently spoke with was "horrified" by quotes from colo vendors.)

1. Commercial hosting facilities have higher security standards than internal data centers - not to mention more reliable connectivity, newer power and cooling equipment, etc. What business value does an enterprise IT manager receive from such amenities? That might not be something he can instantly quantify.

2. In fact, speaking of quantifying value, Martin says the "hosting/support fee" within enterprises is absorbed into generic IT expenses. So companies might not have neatly broken down cost per server/cabinet/square foot stats for use in making apples-to-apples comparison with outsourced alternatives.

3. In a November 2006 survey of 500 Silicon.com readers, 33% had IT equipment between 5-10 years old, and 32% had "fully functioning" hardware that's more than 10 years old. According to Sun, servers are improving at a rate of 40% per year in terms of power efficiency, so the difference between power bills for colo-ing pre-historic versus state-of-the-art machines will widen dramatically with time. This represents a dilemma for enterprise IT managers: should they ditch still-functional gear and migrate to brand new equipment? That sounds like an expensive proposition...

Still, Martin agrees that enterprises shouldn't be in the data center building/running business. He says they should outsourced hardware in addition to facilities requirements - and run their virtualized infrastructure on an on-demand service. According to the folks at Dr Dobbs, this is where Amazon EC2 comes in.

If you don't like their conclusion, you should look into offering virtualized utility computing. A few days ago Rich Lee from Hosted Solutions commented on another post that the compute side of on-demand infrastructure is "clearly the most complex piece to integrate, provision and bill properly for", but Hosted is working on addressing this challenge. I hope that Rich and others proceed with urgency, because the window of enterprise outsourcing opportunity won't last forever.

That's the title of an October 2005 Gartner report. Lydia Leong wrote about a promising market that web hosting providers hadn't (and still haven't) addressed. Large enterprises, she said, often have small projects that don't require industrial-strength managed hosting on stand-alone servers - yet they hesitate to sign up for mass market shared hosting plans that don't offer enterprise-grade redundancy, scalability and accountability.

Lydia recommended that web hosts set up virtualized infrastructure with managed-hosting-like support and SLAs. The goal is to allow enterprises to purchase resources on a cost-per-VE basis, with utility pricing for variable bandwidth needs. A content distribution network, she suggested, might make a good add-on for this product.

Since small projects can grow into large ones (example: Hostway's 100+ server relationship with Fox News began through a $20 shared hosting plan) - and multiple small requirements can add up to a sizable contract, hosting providers who don't offer such a flexible entry point might lose valuable opportunities to build profitable relationships with large enterprises.

I thought of Lydia's report when I read Martin MacLeod's BladeWatch post on cost allocation in an enterprise IT environment. If the IT department has a 350-blade grid, and Application A uses 70 blades worth of resources 15% of the time, should its owner be responsible for 15% of 20% of the cost? What about depreciation? In a separate post, Martin asks whether IT needs to have its own accountants, the better to take each server and split up associated costs per virtual CPU, per virtual GB in RAM and storage... It might take months - at hundreds of dollars per person, per day - to develop processes, procedures and documentation on internal pricing.

My immediate reaction was, wouldn't it be less trouble if they outsourced? AFCOM says 50% of corporate data centers won't have enough power/cooling capacity by 2010 anyway; Gartner thinks this will happen by 2008. The question is, whom should they outsource to? I'm not seeing much discussion of cost-per-VE/utility pricing on managed hosting companies' websites - or enterprise-class support capability from mass market providers. It seems as an industry, we haven't taken Lydia's advice. And maybe we should: doesn't enterprise outsourcing sound like a promising opportunity to you?

IDC, by the way, says we're entering the age of "virtualization 2.0" in which virtual appliances (software packaged into virtual machines; one example is Amazon's EC2 Machine Images) will become household words. I'm a little worried that the web hosting household might not be on the same side of the tracks...

PS - I read about IDC's predictions on Kimbro Staken's Virtualization Daily blog. Kimbro is the CTO of JumpBox; they make virtual appliances.

PPS - Hosted Solutions just issued a press release about - among other things - the importance of "IT as a service". That's way cool - *except* I see only colocation and dedicated servers on their website, no "shared, highly available infrastructure".

I didn't get interested in virtualization until pretty recently. Earlier this year I had a long conversation with Serguei Beloussov from SWSoft. He said virtualization is the future; I said the entry barrier against Virtuozzo adoption seems unreasonably high. If you consider the licensing costs/learning curve on one hand, and the availability of cheaper/better hardware on the other, I wasn't sure the math worked out. But since then...

* The launch of Amazon EC2 made a huge splash in the media and among developers, who rave about the convenience of on-demand virtual server instances with standardized machine images. Much easier than deploying and configuring physical equipment!

* I met 3tera through Nicholas Carr's "software kills hardware" blog post. The company's AppLogic grid operating system allows users to deploy, scale, copy, migrate or backup entire applications with one single command (!) by packaging web/app/db/storage infrastructure into one single logical entity. (I joined the company's advisory board a few weeks ago.)

* The Las Vegas Water Valley District enjoyed seamless disaster recovery because its DNS and domain controllers ran on virtualized infrastructure. Thank goodness for VMWare, said the agency's sysadmins. VMWare turns a server into a file. Because it's just a file, you can copy and deploy it as needed.

* And last but not least, ArvatoMobile won InfoWorld's Top 100 IT Projects Award for betting its server farm on Virtuozzo. The company's entire infrastructure runs on 600 virtual servers. It's also virtualized several hundred TBs of storage into a single file system namespace.

I think these new developments are really, really exciting. So I couldn't believe it when Lance Crosby from SoftLayer mentioned that he thinks the dedicated servers market will survive. How? Why? Who wants the hassle of managing individual machines that are each a single point of failure??

As it turns out, AlertLogic does. (Note: the company's infrastructure consists of several islands of server grids where processing nodes share the load; in its case there's no single point of failure.) According to Misha Govshteyn's blog post from yesterday:

"We know enough about the characteristics of our software that we can tell you with a high degree of accuracy the exact disk I/O, dedicated PCI bus bandwidth, network interrupt needs and even which compiler to use for each component to maximize performance.

While we've considered virtualization, we walked away from the idea every time. Fine tuning hardware resources to each software component and ensuring that your architecture can linearly scale requires a great deal of control. Our architecture encourages the most efficient use of highly distributed, but dedicated, servers and introducing a virtualization layer would only create resource contention. The performance tax of VMware or Xen just doesn't justify the benefits in our case."

The moral of the story is, I've got a lot of learning to do. As does every hosting provider, I think, so as to help customers make well-informed decisions between the pros (such as what the Las Vegas Water Valley District enjoyed) and cons (which AlertLogic is looking to avoid) of virtualization.

PS - Rackspace, for one, has given a lot of thought to this issue. I got an email from Lew Moorman as I was typing this post: "Virtualization will make servers easier to manage, more flexible and more powerful. But don't focus too hard on the idea that they'll go away. They won't." Thanks, Lew! :)

I read this IBM case study in eWeek a few months ago. It said through virtualization, the US Tennis Association was able to run the US Open event website on just 9 servers (instead of 60 the year before) despite increased traffic and implementation of new features. Having spent years equating increasing server count with growth, I wasn't sure this was good news.

The article also mentions an Enterprise Management Associates research report: 75% of surveyed enterprises have already deployed virtualization, and less than 4% have no virtualization plans (!). 65% said server consolidation is a key goal. Yikes.

Gartner, likewise, said back in June that 40% of mid-sized businesses will reduce server count through virtualization by 2007. And last week, research director Jeff Hewitt said virtualization will significantly impact x86-class server sales:

The total number of virtual and physical x86 servers will grow at a cumulative annual rate of 12 percent from 2005 to 2010. But customers using virtualization can be expected to deploy about eight virtual servers on a physical server. So physical server growth rate will only be about 5% annually. The market is still growing. But this could be an early indication of a slowdown; it's something everybody in this market is watching.

If these figures worry you, Sun CEO Jonathan Schwartz' latest blog post might cheer you up.

There's an interesting phenomenon in the computer marketplace, which strikes some as counterintuitive: if you double the performance of a machine, customers don't buy half as many, they tend to double their order. Same goes for utilization, if you can double server utilization, people don't buy fewer computers - they buy more. The value of innovation is growing so fast that if the price declines, the overall return goes through the roof, encouraging a feedback loop. Moore's Law and free software drive relative pricing down, and customers accelerate their growth.

Schwartz offered a Sun sales exec two pieces of advice: sell beyond your current installed base, and trust that the market will grow. All of Sun's large accounts started as small customers, and lower entry barriers help "tomorrow's Fortune 500" gain momentum.

I was talking to someone about my recent post on virtualized complex hosting. What about ARPU, he asked? A system of virtual web/app/db servers won't generate nearly as much revenue as actual machines, but it still takes the same amount of resources to close sales and answer support tickets.

The answer, according to Sun and IBM, is that if customers can do more on a lower IT budget, they will find ways to use way more computing resources.

So, do you agree with Jonathan Schwartz' take? Or are you disheartened by Gartner's projections?

I've been reading up on RHEL5. Beta 2 came out last Friday; it includes Xen. Official release is planned for early 2007, and most hosting providers won't support it immediately. Still, (assuming you don't already have answers) now might be the time to start thinking about how it'll impact your provisioning, support and license tracking systems.

Xen lets multiple operating systems run on the same server. In this interview, Red Hat senior product management director Scott Crenshal says you will be able to install RHEL5 on "a certain number" of virtual environments for free. Within a web hosting environment, this raises some interesting questions:

1. Many data centers offer automated OS installs on physical servers - but what if a customer wanted to run FreeBSD, Debian and RHEL5 within three separate Xen instances on the same box?

2. Until now, shared hosting resellers have had to choose which control panel to run on each server. With RHEL5, they could offer cPanel, Plesk, Ensim, HSphere, Webmin... simultaneously. But does your license management database have the ability to track such combinations?

3. How would a customer submit support requests for cPanel errors, let's say, on one of many Xen instances on his server? Should there be separate IDs for each virtual environment within your ticketing system? In addition, your support team would have to be relatively up to speed on Xen.

4. I was telling my friend Jeff Huckaby over at RackAid that it'd be cool for developers to run separate web/app/db virtual servers on the same machine and move them out to separate physical servers as their sites grow. Of course, they'd want to maintain each Xen instance's IP address throughout the migration. Does your network architecture support beyond-the-box IP portability?

5. Back in June, Gartner predicted that 40% of mid-size businesses will use server virtualization technology by 2007. RHEL5 will no doubt expedite adoption. Unfortunately, according to Gartner, virtualization will cause companies to use 20% fewer servers. Will you make up any impact this might have by reaching out to a larger number of customers? Offering more value added services?