Are We Running Out of Storage Space? IDC is Concerned, but Maxell Says Never Fear

I learned about the IDC storage paradox on Zoli Erdos' blog. Zoli mentions this Associated Press article, which cites IDC's estimate that "the world had 185 exabytes of storage available last year and will have 601 exabytes in 2010. But the amount of stuff generated is expected to jump from 161 exabytes last year to 988 exabytes in 2010".

Even more alarmingly, Dan Farber over at ZDNet reports that according to IBM, "the world's information base will be doubling in size every 11 hours" by 2010. Does this mean that on Jan 1, 2011, our 988 exabytes of data will double to 1,976 exabytes by 11am, and 3,952 exabytes by 10pm?

Fortunately, we don't need permanent storage for all the data we generate. For instance, spam accounted for just 8% of all emails in 2001 (said CNet); its volume rose to 36% by 2002 and 66% by 2004 (MSNBC), and is expected to exceed 90% by the end of this year (IT News). That's a huge amount of data that isn't being saved.

Still, Rich D'Ambrise from Maxell says he expects significant growth in data archiving requirements: in 2007, we will back up 75% more data than we did in 2006. But unlike IDC analyst John Gantz, he's not concerned that we'll run out of space. The storage industry is not standing still. Maxell, for instance, is beta testing 300 GB holographic disks that are no bigger than a DVD, but offer 63x more capacity. 800 GB second generation disks should be on the market by next year, and a 1.6 TB version is planned for 2010. And let's not forget stacked volumetric optical discs (SVOD); each 92-micrometer layer stores up to 9.4 GB. Available storage capacity will absolutely keep up with demand; no question about that!

The real issue is, will we store our zettabytes of data on- or offline? Rich is betting on removable media; he'd rather have mission critical data in his own possession than depend on any service provider. Zoli, on the other hand, says online is more efficient. By sharing/linking to files, we won't each need space for our own copies of the same content. Sun CEO Jonathan Schwartz says offline storage is greener ("when data's at rest, it consumes no electricity") - and easier to transport on a large scale. (As the New Yorker points out, if you made tiny chariots with DVD wheels and hitched them to snails, you'd get faster data transfer speeds than DSL.)

So, what's this got to do with web hosting? For one, you should probably monitor your oversold disk space closely. At the moment, I'm sure hardly any of GoDaddy's $7 hosting customers are using their entire 100 GB quota. But if you consider Rich's 75% growth projection, the number of customers that same 100 GB is allocated to may have to come down.

PS - Here's a GigaOM post on a 10 more fun storage facts.