I lot of things have fallen through the cracks the last few weeks while I have been busy with family and work obligations. One of those was mentioning the updated version of The Expanding Digital Universe. CMS Watch helped remind me when I was catching up on their posts.

This report, written by IDC and commissioned by EMC, basically says that there is more information out there than we thought and that it is being created at a faster pace than estimated. The amount of information from 2006 to 2011 will increase by a scale of magnitude. At that rate, by my calculations, we’ll hit 1 Yottabyte around the year 2025.

The Good News

The good news is two-fold. One, people are going to need systems that can store, retrieve, and manage all of that information for quite some time. The existing products don’t scale that big, so there will be upgrades and new deployments galore as organizations all start to succumb to this glut.

As for number 2? CMS Watch pointed out the other problem. There is more content being generated than we can consume and a way needs to be found to handle the informational dark matter (love that term). Solving these problems will keep ECM experts gainfully employed for decades.

Informational Dark Matter

Now I think that the problem of the informational dark matter may not be as bad as Kas believes. Why? Simple. Let’s look at an example.

Take the standard digital camera. When a friend of mine got his first digital camera, he got a 3 megapixel camera. It was great at the time. Now, the equivalent model would have 6 megapixels. That is twice the storage for the same information. Let us not forget that all the pictures aren’t actually new to the universe. Many would have existed anyway as prints from a standard 35mm camera, which I still use.

Let us not forget the growing size of Excel, Powerpoint, or Word documents that may hold the same amount of information as similar documents 10 years ago. What about all the scanned documents? That is not new information. It is just being stored on a massive SAN instead of a cavern.

All that aside, information is still growing faster than it can be processed. Let’s take a fun little example. Let’s ignore politics and take a look at the NSA. Let’s say they decide to record every phone conversation that the “law” allows. Aside from the massive storage problem, that content is being captured faster than it can be analyzed and processed, not to mention translated. What good is a conversation between two terrorists if you can’t process the conversation until after their plans have been executed?

There are two sub-problems here, as CMS Watch astutely points out.

1. How does one enable all the content to be consumed by the right people?
2. How does one insure that old, invalid information is removed to make the previous problem, and the minor storage issue, less daunting?

The first requires more intelligent systems and the second requires two things: automated purging of obsolete information and faith in that automated purge process. I have yet to meet a records manager with that faith.

So, for all you ECM experts, Buckle up! It is going to a bumpy, profitable, ride.