Preserving Our Stuff

We live in a time where technology has enabled us to do and enjoy things that our ancestors could hardly imagine. For example, my photography hobby makes it possible for me to capture marvelous images of nature, and the people and animals I love.

These images have become very important to me. For others, it may be video tapes, or audio recordings. Or email: I have a lot of personal history saved in emails. Or my web sites — I have even more history and "sweat equity" saved there.

This leads to a concern that's unique to our digital age. All of this "stuff" that we want to keep, and perhaps to pass along to the future: how do we save it in a form in which people can use it in the future?

Lest you think this is an arcane concern, think about what has already happened to 45 RPM records. Could you play one at your home? I certainly could not...

I am by no means the first to worry about this. The same problem applies to business records, research results, and government records; these issues have lots of people worried. The general topic has even acquired a catchy name: the "Digital Dark Age". Danny Hillis, a famous computer designer, makes this observation:

... Danny Hillis also put the problem in perspective at a conference on "Digital Continuity" held at the Getty Center in Los Angeles in February 1998. "Back when information was hard to copy" said Hillis, "people valued the copies and took care of them. Now, copies are so common as to be considered worthless, and very little attention is given to preserving them over the long term." He noted that thousands of years ago we recorded important matters on clay and stone that lasted thousands of years. Hundreds of years ago we used parchment that lasted hundreds of years.

As a result, Hillis suggests, we are now in a period that may be a maddening blank to future historians — a Dark Age — because nearly all of our art, science, news, and other records are being created and stored on media that we know can't outlast even our own lifetimes. We arrived at this situation partly because digitization otherwise offers so many profound benefits. We can now store, search, and cross-correlate literally everything. In fact, according to estimates by Bellcore's Michael Lesk, who calculated the total amount of data there is in the whole world, storage has now surpassed data, probably permanently. There is more room to store stuff than there is stuff to store. We need never again throw anything away. That particular role of archivists and curators has become obsolete.

All of this rings true, for example, for my personal photography work. With my digital camera, it costs nothing to take a picture — so of course I take many more than I used to. I now have over 100,000 of my personal photographs saved; each photo that I've spent time editing and processing is saved multiple times. They all fit on one small computer in my office. I can access any of them in mere seconds, and they're all organized very conveniently. Imagine if I were to save these on slide negatives (which would preserve them without concern for technological changes): they would occupy something like 30 cubic feet, and much more if organized in boxes per topic (like I have my electronic files organized). That's just not going to happen; I'm not willing to give up the convenience and small size of the electronic storage format.

So what can we do to preserve the electronic stuff we care about?

In the world of business and government, so far as I can tell there are two general kinds of efforts underway. One kind tries to impose a (hopefully) durable standard on the world; the other gives up on a standard and instead tries to find a way to convert the wildly variable world to one form that will be durable only within the archival system. Both of these are incredibly difficult to pull off.

The first approach suffers from a problem that seems completely intractable to me, because technology is evolving so quickly (and at an ever-accelerating rate, to boot). The problem is that any standard you might choose today is likely to be rendered completely obsolete by new technology developments. Some of these will be surprising; others are completely predictable. For an example of the latter, consider the way we represent images digitally today. Generally speaking (and I'm over-simplifying), images are represented as rectangular arrays (or grids) of dots (called pixels). Each dot has three values associated with it: one each for the intensity of red, green, and blue light at that dot's position. By mixing these three colors appropriately one can generate any color, or at least we can fool our eyes into seeing that color. If someone were designing a durable standard for image storage today, they'd be quite likely to base it on this kind of representation. The problem is that it's quite likely (in my opinion) that this will change as our sensors get more and more sophisticated. I wouldn't be a bit surprised to discover, at some point in the future, that we were using cameras that essentially took a spectragraph at every pixel. This would be much more information at each pixel than we get today — and the red/green/blue method of storing an image would be completely unable to deal with it. So much for our durable standard...

The second approach has a much different kind of difficulty: it is an almost unimaginably large and complex problem to handle in a comprehensive way. I can't imagine any organization short of a government or a monster company (such as Google or Microsoft) pulling this off. Actually, it seems like a natural fit for Google, as once you had all that information stored, of course you'll have to provide a way to get at it. This, I think, is the approach that is most likely to succeed. I suspect it will be made available as a web-based service to consumers, which will be how we finally solve the problem of how to store our stuff.

But what do we do with our stuff between now and that happy day when Google announces "Google Stuff", or whatever they call the service I postulate? The only general rule I can think of is this: be careful not to store anything you care about in some completely proprietary form. Or, at the very least, if you choose some proprietary storage means, make sure you can "export" your stuff from that proprietary system to something more standard.

Here's a some other interesting articles on the subject, here, here, and here.