Drowning in Data

Sometimes, the best laid plans go awry. Whereas my goal is to exercise my writing muscle at least every few days, this week found me drowning in the depths of a meaningless data analysis required by my current task. I struggle with tasks that require delivery for the sake of meeting a contract, when I know that the underlying data is essentially useless in terms of quality. You can put a bow on it, and spray some air freshener, but in the end dung will always retain its intrinsic qualities.

In the realm of scientific analysis, test and evaluation, the appetite for collecting data is insatiable. So often, we fail to identify what data we need, and in the absence of a focused plan, we collect all that we can. The result is predictable: mountains of data that likely mean little to our purpose, but require perpetual storage just in case its importance is someday discovered. And so we must take great care in the choices we make for storing that data.

This related article in the Wall Street Journal talks about this challenge. Scientific technologies have advanced our data collection capabilities far beyond what have ever been possible before. Digital data collection holds the promise of archiving vast sums of human knowledge, but the Achilles heel remains the storage mechanism. Indeed, a simple forgotten password, or the fact that common digital storage mediums only have a shelf life of 5-20 years can mean that vast sums of research and effort could be lost and made useless forever. A simple example: in my house, we have an external drive connected to our satellite receiver so that we can save off programs for future viewing. For reasons unknown, however, that drive is no longer recognized by the receiver, meaning that an entire season of ABC's Lost is inaccessible (it's okay, I'll wait for the DVD set). But I also have had numerous important floppy disks that have decayed, and are thus useless. So while the digital revolution has made information more accessible than ever before, it has also made that same information much more vulnerable to decay, corruption and irretrievable loss.

In the long run, no scientific data can outlast the storage media that contains it, unless it can be accurately recopied and reliably re-authenticated. Many computer CDs, DVDs and flash drives last only a decade or so. The oldest known star atlas, inscribed on a scroll discovered in Dunhuang, China, has survived for more than 1,000 years. It might have been traced from an even older star map.

I'll take a real book over a Kindle any day. A book will always hold the promise of treasure; CDs, DVDs and flash drives - debatable. To the point:

"Digital information lasts forever -- or five years," says RAND Corp. computer analyst Jeff Rothenberg, "whichever comes first."

Additional reading: RAND's "Digitial Preservation: The uncertain future of saving the past."

I love the subtitle.