A Call For Good Data Stewardship Before The Digital Deluge


Everybody knows that America’s crumbling roads, bridges, power plants, and other physical infrastructure won’t make it into the future without costly maintenance, retrofitting, and construction. It’s expensive but necessary: Our civil infrastructure provides the foundation for modern living, and it’s hard to get anywhere without it.

The same is true for our digital world. Information infrastructure(or cyber-infrastructure)—computers, networks, data, storage, software systems,and the folks who run them – play the supporting role for life and work in the Information Age, and drive 21st Century commerce, entertainment, research, and practice.

Most of us know someone who lost data when the hard drive on their computer crashed. Their digital valuables may have included the music in their iTunes library, photos, personal financial information, or even the draft for a book or important research paper. We are now at risk as a nation of experiencing similar losses on a far larger scale.

One of the most important, yet fragile, parts of our cyber-infrastructure is the electronic information on which the digital world depends. E-documents, digital medical images, scientific data video, audio, and personal digital records are critical to our lives and work, yet they are constantly at risk. Accessing them requires up-to-date storage media and functional software systems that enable us to organize and query the information. Preserving them over years and decades requires migration of this information between many generations of storage media without interruption or loss.

International Data Corp. reported last year that the amount of digital data we’re creating is beginning to exceed our ability to store it. By 2011, the amount of data we’re generating will be 10 times bigger than it was in 2006.

Most important, the care and feeding of digital information requires someone to pay the bills – for new media, data transfer, oversight, etc. – and someone to take responsibility for the access, evolution, and preservation of critical data.

Herein lies the problem. We often don’t include the “data bill” as part of our long-term IT cost estimates—especially when that means the costs of maintaining electronic records for decades. It’s easy to think that data is free and resides somewhere on the Internet where others are keeping an eye on it. We depend on longitudinal climate data, electronic medical records, and electronic financial data, but do we know who’s preserving that data for future access?

The economics of the access and preservation of digital materials is emerging as a fundamental challenge of the Information Age. Sustainable support for data preservation must survive the ebb and flow of competing institutional priorities, and cannot sustain gaps in funding. Current economic models for data preservation include advertising, per-use fees, institutional subsidies, consortium funding, subscription, and other approaches. Regulations such as Sarbanes-Oxley are beginning to provide “sticks” (penalties for non-compliance) to go with the traditional “carrots” (recognition of the importance of future investment, increased opportunities) to accelerate the development of viable sustainable economic models for digital access and preservation.

As the co-chair of a task force that recently released an interim report on this problem, it’s clear that we must think of our cyber-infrastructure in the same way we think of our bridges, utilities and physical infrastructure. We must integrate digital data (and information technology in general) into our economic models. Our cyber-infrastructure cannot be allowed to crumble. Only then will it provide the necessary support for our digital world.

I’ve developed more specific recommendations in a guide to data preservation that I prepared for Communications of the ACM (Association for Computing Machinery), which is available here. Our interim report on the challenges of sustainable digital preservation is available here.

Trending on Xconomy

By posting a comment, you agree to our terms and conditions.

Comments are closed.