One of the topics I like to bring up in the discussion of preserving digital data is the idea of a Digital Dark Age… the notion of a period in our historic knowledge that ends up getting lost due to a failure to plan and preserve our early digital content.

The New Scientist, however, recently published an article (Feb 2, 2010) on something a bit more cataclismic: the concept of  Digital Doomsday.  From the article:

Suppose, for instance, that the global financial system collapses, or a new virus kills most of the world’s population, or a solar storm destroys the power grid in North America. Or suppose there is a slow decline as soaring energy costs and worsening environmental disasters take their toll. The increasing complexity and interdependency of society is making civilisation ever more vulnerable to such events (New Scientist, 5 April 2008, p 28 and p 32).

Whatever the cause, if the power was cut off to the banks of computers that now store much of humanity’s knowledge, and people stopped looking after them and the buildings housing them, and factories ceased to churn out new chips and drives, how long would all our knowledge survive? How much would the survivors of such a disaster be able to retrieve decades or centuries hence?

The article is a compelling read, and offers an intellectual exercise on how much of our “stuff” will survive such a castastrophe.  Ironically, the logic is that the digital content with the most copies oin existence may win out.  So, while scholarly works, theses, research and other important scientific data would be at risk, pop music may surive just fine.

A 1949 Kodachrome Slide of Shaftesbury Avenue from Piccadilly Circus, in the West End of London. Although discontinued by Kodak, Kodachrome is a well-known format by preservationists for its longevity and color accuracy over decades of time.

The announced obsolescence and discontinuation of a number of film formats by their vendors has resulted in an expected outcry from a niche of users who continue to use these older formats for their creative and artistic works. But another, more surprising result of this community outpour is a sort of grass-roots resurrection of these media, through user involvement and investment.  It happened with the Impossible Project and Polaroid film.  Now, after Kodak has signed the death warrant for the once-venerable Kodachrome format, a film enthusiast has built his own film manufacturing device.

Not a whole lot of background detail is available yet, but this photoset on flickr has been making the rounds among gadget blogs and photography discussion forums.  It depicts the “Filminator,” a homemade film manufacturing machine, intended by its creator to make a homebrew version of Kodachrome film.

According to the person who took the photos:

Can’t buy the film you want any more? Just make the stuff!

In this set you will find random photos and information on a project a friend has undertaken – a machine to make his own camera film.

Plastic and goop go in one end, and camera film comes out the other end. This is not a trivial undertaking.

Indeed.  Although it’s gotten a lot of interest from enthusiasts who obviously want to see a way for Kodachrome to keep on living in some fashion, the builder of this contraption is strictly using the Filminator for his own personal use.  No plans exist at this time to go commercial, or to sell any film stock made from it.

The Filminator is apparently working “quite well,” but so far there’s no known images online for people to assess the results for themselves.  Nor is it known whether film stock coming out of this machine has the same longevity and resiliency that the real Kodachrome was celebrated for in its time.

YouTube is definitely the wild-west for video content, and if you look hard enough, you’ll find a lot of vintage video content.  Quite a few of it is uploaded by people whose rights to publish the stuff is questionable, but while it’s there and before it gets taken down, it’s pretty fascinating to come across old commercials, television station idents, and little vignettes that exemplify past culture.

Here’s a good example:  a 1971 Woolworth TV ad, promoting a sale on LP albums and even some 8-tracks:

An ad from a defunt store depicting a obsolete formats.  Fascinating.

The facility I work in at Rutgers, known as the Scholarly Communication Center (SCC), has a fairly short history in the grand scheme of academia, and yet a fairly long one when it comes to the rapid changes in technology it has seen in its lifetime.  It was originally started in the 1996, and meant to be a location for university students and faculty to access a growing body of the then-nascent collection of digital content.

Back then, the internet still wasn’t very fast and wasn’t nearly as media-rich as it it seems today.  And so, most of the data-heavy reference materials arriving in digital form came to the SCC as CD-ROMs (and later, DVD format).  To accommodate this, the SCC had a lab of ten desktop computers (known as the Datacenter), dedicated solely to accessing this type of material.

But the times changed, and so did the way people accessed digital material.  As the ‘net grew in size and capacity, it no longer made sense to ship reference material on disc, and so the access moved online.  Students migrated from visiting computer labs to bringing their own laptops (and later, netbooks and handheld mobile devices).  Traffic at the datacenter dropped to virtually nothing.  The space had to be re-tooled to continue to be relevant and useful.

And so, with my taking on the newly-minted role of Digital Data Curator, and in collaboration with my colleagues, a new plan for the former datacenter was developed.  Instead of being a place to merely access content, we would be a place to create it.  Analog items that needed to be digitized would be assessed and handled here.  New born-digital content would be edited, packaged, and prepared for permanent digital archiving in our repository.  We would be a laboratory where students getting into the field – and even faculty and staff who have been here a good while – would learn, hands-on, how to triage and care for items of historical significance, both digital and analog, and prepare them for online access.

The concept for a new facility was born.  And we call it the Digital Curation Research Center.

The center is still in “beta,” as we plug along with some internal projects for testing purposes along with a couple of willing test subjects within the university and surrounding community.  This is so we can test out the workflow of the space and make tweaks and optimizations as needed.  Our plan is to officially launch the space in the Spring of 2010, with a series of workshops and how-to sessions for the various things that make digital curation vital (e.g. digital photography, video editing, audio and podcasting, and scanning).

The plan is that this will be a continual, evolving learning experience for all involved.  People who have never really used cameras and recording equipment in a historical context will learn just how increasingly valuable the content they create, and the stories it will tell, can become over time.  And those of us in the DCRC day in and day out will encounter things that we’ve never run into before, and will have to wrap our heads around the issue of preserving it effectively.

Below are related documents that provide additional information about the DCRC.  More information will be coming up as we get closer to the official launch:

• Initial Proposal for converting the Datacenter to the Digital Curation Research Center – Includes Draft Floorplan (PDF File)
• Draft Access Policy (PDF File)
• Poster Session Presentation (PDF File)

As I’ve mentioned in some of my previous articles, I’m incredibly skeptical of the long-term success of any digital storage attempt that relies on a single medium as its container. In particular, I find CDs and DVDs to be extremely suspect in terms of their longevity. Those of us who have been around the block with these technologies know that Gold CDs and DVDs were supposed to be the last word on digital preservation, lasting decades if not centuries. But mere years later, we all learned a new term to add to our archival vocabulary: Disc Rot.  It became very clear to all of us that not all discs – not even the gold ones – are made equal, and some even from “trusted” brands can delaminate, fade, have their media layers flake off, or otherwise deteriorate and become unreadable in rather short periods of time… sometimes with painfully devastating results for smaller archives who banked it all on CDs and DVDs.

As a result, a type of storage medium that we were told would last decades has been widely recognized as only being good for about two to five years.

So clearly, it’s not without some deep suspicion and trepidation that I view a recent announcement by an unknown start-up known as Cranberry, in which they claim to solve the Disc Rot problem, and has introduced a type of DVD that they say will last 1,000 years.

A Cranberry DiamonDisc is a DVD made of high tech stone.
Memories carved on a DiamonDisc will last as long as the pyramids. No reflective surface. No ink layer. No fading. Problem solved. The Library of Congress is studying our technology for storage of the national archives. It’s the only solution for permanent, digital storage.

Oh, really?

This is a pretty bold claim from a company that no one has heard from until recently.  But what hard evidence does Cranberry have to – forgive the pun – back it up with?

Unfortunately, you have to dig pretty deeply into their site, to a FAQ section way down at the bottom before you get to a far more realistic statement:

How can you prove that the Cranberry Disc will last for centuries?
No one can prove that anything will last for centuries, but there are international standards for estimating the archival lifetime of optical media. The ECMA‐379 (2nd edition, December 2008) which tests the effects of temperature and relative humidity is widely recognized. Researchers at Millenniata have tested the Cranberry Disc using the ECMA‐379 temperature and humidity (85°C / 85% RH) testing as a standard to develop the most rigorous testing possible. They have combined temperature and humidity (85°C / 85% RH) tests with exposure to the full‐spectrum of natural light. The Cranberry Disc is the only survivor after this rigorous testing. Considering the combination of the Cranberry Disc’s test results and its rock‐like data layer, it is reasonable to conclude that the Cranberry Disc has a greater longevity and durability than competitors who claim a 300‐year shelf life.

Here’s what worries me about the above statement: Millenniata does not appear to be a dedicated research firm.  In fact, if you go to their website, they seem to be marketing their own archival-quality disc storage under the brand name of M*ARC.  Per Millenniata:

Millenniata is the sole provider of a permanent, backwards-compatible archiving solution for the digital age. Located in Springville, Utah, Millenniata is poised to become the world’s leader in digital data preservation. Millenniata is the result of pioneering inventions from Brigham Young University.

So… what’s going on here exactly?  It’s difficult for me to accept that Cranberry claims to have cornered the market in creating DVDs that last a millenium, only to point to research done by a company that claims it holds the sole solution to the disc rot problem.

Both companies claim that their respective formats are not something that the mundane DVD burners on standard computer equipment can burn.  Cranberry claims the needed equipment to make a DiamonDisc is “out of the reach of most consumers,” and it wants you to send your data to them so that they can “etch” a DVD for you, at a cost between $29.99 to $34.95.  Millenniata, however, will gladly sell you the special burner required to author these discs, which according to some news sources will cost up to $5,000 for the drive and a pack of 10 discs.  Once created though, both products will purportedly read just fine on any DVD drive. The capacity of these discs are 4.7GB, the same as a single layer DVD-R.

My as-yet-unconfirmed speculation is that, based on the descriptions both entities provide for their products, the storage products both companies offer might actually be one and the same.  If that’s true, then I’d have to conclude that the research done on these discs’ longevity is hardly objective and unbiased.

So, should archivists invest money in these discs?

My philosophy has always been that you should never blindly trust a vendor’s claim about how long their storage media should last, nor should you trust a single storage solution for archiving your digital content.  If a curator wants to spend multiple thousands of dollars for an M*ARC drive, or spend $30-$35 for Cranberry to author each disc, that’s their decision, but it should be supplemented by a secondary solution, be that hard drives, tape, solid-state media, or some other well-known container format.   And that data needs to be checked periodically to verify its integrity.

Cranberry claims that the Library of Congress is studying the technology for its own archival use. Personally speaking, I would like to see the definitive results of such a study, before I would feel comfortable making a decision on whether M*ARC or DiamondDiscs are worth the premium.  And even then, I still would not back away from my two-format philosophy to long-term digital storage.

It’s ironic that Cranberry is hyping up NARA and LoC studies, citing them as reasons why you should be very,very afraid about storing your precious data on standard writeable disc media.  Let’s hope for their sake that the LoC verifies their claims are true, or else their marketing literature would prove pretty embarrassing.