Monday, March 23, 2009
Reconstructability and digital preservation
In my post on Preservation vs. format tolerance, I briefly addressed the issue of how strictly format compliance should be required for preservation purposes. Here I'd like to expand on that. A frequently-mentioned idea in digital preservation is that people in the future might need to examine old files armed only with the specification for the format. Following this view, files worthy of preservation should strictly follow the specification, so that our descendants can write a file reader that implements the specification and open the ancient documents without difficulty.
There are several problems with this view. First, many specifications aren't unambiguously written. It's often unclear what is a requirement and what is desirable. (The TIFF specification has this problem.) Second, future programmers won't just give up if their software won't open a file; they'll look to see what's going wrong and will be able to work around some problems. Third, not all deviations make the file unreadable; some cause the loss of only limited information. For instance, a malformed date in TIFF or PDF may cause problems reading the date, but won't otherwise impact the ability to read the file.
JHOVE 1.x takes a "one strike, you're out" approach to compliance, with certain exceptions. If a file violates the specification in any way, it's ill-formed or invalid and you're told only that it's a "bytestream." (We have compromised this approach on specific features, but it's still the basic approach. One character with the high bit set in an ASCII file, for instance, says it's not well-formed ASCII.)
JHOVE2 improves on this by introducing the concept of assessment. According to the JHOVE2 functional requirements (version1.3), "Assessment is the process of determining the level of acceptability of a digital object for a specific purpose on the b[asis] of locally-defined policy rules. Since these rules are configurable, assessment is considered (vis-a-vis validation) a subjective determination." From what I've seen, this concept hasn't been very well nailed down yet, though I'm sure Stephen Abrams has thought a lot more about it than what I've read.
What I'm proposing here is that a metric of assessment should be reconstructability, the degree to which a file can be reconstructed in spite of non-standard aspects in its content or structure.
To a certain extent, estimates of reconstructability require guesswork about what information people will have in the future. In the case of TIFF, will they have just the 6.0 specification? Will they have the Photoshop supplement? The contents of technical mailing lists? But we can recognize that some deviations are easier to reconstruct by guesswork and common sense than others. If strictly following byte-alignment requirements makes files look broken, it won't be hard to try relaxing the requirement. If being unable to decipher a field means losing just one piece of metadata, that's not usually a disaster.
Conversely, a file can be completely well-formed and valid, yet have reconstructability issues because of external dependencies. Interestingly, all schema-based XML files have this problem. Schemas are necessarily external to the data file, and the schema's location is a URI which might easily become worthless in the future. Sometimes schemas change, even though they shouldn't; for instance, the schema for MIX metadata used to be the one for MIX 0.2, but was replaced with the schema for MIX 2.0, breaking the validation MIX 0.2 documents. So unless an XML file is packaged together with its schema, it should lose points for reconstructability. On the positive side, anyone with knowledge of the ASCII character set and a few representative files to study should be able to reconstruct the basic rules of XML, even without documentation of the format, and if the tags reflect their intended meaning, a file can be largely self-documenting.
HTML provides a worrisome case. The large majority of the HTML on the Web is defective in one more more ways, and browsers use ingenious tricks to approximate the intent of the author. Read a file with the wrong browser, and it may be a complete mess. Then there's the matter of external links, which may include JavaScript and CSS files that are essential to the presentation. HTML accounts for a large portion of the digital information available today, but it presents severe reconstructability problems.
This post is just a first shot at describing the idea, and it's possible I'm repeating what someone else has said (if so, someone please tell me). But I hope it will be a springboard for discussion.
Labels: JHOVE, preservation
