Balisage Paper: Portico: A Case Study in the Use of XML for the Long-Term Preservation of Digital Artifacts

Syntax

Many business applications of XML entail the exchange of messages (i.e. content) between different entities – often outside a single organizational or corporate entity. This has the advantage of enforcing early detection of validity errors in that content: if the receiving party cannot parse the message you send, the feedback is often swift and noticeable. Prior to submission of their XML (and SGML) content to third-party archives for long term preservation, however, this was not necessarily the case for either the document type definitions (DTDs and schemas) nor the documents themselves produced by scholarly publishers. Content was processed by their internal systems only; and it was not always processed with standard tools that enforced well-formedness or validity. While this may have made for internal development or processing efficiencies, such practices open up the possibility that some content can “break” when users downstream in space or time, external to the producing entity, and operating under assumptions of either well-formedness or validity, attempt to read it using standard tools.

Some of the DTDs Portico has received failed validity tests. Some documents were not well-formed; some would not validate against the DTD or schema which defines them. Some document type declarations contained unresolvable public or system identifers. Some contained white space or comments before the XML declaration. Encoding declarations were sometimes omitted where the default (UTF-8) encoding was not the one employed by the document. Some documents incorrectly declared one encoding and employed another. Some documents declared they are instances of one version of a publisher DTD, but employed elements from a later version of that DTD. Some documents incorrectly declare that they are standalone. Such errors are relatively rare -- comprising less than 1% of the content Portico has processed. They are also easily avoided –- by mandating and enforcing a check of well-formedness and validity at the point of origin of XML documents (as Portico itself does when it produces both normalized XML article content and XML metadata).

By policy, Portico does not ingest into the archive any XML full-text or bibliographic metadata document that does not pass well-formedness and validity tests. Portico maintains and preserves a local copy of every DTD or schema that defines XML content in the archive, along with (where permitted by contract) a copy of any supporting documentation the publisher provides. Portico uses these local copies to perform checks of well-formedness and validity. Access to a copy of the document type definition has proved essential for validation, in particular in the case of those definitions which include the use of defaults for attribute values. Portico has developed a set of Java filters to enable resolution of system identifiers to these locally-maintained document type definitions, to handle incorrect encoding or standalone declarations, and to stream past content before the XML declaration.

For many publishers, a single XML document is comprised of more than a single XML file. Sometimes the document fragments are referenced from within one of the files, as external parsed general entities. Often, however, there is no such intrinsic, express linking amongst the components of the document. The publisher's creating application assumes that a downstream process (presumably one of the publisher's own) will know how and where to inject these fragments. Any downstream process outside of the publisher's own workflow, of course, must reverse-engineer this association of components. The XML profiles described above provide the Portico workflow with sufficient information, typically based on file path and file name signatures, to associate the component files, and to pass them to a concatenating XSL transform prior to normalization. (The Portico-generated XML metadata that is associated with the archival artifact details the links among the components of a multi-file XML document, as well as the links from that document to the other supplemental files described above). A somewhat more challenging operation is required in the case where the constitutent files contain something other than XML fragments. The workflow has been instrumented to handle the case where one or more of the components of a document are fragments of HTML. These fragments are filtered using the JTIDY tool to create well-formed fragments, and then are concatenated, along with other XML fragments, within a Portico-defined wrapper element, and are passed to a normalizing transform. The transforms similarly invoke JTIDY, wrapped in Portico-developed XSL extension functions, when a document comprised of just a single file contains HTML markup interspersed in the text.

Semantics

A yet more subtle challenge to be faced in the use of XML as an instrument of long-term preservation is the issue of semantics. As Jerome McDonough has pointed out (McDonough), information interchange even in the present between users of what nominally are the same XML vocabularies, but whose semantics are fact construed in different ways, is far from seamless. As Wendell Piez has discussed Piez 2001, there is a gap between formalism (which lends itself easily to automated tools) and the meanings people attach to those formalisms. As Piez says, we cannot automate signification itself.

To perform the normalization of the many publisher XML vocabularies in which content is received, Portico undertakes an intensive analysis of each publisher’s use (or, more commonly, given the frequent subcontracting of content creation to external vendors, uses) of its vocabularies, whether the DTD or schema is supplemented by publisher documentation or not. When a new publisher stream is initiated, Portico undertakes a non-automated (though aided with automated tools) investigation of the distinct contexts of all elements and attributes, the actual versus the possible use of element contexts or attribute values, and the manifestation of the content of those elements and attributes in print and online versions of the digital resource. XSL transformations to normalize that content to NLM are coded defensively: if the transformation encounters an element or attribute in a context not detected in the analysis, for example, an error is raised and processing of that document halts for investigation and, as required, modification of the transformation.

Portico's analysis of publisher content adumbrates the publisher DTD’s use of character entity files. These are analyzed for conformance to standard entity files and to identify publisher-specific non-Unicode private characters. Any discrepancies with the standard entity files are resolved with the publisher. Portico transforms any publisher character entities to Unicode where appropriate. Similarly, comments and processing instructions are scrutinized. Any part of the transform which results in making explicit the “generated text,” or implied data in a publisher document (described below), clearly labels the resulting elements as having been generated by the archive, and not by the publisher.

The transformations are tested in a setup environment which runs the same tools as the production content preparation system. The outputs on this setup system are subject to visual inspection by a quality assurance team. Only after this period of analysis and test are the publisher-specific tools directed to the production (archive ingest) system, where processed content is again sampled and subject to visual inspection after automated quality assurance checks have been performed, and before content is ingested into the archive.

setup: Portico Content Setup and Ingest Environments

Generated Text

Two key components of Portico's data normalization strategy are neither to lose data from, nor tacitly to add data to, publisher content. The use of what Portico refers to as generated text -- actual textual content, such as citation punctuation or boiler plate text, that does not appear in the marked up document, but does appear in print and online manifiestations -- is an example of the challenges of what Wendell Piez has called procedural markup (Piez 2001). It is also the near occasion of loss of meaning in transformation, if not scrupulously handled. As an example, consider the following two instances of markup. The first is a reference to a book:

 
                    <ref id="B1" content-type="book">
                        <label>1</label>
                        <nlm-citation citation-type="book">
                            <person-group person-group-type="author">
                                <name>
                                    <surname>German</surname>
                                    <given-names>R. M.</given-names>
                                </name>
                            </person-group>
                            <source>
                                <italic>Powder Injection Moulding</italic>
                            </source>
                            <year>1990</year>
                            <publisher-loc>Princeton, NJ, USA</publisher-loc>
                            <publisher-name>Metal Powder Industries Federation</publisher-name>
                        </nlm-citation>
                    </ref>
                

The second is a reference to a thesis:

                     <ref id="B16" content-type="mastersthesis">
                        <label>16</label>
                        <nlm-citation citation-type="thesis">
                            <person-group person-group-type="author">
                                <name>
                                    <surname>Karataş</surname>
                                    <given-names>Ç.</given-names>
                                </name>
                            </person-group>
                        <source>
                            <italic>Toz Enjeksiyon Kalıplamada Karışımın Reolojisi</italic>, 
                            Doktora Tezi
                        </source>
                        <year>1997</year>
                        <publisher-loc>Ankara, Turkey</publisher-loc>
                        <publisher-name>Gazi Üniversitesi Fen Bilimleri Enstitüsü</publisher-name>
                        </nlm-citation>
                     </ref>                 
                    

Here is the rendition for the first reference on the publisher website:

book

Here is the rendition for the second reference on the publisher website:

thesis

In the first case, the <source> element is rendered in italic, and no quotation marks are added; in the second case, the element is not rendered in italic, and quotation marks are added. This proved to be a consistent, not anomolous, application of behavior based on an identical content model, triggered by a different attribute value. The business rule however was implicit in the document, and was undocumented in the DTD.

This challenge of detecting procedural semantics, whether from syntactic cues in the XML document, or from a comparison of the source XML document to its various renditions (in PDF, or on the publisher website) manifests itself even when converting from what is nominally the same, or nearly the same, document type definition. Nearly a third of the different publisher vocabularies currently processed by Portico are some version or variant of the NLM journal archiving or journal publishing DTDs. In theory, normalizing this content should amount to not much more than an identity transform. In practise, while these transforms are in fact considerably simpler than those from document types outside of the NLM family, care must be taken to make explicit the implicit semantics sometimes buried in the publisher's implementation of the document type definition. By design, the NLM DTDs allow for a great deal of leeway in, for example, declaring whether an element or attribute is required or optional, or providing suggested rather than controlled lists for attribute values for such things as a publication type, or suggesting but not enforcing the use of values as defined in RFC 1766 for language codes in the xml:lang attribute. For this and for other reasons, normalization is not always straightforward. For example, we have found:

• a publisher who places text (for example, punctuation, such as a period after an author's middle initial) in a processing instruction, rather than in the the text of the element

• a publisher who extracts the content of the id attribute of a <p> (paragrah) element and inserts it at the beginning of the text of that paragragh (but only for two of the journal titles in that publisher content stream)

• a publisher who employs default style information for table cell (<td>) elements when the style attribute is omitted

• a publisher who generates the text (Russian). or (French). in citation elements whenever the xml:lang attribute of the article-title element has a value of ru or fr, respectively, and does not otherwise employ the xml:lang attribute for that element

As mentioned above, while careful to ensure such implicit text is not lost in translation, Portico does not tacitly alter publisher content. Any generated text is demarcated within an <x>element, whose x-type attribute is set to the value archive, to distinguish the source of that text as Portico. So, for example, in the case of generated language name mentioned above, the transform for that element looks like:

 
                        <xsl:choose> 
                            <xsl:when test="article-title[@xml:lang='ru']"> 
                                <x x-type="archive">(Russian). </x> 
                            </xsl:when> 
                            <xsl:when test="article-title[@xml:lang='fr']"> 
                                <x x-type="archive">(French). </x> 
                            </xsl:when>
                            <xsl:when test="article-title/@xml:lang"> 
                              <-- defensive error handling code for unexpected attribute 
                              use in this context here -->
                            </xsl:when>
                        </xsl:choose> 
                    

Documenting and Enforcing Usage Policies

In producing a normalized version of an article, Portico itself is a content creator. Portico endeavors to document in detail its interpretation of the semantics of the NLM DTD to which the content is normalized, as well as Portico usage policies. Portico maintains a customized version of the public NLM documentation created by Mulberry Technologies. This customized documentation contains a section accompanying each element and attribute, detailing Portico’s interpretation of the semantics and usage of these objects, especially where such interpretation or usage is in any way a refinement or restriction of that expressed in the standard NLM documentation. In addition, Portico enforces its usage conventions where possible with immediate Schematron validation (in addition to DTD validation) immediately upon creation of the transformed file.

Figure 6: Portico NLM DTD Documentation

Linking

Many of the XML article files received by Portico contain links to supplemental files provided along with the XML file, as well as other sorts of linking information. These external links can be addresses (email, ftp, web URL) or well-known persistent identifiers of one sort or another, such as CrossRef’s DOI, or keys that map to content in the gene or protein databanks

The presence of these links has obvious implications for the long-term. preservation of this content. No one can guarantee the future stability of the targets of any of these links over the very long term. Portico preserves all link information, and, if enough information is provided in the source document, attempts to characterize the link type. But as a matter of policy, Portico can only warrant the stability of links to those supplemental artifacts actually packaged with the source document, and preserved in the archive.

Versioning

Both publisher and archive document type definitions are subject to revision, some backwardly-compatible, some not.

With respect to publisher document type definitions, even absent prior notification of a change, Portico’s practice of validating all provided input and defensively coding transformations to detect new elements and attributes, new contexts for elements and attributes, or new values for attributes, enables automated detection of any such changes.

Since Portico’s inception, there have been six minor (i.e. backwardly-compatible) and one major (non-backwardly-compatible) revision of the Portico XML archival unit metadata schema, in concert with a single non-backwardly-compatible extension of the content model of the objects comprising the archive. The major version change was made with a view toward enabling a richer set of content types, and more complex relationships among the components (text, images, supplemental files, versions of the same object, different manifestations of the same conceptual object) of those content types. Portico’s experience as it moved through the six minor versions was instructive when it came time to develop a schema that can be more flexibly extended with, for example, new event types from a controlled list. This flexibility comes at the reasonable price of a richer consistency enforcement mechanism than validation against a schema alone. Consistency in the case of the new metadata model is enforced via Schematron, which was also applied, along with schema validation, to metadata migrated from the version 1.x to version 2.0. The content model underlying the new schema, the data dictionary for the schema, the mapping from the 1.x to 2.0 versions, as well as the mapping to analogous elements in METS and PREMIS, are all documented in detail.

Scale

Fifteen million of anything is a lot, and can take a long time to process, and a lot of space to store. The fifteen million archival units presently preserved in the Portico archive comprise over 176 million files, and fifteen terabytes, with nearly 100 gigabytes of metadata for every terabyte of content. It required careful planning to scale up the Portico content preparation system from an initial capacity of roughly 75,000 archival units per month, to a capacity of 2,000,000 archival units per month – scaling that involved not just hardware and software, but organizational practices and procedures as well (Owens et al 2008. Any attempt to get our arms around long-haul preservation of digital content, XML or otherwise, necessitates getting our arms around issues of scale. As Clay Shirkey Shirky noted, commenting on the Archive and Ingest Handling Test,

Preserving content at scale means automation, including automation of the generation of metadata: technical metadata about file format instances, bibliographic metadata, including search term generation and indexing, event metadata recording provenance information about the handling and processing of digital artifacts. Not just the quantity, but even the individual size of these artifacts, generated by machine, and by and large intended for machine processing, can be problematic. As an example, Portico uses the JHOVE object validation tool to extract technical feature information and to perform object validation for digital objects in XML, SGML, PDF, TIFF, GIF, ZIP, GZIP, TAR, WAV, JPEG, and JPEG2000 formats. The XML output containing the technical metadata generated by JHOVE is stored in the Portico medatafile associated with each archival unit. This XML output can be quite large, and quite deeply nested. Portico has encountered output describing a technically valid PDF instance, which validates successfully against the JHOVE schema, but which could not be processed by Saxon 8.7.b without causing a stack overflow, because of the deeply-nested nature (over 2000 levels deep ) of the metadata. These automated artifacts must also be subject to scrutiny, to ensure that they too can successfully be handled by downstream processes, whether downtream means in the very near, or the the very long, term.

Scale will likely be a determinant in whether and how XML can be used to solve the problem of preservation of very large digital data sets.

Non-XML artifacts

Both electronic journals and e-books in their current state are at base electronic realizations of an underlying print paradigm. As a consequence, they and their associated bibliographic, technical, provenance and other event metadata map very well to XML. Some of the supplementary files accompanying journal articles, for example, such as HTML fragments, are amenable to conversion and preservation as XML. By and large, however, it is not at all clear that, apart from their metadata, supplemental non-text files -– not to mention other born-digital content such as very large data sets, or electronic games, or such ephemeral digital objects that might be described as behaviors (the effect of an interaction with server-side software) -- are as well served by preservation in XML. However, even if such artifacts are not best preserved as XML, we may well find that such XML-based preservation tools as the PLANETS project's eXtensible Characterization Language (XCL) Becker et al and its associated file characterization tools will give us information essential to evaluate different automated methods of migration or emulation at scale of non-XML artifacts, and to assess the lossiness of any such preservation actions.

Culture

Despite the litany of challenges rehearsed above, the critical issues, the difficult problems in the preservation of digital objects are not technical - -they are institutional, organizational, and financial. Will perduring organizations with adequate funding, will, and technical know-how self-organize to meet the needs of ongoing stewardship of digital assets?

There is a cultural aspect as well to the viability of XML as a tool in the kit of digital preservationists. While it is true that an XML document can be read by nearly any plain text reader application, it is nevertheless the case that the power of XML comes from a rich ecosystem of public standards, a rich application toolset (including many free and open source tools and libraries in a variety of languages), and a large community of practice. Many of the tools and libraries in particular have had substantial corporate financial support, from which the XML community at large has benefited

What happens if this support disappears? What happens when a new generation of developers, for whom XML is no longer the new, new thing, looks to new technologies, or new paradigms for modeling information – information that no longer finds its natural form as page image or text? What if a (possibly cyclic) graph rather than an XML tree becomes the mental idiom for structuring digital content –- even essentially textual content? What if the tree depth of automated XML metadata becomes, practicably, too deep? Will structured key-value stores be seen as the solution to this difficulty in semi-structured or even in structured data, as it has begun to be for large-scale distributed data stores? There is a life-cycle not just to platforms and languages, but also to the larger idioms of coding, modeling, and design Morrissey. Will we be able to detect the sell-by date for XML applications?