Balisage Paper: Beyond Well-Formed and Valid

What is Portico?

Portico is a digital preservation service for electronic journals, books, and other content. Portico is a service of ITHAKA, a not-for-profit organization dedicated to helping the academic community use digital technologies to preserve the scholarly record and to advance research and teaching in sustainable ways. As of April 2012, Portico is preserving more than 17.7 million journal articles, nearly 17,000 books, and nearly 1.5 million items from digitized historical collections (for example digitized newspapers of the 18th century).

Content comes to Portico in approximately 300 different XML and SGML vocabularies. These XML and SGML documents are accompanied by page image (PDF) and other supporting files such as still and moving images, spreadsheets, audio files, and others. Typically content providers do not have any sort of manifest or other explicit description of how files are related (which ones make up an article, an issue of a journal, a chapter of a book). This content is batched and fed into a Java workflow that is driven by XML configuration files, which Portico calls profiles (about 190, one for each publisher content stream), and registries (shared across all content streams).

The Portico workflow maps the publisher-provided miscellany of files into bundles that comprise an article or book or other content item.

Figure 1

Figure 1: Mapping Files to Content Units

Publisher-provided XML and SGML files are normalized to the Portico profile of the National Library of Medicine’s Journal Archiving DTD. The workflow identifies the format of each of the component files, and, where a format specification and validation tool is available, validates each file against its format specification. It generates metadata considered important for preservation (descriptive, or bibliographic, metadata; technical metadata about files and their formats; provenance and event metadata, detailing the tool chain, including hardware and software information, used in processing the content). These metadata are formatted as XML, and are stored with the preserved digital object.

Figure 2

Figure 2: Portico High-Level Workflow

Some of the sub-steps in this workflow are explicit QA checks of the XML content – both that provided by the publishers, and that produced by Portico in the workflow itself. This QA includes XML validation, the assertion (via Schematron) of other constraints on content values, and visual inspection of sample content. We have written about some of the QA techniques and challenges associated with these content files for Balisage and other venues (see, for example, [morrissey et al] and [morrissey 2011]). In this paper we would like to focus on the QA challenges associated with those XML registry files that drive our workflow.

Portico XML Configuration Files: A Description

The Portico workflow is a pluggable framework. At each step, or activity, in the workflow, the particular tool to be employed is dynamically selected, based on the format or mime type of the file or files being processed at that step. Thus, for example, the “de-layer” activity would invoke standard tar, gzip, or zip tools to expand and separate out the content of .tar, .gzip, or .zip files. The same activity would invoke an XSL transform to split a publisher XML file containing bibliographic metadata for all the articles in an issue of a journal into separate XML files for each article (and would invoke a different XSL transform for each different publisher XML format).

Figure 3

Figure 3: Format-driven Tool and Tool Component Plugins for WorkFlow Step

So, at the root of all the XML configuration files that drive and parameterize the Portico workflow is the format registry file: FormatRegistry.xml. There is a <Format> element for each distinct format for which the archive contains at least one instance. This, for example, is part of the <Format> element for one publisher’s profile of one version of the NLM Journal Publishing DTD:

  <Format FormatId="XXX_NLM_Journal_Publishing_DTD_2.1" 
          CreationTimestamp="2006-06-13T13:00:00-05:00">
	<PorticoDefinedName>XXX Journal Publishing DTD v2.1 20050630</PorticoDefinedName>
	…
  <Format>
                

For each workflow step or activity, the tool registry file, ToolRegistry.xml, maps each format to the Java class to be plugged in, configured, and executed at that step. So, for example, in the tool registry, we have entries such as:

  <TransformationSet>
	  <ToolStrategy SupportingFormatId="XXX_NLM_Journal_Publishing_DTD_2.1">
		  <Script Rid="scrxxx"/>				
	  </ToolStrategy>
	  ...
  </TransformationSet>
                

This entry indicates that in the workflow Transform Files step, if the Portico identifier (defined in the format registry) for the format of the file to be transformed is XXX_NLM_Journal_Publishing_DTD_2.1, then we should look for plug-in information about what tool to employ, and how to parameterize it, in a subsequent ScriptInfo element with a ScriptID attribute value of scrxxx. In that element, we will see specified such things as a list of relative file paths to XSL transforms that comprise the transformation pipeline for instances of this format, along with (relative) directory names where those files are located; the Portico identifier for the format of the output of this transformation (also defined in the format registry); a fully-qualified Java class name for a filter through which the input file is to be passed; and an Rid attribute referring to yet another subsequent element containing full information about the Java tool class that will invoke the filters and the XSL pipeline.

  <ScriptInfoSet>
  <ScriptInfo ScriptId="scrxxx" 
              ScriptType="transformation" 
              ScriptDir="xxx">
   <Tool Rid="BaseTransform_1.0">
    <Parameters>
     <Parameter>
      <Name>StyleSheetList</Name>
      <ValueOrderedList>
        <ValueOrderedListItem>
          <Number>10</Number>
          <Value>xxx2ptc_1_10.xsl</Value>
        </ValueOrderedListItem>
        <ValueOrderedListItem>
          <Number>20</Number>
          <Value>fix-data_1.xsl</Value>
        </ValueOrderedListItem>
        ...
      </ValueOrderedList>
    </Parameter>
    ....
    <Parameter>
      <Name>outputFormatId</Name>
      <Value>PTC_Article_DTD_2.1</Value>
    </Parameter>
    <Parameter>
      <Name>InputFilterClass</Name>
      <Value>
         org.portico.threadedtool.tool.transform.filter.XmlPrologTransformFilter
      </Value>
    </Parameter>
    ...
   </Parameters>
  </Tool>
 </ScriptInfo>
...
</ScriptInfoSet>                   
                

Later in the tool registry, information about the BaseTransform_1.0 tool is specified, including its Java class name, and information about the parameters to be passed when instantiating that class (as we did in the ScriptInfo element above), including whether or not the parameter is required, and what its type should be:

<ToolInfoSet>
   <ToolInfo Id="BaseTransform_1.0">
      <Name>BaseTransformTool:1.0:2007-05-01</Name>
      <Description>
         Tool for transformation of XML files via XSL stylesheets.
      </Description>
      <Status>ACTIVE</Status>
      <ClassName>
         org.portico.threadedtool.tool.transform.BaseTransformTool
      </ClassName>
      <ToolParameters>
        <ToolParameter Name="StyleSheetList" 
                       Required="true" 
                       ParameterType="ValueOrderedList">
          <Description>
             This is the list of XSL stylesheets that are to be processed and the 
             order in which they should be processed.
          </Description>
        </ToolParameter>
        <ToolParameter Name="outputFormatId" 
                       Required="true" 
                       ParameterType="Value">
          <Description>
             This identifies the output format ID of this set of transforms.
          </Description>
        </ToolParameter>
        <ToolParameter Name="InputFilterClass" 
                       Required="false" 
                       ParameterType="Value">
          <Description>
             This identifies the filter to be applied to the file before transforms.  
             It should contain the fully qualified class name of the filter. 
             If this Parameter is not supplied, the default BaseFilter 
             class is used. Legal values come from package
             org.portico.threadedtool.tool.transform.filter.
          </Description>
        </ToolParameter>
	...
      </ToolParameters>
    </ToolInfo>
    ...
<ToolInfoSet>
                

Portico XML Configuration Files: Categories of Configuration Issues

There are a lot of moving parts in even this condensed description of the semantics of these two configuration files. As more and more publisher streams were added to Portico’s workflow, more and more configuration information was added to these files, by more and more developers working at the same time to add new content streams and their accompanying Java tool and filter classes, and associated XSL transformations. And these additions had to be made in several different workflow environments: a developer environment; an integration environment where profiles and registries for new publisher streams are first worked out; a QA environment for regression testing of tool, transform, workflow, and configuration changes; and, finally, to the production environment.

Perhaps to no one’s surprise, Portico began to experience deployment glitches. None of these glitches occurred because the XML registry files were either not well formed or invalid, as developers consistently validated the files against their respective schemas before committing to Portico’s source control system, and the deployment scripts also invoked a parser to validate the files. The workflow was paused; new configuration files and other resources were deployed; the workflow was cranked back up. Then it would hum along through several workflow steps, before encountering what effectively was a configuration error that would bring one or more batches to a halt.

So the first question we asked ourselves was, what is it about the semantics of the content of the elements in these files – and the relationship among elements in the same and in different XML configuration files, and the relationship between the content in those elements and other components of the workflow software and other resource files – that enabled configuration errors to pass undetected through the sieve of standard XML validation tools?

Consistency Checker

The tool registry’s ToolInfo element documents all the possible calling parameters that can be passed to the Java tool class it specifies, indicating whether they are required or optional, and specifying the type of each parameter. Assuming correct documentation (for ensuring which, to probably no one’s surprise, we have not yet invented a completely automated tool), this meant that we had sufficient information in the tool registry itself to perform consistency checks. We use an XSL transform (Schematron is another obvious candidate for this) to compare the various Paramter elements in the ScriptInfo element that configures each invocation of a particular tool for a particular format, with the information in the ToolInfo elements. If an error message is created, the registry is not deployed until it is repaired.

“Referential Integrity” Checker

Again, we used an XSL transform. The transform extracts format and mime type information from the format registry being deployed with the tool registry, and uses that extracted information to verify that any referenced format id or mime type in the tool registry has been declared in the format registry.

Existence Checker

Our existence checker also uses XSL, aided by extension functions, to look outside the XML box to determine that expected directories and XSL and other files exist in the target deployment directories, and that classes referenced in the tool registry exist in the .jar files also on the deployment path. In the case of XSL and other XML files, it would of course be possible to use xsl-document to check for the existence of necessary resources. For non-XML file resources whose existence we wished to confirm, we could have used the unparsed-text() function, interpreting an empty string result as a non-existent file. As we were using Java extension functions to confirm the existence (i.e. deployment) of Java classes on what would be the workflow's runtime Java classpath, and as the workflow uses the classpath to resolve the location of XSL and other file resources, we used Java extension functions to confirm the existence of these resources at deployment.

 <xsl:stylesheet version="2.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
    xmlns:jexist="java/org.portico.conprep.util.deployment.validatetoolregistry.ExistenceChecker" >
  
  <xsl:variable name="jExistInstance" select="jexist:new()"/>
  
    <!-- ==================================================== -->
    <!--          name="check-class-exists"                   -->
    <!-- ==================================================== -->  
    <xsl:template name="check-class-exists">
        <xsl:param name="className"/>
        <xsl:param name="Path"/>
        <xsl:variable name="result" 
                select="jexist:classExistsOnClasspath($jExistInstance, $className)"/>  
        <xsl:if test="not($result)">
            <xsl:message  terminate="no" >
               Class  <xsl:value-of select="$className"/> does not exist.  
               <xsl:value-of select="$Path"/>
            </xsl:message>
        </xsl:if>
    </xsl:template>
    
    <!-- ==================================================== -->
    <!--           name="check-file-exists"                   -->
    <!-- ==================================================== -->  
    <xsl:template name="check-file-exists">
        <xsl:param name="scriptType"/>
        <xsl:param name="scriptDir"/>
        <xsl:param name="fileName"/>
        <xsl:param name="Path"/>
        <xsl:variable name="result" 
                select="jexist:fileExistsOnClasspath($jExistInstance, 
                         $scriptType, $scriptDir, $fileName)"/>  
        <xsl:if test="not($result)">
            <xsl:message  terminate="no" >
               File  <xsl:value-of select="$fileName"/> does not exist.  
               ScriptType = <xsl:value-of select="$scriptType"/>, 
               ScriptDir = <xsl:value-of select="$scriptDir"/>  
               Path to script = <xsl:value-of select="$Path"/>
            </xsl:message>
        </xsl:if>
    </xsl:template>