Balisage Paper: Marking up changes to ISO standards: A case study

Introduction

The ISO Standards Tags Set (ISOSTS) [isosts] is a customization of NISO’s Journal Article Tag Suite (JATS) developed by Mulberry Technologies [mulberry] for the International Standards Organization for authoring standards documents. Documents authored in this format can then be converted into multiple publishing formats such as Adobe InDesign, PDF, HTML or EPUB, using a solution from Typefi called Typefi Publish [typefi-publish]. As part of the authoring workflow used at ISO, there was a requirement to produce documents in these published formats that displayed changes between different versions of a standard using redlining. Redlining is a technique for marking changes in a document, typically using text styling to highlight deleted and/or added content. For example, all deleted text could be highlighted by colouring it red and striking through the text. Added content could be highlighted with an underline. It is also common practice to highlight only added content and mark the position of deleted content using a caret such as ‸ or ⁁.

Typefi approached us and asked us to be involved in producing a proof of concept, building on top of their existing solution for ISO. Our expertise in XML comparison and change representation matches perfectly with Typefi’s expertise in content layout and document publishing to provide a solution to this new requirement for ISO. Our contribution included producing a tailored comparison of ISOSTS documents, ignoring certain types of change that were not important to ISO. The result from this is fed into Typefi Publish which handles the DeltaXML change representation format, DeltaV2, to produce the redlined final output. As an alternative, we also provided modifications to ISO XSLT stylesheets that convert ISOSTS into XHTML directly. These modifications used CSS to provide the redline change highlighting.

Document Comparison Overview

Comparison is a key component of any system that is dealing with documents that change during their lifetime. Understanding the differences between different versions of a document is absolutely vital and, in some industries, can even be a legal requirement. It is therefore important to understand the implications for document comparison when those documents are stored in an XML format such as ISOSTS.

Because of the structure and syntax of XML, line based comparison tools can often produce incorrect comparison results. Line based tools are often unaware of XML syntax, and changes which can usually be ignored, such as indentation, namespace prefixes, and attribute ordering can lead to false notification of changes. Consider the following two XML documents and the line-based comparison provided by a UNIX diff.

<document xmlns="demo-namespace">
  <title>This is the document title</title>
  <p>An example paragraph</p>
  <p>Lack of namespace awareness makes line based diff ineffective</p>
</document>

<demo:document xmlns:demo="demo-namespace">
  <demo:title>This is the document title</demo:title>
  <demo:p>An example paragraph</demo:p>
  <demo:p>Lack of namespace awareness makes line-based diff ineffective</demo:p>
</demo:document>

1,5c1,5
< <document xmlns="demo-namespace">
<   <title>This is the document title</title>
<   <p>An example paragraph</p>
<   <p>Lack of namespace awareness makes line based diff ineffective</p>
< </document>
---
> <demo:document xmlns:demo="demo-namespace">
>   <demo:title>This is the document title</demo:title>
>   <demo:p>An example paragraph</demo:p>
>   <demo:p>Lack of namespace awareness makes line-based diff ineffective</demo:p>
> </demo:document>

The result of this particular line based comparison gives no advantage over visually inspecting the two documents to detect the differences. The tool’s lack of understanding of XML syntax means that it detects far too much difference in the two documents.

It is possible to improve the results of a line based comparison by canonicalizing the documents to be compared to ensure consistent use of namespace prefixes, defined attribute ordering, and consistent indentation. This will go some way to reducing the amount of false differences identified by line based tools but there are still limitations with what can be achieved. Change identification at the word level is potentially still problematic and if the documents reference a DTD for example, correct parsing of the documents is necessary for a comprehensive comparison since parsing may actually affect the content of the document itself (this is covered in more detail in section “Infoset Augmentation”).

An XML aware tool has the advantage of being able to parse the document, taking into account any references to external content (e.g. DTDs, XIncludes) where necessary. As well as parsing, it is then a simple task to add pre-processing of documents using various XSLT steps to perform tasks such as segmenting sentences into individual words. If we use an XML aware tool such as DeltaXML Core to compare the two documents, we get a result more like that shown in Figure 4. This is an HTML rendering of the underlying result format produced by DeltaXML Core.

Figure 4: Result of comparing Figure 1 and Figure 2 with DeltaXML Core

<document xmlns:deltaxml="http://www.deltaxml.com/ns/well-formed-delta-v1"
          deltaxml:deltaV2="A!=B" deltaxml:version="2.0" deltaxml:content-type="full-context">
  <title deltaxml:deltaV2="A=B">This is the document title</title>
  <p deltaxml:deltaV2="A=B">An example paragraph</p>
  <p deltaxml:deltaV2="A!=B">Lack of namespace awareness makes line
  <deltaxml:textGroup deltaxml:deltaV2="A!=B">
    <deltaxml:text deltaxml:deltaV2="A"> </deltaxml:text>
    <deltaxml:text deltaxml:deltaV2="B">-</deltaxml:text>
  </deltaxml:textGroup>
  based diff ineffective</p>
</document>

Making use of DTDs and schema

We encountered issues with some of the ISO documents we tested as the use of DOCTYPE instructions was not consistent. For a given pair of documents, for example, one included a DOCTYPE instruction but the other did not. This led to issues, particularly with infoset augmentation, described below. The ISOSTS standard does not explicitly state the intended use of the DTD and whether it should be included in instance files in a DOCTYPE instruction. This could lead to different behaviour for documents provided by different implementers. Many specifications, for example the XHTML specification [xhtml-spec], explicitly state that a conforming document MUST include a DOCTYPE instruction.

While many XML authors will understand that a DOCTYPE instruction has “something to do with validating” the XML they are writing, they will not always be clear about the full implications of this instruction. While it is true that a DTD provides validation of the document being authored, it also has implications on the meaning of whitespace in a document and on the appearance and/or value of certain attributes.

<document>•
••<p>This◦text◦contains◦<b>bold◦and</b>*<i>italic</i>◦text</p>•
</document>

<!ELEMENT document  (p)* >
<!ELEMENT p         (#PCDATA | b | i)* >
<!ELEMENT b         (#PCDATA) >
<!ELEMENT i         (#PCDATA) >

Table Comparison

The ISOSTS specification uses the XHTML table model to define how tables are declared. While not as complex as the CALS table model, there are still significant issues with this model if tables are compared as ‘plain’ XML, without knowledge of the table structure. One example of this is adding row spanning to a cell.

<table>
  <tbody>
    <tr><td>Cell 1</td><td>Cell 2</td></tr>
    <tr><td>Cell 3</td><td>Cell 4</td></tr>
    <tr><td>Cell 5</td><td>Cell 6</td></tr>
  </tbody>
</table>

<table>
  <tbody>
    <tr><td rowspan="2">Cell 1</td><td>Cell 2</td></tr>
    <tr><td>Cell 4</td></tr>
    <tr><td>Cell 5</td><td>Cell 6</td></tr>
  </tbody>
</table>

<table deltaxml:deltaV2="A!=B" ...>
  <tbody deltaxml:deltaV2="A!=B">
    <tr deltaxml:deltaV2="A!=B">
      <td deltaxml:deltaV2="A!=B">
        <deltaxml:attributes deltaxml:deltaV2="B">
          <dxa:rowspan deltaxml:deltaV2="B">
            <deltaxml:attributeValue deltaxml:deltaV2="B">2</deltaxml:attributeValue>
          </dxa:rowspan>
        </deltaxml:attributes>
        Cell 1
      </td>
      <td deltaxml:deltaV2="A=B">Cell 2</td>
    </tr>
    <tr deltaxml:deltaV2="A!=B">
      <td deltaxml:deltaV2="A">Cell 3</td>
      <td deltaxml:deltaV2="A=B">Cell 4</td>
    </tr>
    <tr deltaxml:deltaV2="A=B">
      <td>Cell 5</td>
      <td>Cell 6</td>
    </tr>
  </tbody>
</table>

As can be seen, the resultant table does not render well as the second row now includes too many cells, thus pushing Cell 4 too far to the right. A better result would be to handle the change to row spanning by including the problematic rows from the original table, marked as deleted, followed by the matching rows from the modified table, marked as added. This can be seen in the example below.

<table deltaxml:deltaV2="A!=B" ...>
  <tbody deltaxml:deltaV2="A!=B">
    <tr deltaxml:deltaV2="A">
      <td>Cell 1</td>
      <td>Cell 2</td>
    </tr>
    <tr deltaxml:deltaV2="A">
      <td>Cell 3</td>
      <td>Cell 4</td>
    </tr>
    <tr deltaxml:deltaV2="B">
      <td rowspan="2">Cell 1</td>
      <td>Cell 2</td>
    </tr>
    <tr deltaxml:deltaV2="B">
      <td>Cell 4</td>
    </tr>
    <tr deltaxml:deltaV2="A=B">
      <td>Cell 5</td>
      <td>Cell 6</td>
    </tr>
  </tbody>
</table>

This is one example of the way that tables are handled intelligently during the comparison phase. As mentioned above, the XHTML table model is simpler than the CALS table model leading to fewer potential issues during comparison, but there were still a number of problems that needed to be solved.

Text formatting changes

Changing the format of specific pieces of text, e.g. highlighting a word by making it bold or italic, is common during text editing but should this constitute a change in a redline document? The answer will depend on the context of the change, whether the subject domain places meaning on such formatting, and whether or not there is a requirement to see these kind of changes in the redline document. In the case that it should be highlighted, there may be different ways of doing so. The document reviewer may wish to see the text with its old formatting marked as deleted and the text with its new formatting marked as added so that a complete view of the change is present. In other situations, it may be sufficient to mark the text with some other kind of highlighting to show that there has been a formatting change but not include details of how the formatting has changed.

Many content authors may not even understand that there is an XML structure underlying their document and that a format change actually constitutes a structural change. Thus, when they make a word bold and the resultant comparison result shows the word deleted and then added again, they see this as a mistake.

<p deltaxml:deltaV2="A!=B" ... >The addition of
  <deltaxml:textGroup deltaxml:deltaV2="A">
    <deltaxml:text deltaxml:deltaV2="A">bold</deltaxml:text>
  </deltaxml:textGroup>
  <b deltaxml:deltaV2="B">bold</b>
  formatting.
</p>

In order to have the ability of marking formatting changes in a different way, or in fact ignoring them completely, we need to have some way of detecting the structural change without having to mark the underlying text as changed as well. One technique we have utilised for this is to pre-process the documents to flatten the structure of formatting elements. The following example shows a document with a bold word that has had its formatting flattened.

<p xmlns:deltaxml="...">
  The addition of 
  <deltaxml:format-start>
    <deltaxml:element><b/></deltaxml:element>
  </deltaxml:format-start>
  bold
  <deltaxml:format-end/>
  formatting.
</p>

This flattened structure can handle formatting elements that are a simple tag, e.g. <b/> or <i/> and also more complex formatting such as <span style="font-size:14; font-weight:bold;"/>. Processing the input documents in this way then allows the text to be compared more intuitively, as it is all at the same level in the XML structure. Format changes are detected as changes to the <deltaxml:format-start/> and <deltaxml:format-end/> elements and the structured formatting can be reconstructed after comparison. There is the potential for overlapping structures in the result when formatting is flattened; to solve this problem, the formatting from one of the input documents, typically the latest or ‘B’ document, is given priority when reconstructing.

ISO’s requirement was to ignore formatting changes completely and, for content that was in both input documents, to include the formatting from the latest or ‘B’ document. This makes reconstructing the formatting elements a lot simpler because in the case where formatting has changed it is possible to ignore all of the elements marked as being only in document ‘A’.

ID and IDREF attributes

ID attributes and their associated IDREFs are typically used for internal cross-referencing in documents. It is important that the target of a cross-reference is declared as an attribute having type ID in order to ensure uniqueness within the document. Unfortunately, this uniqueness constraint can cause problems in the result file, which must be overcome. Imagine the situation where an image, e.g. an <img/> element, is used to display a diagram and defines an ID, e.g. <img xml:id="widget"/>. An editor of the document decides that this should have been defined using a figure element but, to avoid having to update references to the diagram, uses the same id: <fig xml:id="widget"/>. This is all perfectly valid because each document maintains uniqueness of its IDs. However, the comparison result file will contain the following content because of the requirement to view both added and deleted content in the same document.

...
<img deltaxml:deltaV2="A" xml:id="widget" />
<fig deltaxml:deltaV2="B" xml:id="widget" />
...

The document now contains two elements with the same ID value, which makes it invalid. This situation can be resolved by renaming the IDs on any deleted, or ‘A’ document elements and also updating any references to that element (these will be elements in the ‘A’ document only, that contain an IDREF whose value is the ID in question). The following figure shows an example of a fixed result file.

<document deltaxml:deltaV2="A!=B" ... >
  <img deltaxml:deltaV2="A" xml:id="widget_deleted_1" />
  <fig deltaxml:deltaV2="B" xml:id="widget" />
  <p deltaxml:deltaV2="A">This reference to the img will be deleted <xref linkend="widget_deleted_1" /></p>
  <p deltaxml:deltaV2="A=B">This reference will be kept in the new document <xref linkend="widget" /></p>
</document>

This document is now valid in respect of its ID uniqueness. The deleted first paragraph contains a reference to the old diagram as that is what it was referencing. The remaining second paragraph now points to the new version of the diagram. The naming scheme for updating deleted ID attributes can ensure uniqueness by using a number suffix that does not exist in the document. This can be checked against all existing IDs in the document.

Another potential use of ID values is to use them during comparison to align elements of the same type with matching IDs. This can improve comparison results, particularly for documents that include repeated sentences and phrases as can be typical in legal documents for example. For this technique to work, an element must maintain its ID value across different versions of the document so that its identity is consistent. Many XML documents are auto-generated from some other format and part of this process will involve the generation of ID values. If these are randomly generated, they will not be suitable for this use as equivalent elements in different versions of a document will not have the same ID. Even if they are not random and use a naming scheme, e.g. fig1, fig2, fig3 etc., removal of an element in this sequence could have a ripple effect on the ID values for all subsequent elements, again making them unsuitable for use during comparison. This was the case for the ISO documents and the ripple effect of ID values changing caused a large amount of change to ID attributes that had to be handled using the technique above.

Processing Instructions

Processing instructions are used to supply a consuming application with information. One thing they are increasingly used for is to insert data and/or content into a document format that does not allow for that content in its model. This is a way of providing a customized extension to a document format but is often used as a quick fix when a more appropriate solution would be to add the required functionality to the language specification. An example of this is the use of a processing instruction to specify the size at which a table should be rendered on a page. In the ISOSTS documents we tested, we saw the use of processing instructions to specify an external image location that could have been included as an attribute, e.g. <img><?img-id D09291AZ.PNG?></img> instead of <img href="D09291Az.PNG"/>.

One of the problems this causes is that if you compare documents containing such processing instructions and you want the result file to include the processing instructions, there is no sensible way of representing change to them as they are not XML elements. It is possible to preserve processing instructions, and even detect change in them by first converting them into an XML structure, comparing documents, and then converting the XML structure back into processing instructions. A potential solution to representing change is to duplicate the containing element whenever a change is detected in a processing instruction. For example, and <img/> containing a processing instruction as above with a change to the external location of that image could be represented as an image deletion and addition e.g.

...
  <img deltaxml:deltaV2="A"><?img-id old-image.png?></img>
  <img deltaxml:deltaV2="B"><?img-id new-image.png?></img>
...

This solution is not as good as being able to represent change to an href attribute as it is not as easily processed but it provides a reasonable result. This can, however, be problematic if the element containing the processing instruction is very large, e.g. a table containing a processing instruction that gives information on how it should be rendered. Including two versions of the whole table in order to represent the processing instruction change does not give a sensible result.

Word Capitalization

Word capitalization, like formatting change, is often viewed as an insignificant change that should not be highlighted in a redline document. This was indeed the case with ISO’s requirements. Like formatting, the result document needed to include the version of the text that was in the latest, or ‘B’ document.

A potential solution to this problem is to pre-process the input documents to ensure that all text uses only lower case. For documents whose text is mainly prose, this is not appropriate as upper case letters are an important feature of the text and should be preserved during comparison. Because pre-processing the inputs in this way does not make sense for the ISOSTS documents, the solution was to post-process the result file to detect those text changes where the only difference between the two versions was letter case. The following figure gives an example of the kind of change that can be detected.

<p deltaxml:deltaV2="A!=B" ... >
  Word capitalization is often seen as an
  <deltaxml:textGroup deltaxml:deltaV2="A!=B">
    <deltaxml:text deltaxml:deltaV2="A">insignificant</deltaxml:text>
    <deltaxml:text deltaxml:deltaV2="B">Insignificant</deltaxml:text>
  </deltaxml:textGroup>
  change.
</p>

A text-based comparison of the ‘A’ and ‘B’ branches of the <deltaxml:textGroup/> element after converting both strings to all lower-case, shows that there is no change. In this situation, we can remove the marked changes and include only the text from the ‘B’ document.

This technique works well for the cases where a text change is purely a capitalization change. More complex changes that involve capitalization in conjunction with addition and/or deletion of surrounding words will still include the capitalization change in the final output. As the capitalization is part of a larger change which will need to be reviewed anyway, this is not likely to be a significant inconvenience.

HTML change visualization

As well as the ISOSTS specification, ISO provide XSLT stylesheets that convert an ISOSTS document into standalone XHTML. These stylesheets provide a useful and simple way of producing a published version of standards documents for previewing during authoring. They can also be used to publish an online version of a standard.

As well as providing the intermediate change representation for input into Typefi Publish, we were able to extend the XSLT stylesheets to provide some redline functionality in the XHTML output. In the simplest cases, this involved first categorizing the elements in ISOSTS as either block-level or inline elements and then extending the output templates to wrap block-level elements in a <div/> and inline elements in a <span/> with these wrappers defining a class attribute containing the value of the intermediate result’s deltaV2 attribute where it was ‘A’ or ‘B’. These classes were then styled using CSS to highlight deletions with a red background and additions with a green background.

Other cases were more complicated and involved the overriding of whole processing templates in the original XSLT but the final result was a useful rendering of redlining in XHTML.

Results

The following figures show an excerpt from each of the different types of redline result that were produced. The PDF result was produced using the intermediate result delta, published through Typefi Publish and the HTML result was produced by transforming the intermediate delta file using our XSLT extension to the ISO stylesheets. Unfortunately, images were not available for the HTML output at the time of writing.

Figure 18: A PDF rendering of the redline result

Figure 19: An HTML rendering of the redline result

Summary

Document comparison is a key part of any workflow involving changing documents and, with more and more documents being stored as XML, it is important to provide tools that understand the XML structure and the implications that it has on comparison results. As we have demonstrated, there are many subtle areas to consider when looking at XML comparison and change representation and many of the problems we have encountered could have been made simpler by designing the document formats with comparison and change representation in mind. This case study shows that the problems arising during comparison of structured content are not insurmountable and those considering moving to an XML representation for their document storage should not be reluctant to do so based on any of issues seen here.

Structured content offers huge benefits, not least of which is the processability of content to multiple published formats. This case study has shown that the production of an intermediate document containing change representation can be used to produce redline documents in both PDF and XHTML. This intermediate file can quite easily be further processed to select the types of change which should be highlighted and those which should be ignored.

Coupling this technology with Typefi Publish, which provides the flexibility of multiple output formats and professional layout and design capabilities provided ISO with a comprehensive solution to their requirements for published redline documents.