Balisage Paper: Using XML to Implement XML

The Process Manager

Overview

Most documents in the system are modularised, with a root XML file linking to other XML and images. The linked XML, in turn, often link to further XML and images, effectively creating a link tree.

Figure 1

A Link Tree

The Process Manager points out the root XML^[1] and maps the linked resources in a link tree representation, with status information, target IDs, versions, etc (see Figure 2). This info is stored in a configuration.

Figure 2

A Tree View of the Master-Language Document

The link tree is the same, regardless of document language, and therefore useful when handling translations. It can show what translations exist, help create new translation packages, etc.

Figure 3

Managing Translations

More importantly, the Process Manager uses the configuration to publish the link tree. It starts a process that fetches the necessary XML and images, and normalises them to one big XML file. It then applies a FO stylesheet to output PDF using Apache FOP. Here's the basic process handled by the Process Manager today to publish a modularised document.

Figure 4

Publishing Using the Current Process Manager

The four numbered steps at the top show what is actually exposed to the user. They hide a lot of background processing, including several XSLT conversions, DOM manipulation, etc. Everything is glued together using C#. The GUI itself is static and always more or less the same.

Other processes might output, say, HTML or ePUB, or simply a report, but currently, only copying the original process, translation handling and PDF publishing, all of which depend heavily on C# code, are supported.

URNs Rather Than URLs

Everything is identified, handled and linked to using URNs. Whether a link points out XML or an image, the pointer is a URN. The URNs are unique and include language/country and version information, so there's full traceability; link trees can always be recreated later, regardless of what's been done to a module or image since.

This works well. The URNs define basic semantic documents (with base URNs), master language documents (base URN plus the master language/country) and their translations (the base URNs plus the target language/country)^[2].

Figure 5

Everything Is Identified Using URNs

Stylesheet modules, while handled in the system in the same way as any other XML, using URNs, are developed outside the system and use relative URLs to refence each other. Thus, the current Process Manager can only use them in the temp location when publishing, because otherwise the relative URLs would break.

Process Details

The Process Manager can currently do the following:

• Create and save new configs, with titles, descriptions and other metadata, and point out the exact version and language of a root XML document using its URN.

• Open and edit an existing config.

• Copy an existing configuration and the root XML pointed out by it, that is, create a fork of the XML^[3].

Figure 6

A Saved Configuration

A configuration is used by the Process Manager to process the document it identifies:

• View the document as a link tree, including any linked resources.

• Create a translation package of the document, selecting the target language and the modules to be included.

• Publish the document. Currently, only PDF.

• Profile^[4] a copy (fork) of the document, setting conditions that include or exclude content from the output^[5].

Copying Configurations and XML

In terms of pipeline processing, this is what happens when copying and profiling a configuration:

• Copy the root XML^[6].

• Copy the configuration (copied config points to copied XML).

• Change the filenames of the copied XML and config.

• Generate new URNs for the copied config and XML^[7].

• Profile the root XML copy using a GUI in the Process Manager, adding and removing values controlled by the CMS.

• Insert the selected profiles as attribute values in the copied XML's root element.

• Store both copies in the CMS.

Figure 7

A Forked Configuration

The copied XML is then processed.

Problems

The Process Manager is written in a non-XML way in C#, by developers who understand C# but not all that much XML. It uses XSLT for the transforms, but if I want to use more than one set of stylesheets, say for HTML or just a different-looking PDF, I need developers to write more C# to put it all together because I don't speak C#. I speak XML.

The GUI is static, too. While it is sufficient when presenting PDF publishing options, any other processing, be it validation, HTML output or something else, requires changing it, meaning more C# code and (for me) a dependency to developers.

Any additional steps, for example, cross-reference validation of the normalised XML document before publication, also require coding.

All sorts of things (XSLT, DOM APIs, etc) are used to manipulate XML, of course, but all is glued together using C# code, called using C# code, and changed using C# code, with whatever methods there are in that language for manipulating XML.

Suffice to say, most changes require C#.

Some less obvious problems become apparent when handling XSLT:

This applies to any set of XML files that needs to be handled as a package by a process in the CMS.

A First Attempt at XProc

The Process Manager, of course, should be a showcase for XProc. Replace the current functionality with an XProc engine that runs pipelines for publishing and an admin interface to add and manipulate pipelines and stylesheets with, without having to resort to more C# code. And, of course, a GUI for the end users to select pipelines from.

The developers agreed. Their lives would be easier, too. If XProc pipelines can take care of any XML processing while leaving the non-XML stuff to the middleware, everybody wins.

Manipulating pipelines efficiently was far more complex than we first thought, however. We needed a GUI for the XProc engine, ways to add parameters and input to it, including pointing out modularised stylesheets and other XML, etc. This required more C#, not less.

Also, XML pipelines are only as flexible as the user interfaces to run them with. I could think of using and reusing pipelines in more ways than would be practical to present in any GUI. The time estimate for the admin functionality alone was far bigger than our initial guesses, but the end user interface added enough hours to the point where the project was turned down.

So I started thinking about ways to generalise all this, to accomplish it all without admin GUIs and a million lines of new C# code. This paper presents what I came up with.

XProc and Calabash Command Lines

Here's an example Calabash command line to handle a simple publishing process in the CMS's temp location:

java com.xmlcalabash.drivers.Main 
-isource=[runtime URN/URL] 
-istylesheet-fo=[URN/URL for FO main stylesheet] 
normalize-stylesheet=[URN/URL string for normalize]
pdf=[runtime PDF URN/URL]
[URN/URL for FO XPL]

Most inputs are URNs mapped to temp URLs as part of the publishing process, but others are generated at runtime. Generating a .bat file directly from today's system is doable, but since XProc allows writing pipelines in many ways, it is hard to do consistently. For example, input to the pipeline can be handled in several ways. Note how the normalize stylesheet is defined using an XProc option rather than the usual input port:

...
<p:input port="stylesheet-fo"/>
<p:option name="normalize-stylesheet" required="true"/>
...

Taking this kind of variation into account when creating an admin interface for handling pipelines would cause problems, as would, for example, optional parameters. User-selectable options (such as stylesheets) would further complicate things. Etc.

If we expressed the command line options in XML, generating the .bat file would be far more straight-forward:

<pipeline>
    
  <!-- XProc script for PDF -->
  <script href="[URN/URL for FO XPL]"/>
  
  <!-- Calabash command line -->
  <cmdlines>
    <cmdline>
      <inputs>
        <input>
          <port>source</port>
          <value type="ti">[runtime URN/URL]</value>
        </input>
        <input>
          <port>stylesheet-fo</port>
          <value type="uri">[URN/URL for FO main stylesheet]</value>
        </input>
      </inputs>
      <options>
        <option>
          <name>normalize</name>
          <value type="uri">[URN/URL for normalize main stylesheet]</value>
        </option>
        <option>
          <name>pdf</name>
          <value type="ti">[runtime PDF URN/URL]</value>
        </option>
      </options>
    </cmdline>
  </cmdlines>
</pipeline>

This describes a single command line for one specific script. Other command lines for that same script might use other stylesheets, optional input, XProc engine options, etc. My basic DTD can easily include such variations:

Figure 8

The Pipeline Structure

The pipeline element allows for different command line alternatives to be inserted or linked to^[10]. A GUI could easily present them as user-selectable options. Note the extension mechanism to provide some support for changes to XProc (perhaps a step imported from another namespace) and the engine (see Figure 9). The extension capabilities are limited, however; any addition requiring preconfiguring or input beyond the basics will cause problems (see section “Limitations and Other Considerations”.

Figure 9

Extensions

Here's an example with several command lines:

<pipeline>
  <metadata>
    <title>PDF Output</title>
    <description>This process outputs PDF.</description>
  </metadata>
  <!-- XProc script for PDF -->
  <script href="[URN/URL for FO XPL]"/>
  <cmdlines>
    <cmdline id="cmd1">
      ...
    </cmdline>
    <cmdline id="cmd2">
      ...
    </cmdline>
    <cmdline id="cmd3">
      ...
    </cmdline>
  </cmdlines>
</pipeline>

A command line alternative is easily expressed as /pipeline//cmdline[@id='cmd1'] and so easily transformed to a .bat file once selected. Also, the alternatives are easy to represent in a GUI, for example, with check boxes^[11] or a list of choices. The pipeline's metadata element contents can be used to add labels and help texts to the GUI^[12].

A pipeline structure instance with its associated command lines^[13] is a blueprint for possible ways to use the pipeline. There is no one-to-one correspondence with an actual XProc script; rather, the XML represents the available choices. For example, several command line alternatives might exist for a pipeline, a command line might include several alternative input stylesheets, etc. A user would have to make choices for each option before running the pipeline.

The GUI needs to reflect this by only offering such choices once the prerequisite choices have been made (for example, you'd have to select a command line before selecting a stylesheet).

The input step in the below example points out the root XML, but the value is not known until runtime. The attribute type="ti" means that the system is expected to provide the content^[14].

<input>
 <port>source</port>
 <value type="ti">
  <!-- Runtime URN/URL -->
 </value>
</input>

Packages

A package is a collection of resources (XML, XSLT, etc) identified with URNs. As the URNs identify specific versions (and languages), a package can identify an arbitrary but well-defined set of files in the database. Usually but not always^[15], there will be a main file that links to the others. Here's an example FO stylesheet package:

<package id="id-fo-package">
  <metadata>
    <title>XSL-FO Stylesheet Package</title>
    <description>FO stylesheets for
      Company X.</description>
  </metadata>
  <locator href="urn:x-example:xslfo:0012:3" type="root"/>
  <locator href="urn:x-example:xslfo:0019:10"/>
  <locator href="urn:x-example:xslfo:0117:21"/>
  <locator href="urn:x-example:xslfo:0010:1"/>
 </package>

The package consists of URN-based links to the stylesheet files, plus some metadata for the GUI. Also, the main stylesheet file is identified with a type attribute with the value root.

Figure 10

A Package Is Basically A Set of Links

If the package XML is identified with urn:x-example:packages:123456:en-GB:1 in the CMS (the last part identifies the package version, 1), the relevant input in the initial pipeline processing XML from the last section is this:

<input>
  <port>stylesheet-fo</port>
  <value type="uri">urn:x-example:packages:123456:en-GB:1</value>
</input>

When generating the pipeline's .bat file, the whole package, including the package XML, is copied to the temp location, with URNs in the package XML replaced with corresponding URLs (for example, urn:x-example:xslfo:0012:3 with main-fo.xsl) . Each user-selectable stylesheet alternative in a pipeline's command line (see section “XProc and Calabash Command Lines”) would reference a different such package.

Packages, then, provide the necessary abstraction layer to define input from the CMS to XProc pipeline processes.

As we've seen, the XML that describes the packages is fairly static, easily definable beforehand. With more frequent package updates, however, it might be useful to call the package with a wildcard in place of the URN version string:

urn:x-example:packages:123456:en-GB:*

The wildcard means use latest version with workflow status approved for a user with technical author permissions, but use latest version with workflow status reviewing for someone with test privileges. The permissions control how URNs are processed, significantly lessening the need to change or update system behaviour for small updates^[16].

Finally, the package abstraction very neatly separates offline versioning (in subversion, in my case) of the participating files from their online versioning. The latter are a conscious decision from the developer to release a set of files, while the former is simply work in progress.

Putting It All Together

The Process Manager is meant to handle processes (the name sort of gives it away). Normally, a complete process and the primary action that process performs (such as outputting a PDF) are largely the same, but until now, it has not been possible to produce variants of the process in any case.

It should be. For example, two different sites sharing the same CMS might use almost the same basic process to publish documents with, with the same basic pipeline and options, but use different stylesheets for their respective brands. It would be useful to glue the pipelines and packages together to describe such variations.

Here's a basic XML structure defining two processes A and B that use more or less the same pipelines, command lines and packages:

<processes>
    <!-- Process for A -->
    <process id="id-process-A">
      <metadata>
        <!-- Metadata for A Process -->
      </metadata>
      <!-- Pipelines for A -->
      <pipelines>
        <pipeline>
          <metadata>
            <!-- Metadata for pipeline for A -->
          </metadata>
          <!-- XProc script for A FO -->
          <script href="a.xpl"/>           
          <cmdlines>
            <cmdline>
              <!-- Command line for A FO -->
            </cmdline>
            <cmdline id="id-cmdline-fo-a-and-b">
              <!-- Command line for A and B -->
            </cmdline>
          </cmdlines>
        </pipeline>
      </pipelines>
      <!-- Packages available for A -->
      <packages>
        <!-- Normalize stylesheet reference -->
        <locator href="packages.xml#id-norm"/>
        <package id="id-fo-a">
          <!-- FO stylesheet package for A -->
        </package>
        <!-- Link to A and B FO stylesheet package -->
        <locator href="packages.xml#id-fo-a-b"/>
      </packages>
    </process>
    
    <!-- Process for B -->
    <process id="id-process-b">
      <metadata>
        <!-- Metadata for B Process -->
      </metadata>
      <!-- Pipelines available for B -->
      <pipelines>
        <pipeline>
          <metadata>
            <!-- Metadata for B pipeline -->
          </metadata>
          <!-- XProc script for B FO -->
          <script href="b.xpl"/>
          <!-- Links to command lines -->
          <cmdlines>
            <!-- Link to A and B FO cmdline (above) -->
            <locator href="#id-cmdline-fo-a-and-b"/>
          </cmdlines>
        </pipeline>
      </pipelines>
      <!-- Packages available for B -->
      <packages>
        <!-- Normalize package ref -->
        <locator href="#id-norm"/>
        <!-- Link to A and B FO package -->
        <locator href="packages.xml#id-fo-a-b"/>
      </packages>
    </process>
</processes>

Both A and B link to packages.xml that defines common packages:

<packages>
  <package id="id-norm">
				<!-- Normalize stylesheet package  -->
  </package>
  <package id="id-fo-a-b">
				<!-- FO stylesheet package for A and B -->
  </package>
</packages>

A processes instance is a blueprint for every possible allowed process, each of which is associated with one or more possible pipelines. The user has to select a process before selecting from the pipelines for that process.

Figure 11

Putting It All Together

The design is modular, so everything from command lines to packages can be reused. The modules are identified with URNs, just as everything else in the system, offering traceability and allowing whole processes to be repeated when required.

Writing This Paper

This whitepaper, written in our system, provided me with further motivation:

The additional processes (URN/URL conversion, export to DocBook, zip) are simple enough to implement, but right now I'd need a C# developer to do it. I'd have to explain what I want done and why, and the GUI would need changing. And it would all have to be done again if I wanted to add some other processing.

Generating GUIs

The processes structure is a blueprint describing every available process to manipulate XML documents with. Every process is associated with one or more pipelines, run using one or more command lines, and each command line may allow for one or more stylesheets.

An XForms (or HTML5) GUI generated from the process XML blueprint can help a user visualise these choices. Figure 12 provides a blueprint example. The Delivery process, for example, can use three pipelines but Validation only one. The Printed pipeline can only use one command line, but the Web pipeline four. Etc.

Figure 12

Process Selection Relations

Using XSLT for GUI generation is the obvious choice as the underlying database is relational, SQL rather than XML, and there's no XQuery implementation^[18].

The process XML requires input from the system for pointing out the root XML document to be processed. My knowledge of the JScript-based tree controls we use to select XML in a web browser is limited, but it is supposedly easy to query the tree and get lists of the XML (as URNs) and other resources, including translations and older versions of the listed XML.

Figure 13

Selecting the Root XML Document

The root XML selection is used as a starting point for pipeline processing. The next step is to generate a GUI from the process XML blueprint. Figure 14 shows a GUI example that follows the logic in Figure 12.

Figure 14

Process Selection Page Generated From XML

Apologies for the crude mock-up. Generating new selection lists based on a previous selection can probably be represented better, but hopefully the principle is clear. The labels are fetched from the process metadata, as are help texts. Also, note the checkbox to enable logging; this is a command line alternative^[19].

The selections provide data to refine the process instance. I'm not enough of a programmer, nor sufficiently familar with XForms, to tell you exactly how this is handled best. For example, the GUI needs to handle a Back or Previous button, Undo, and more, and I don't know what the best way to implement those would be. Here, I work from the assumption that the user's choices can be stored in a temporary process XML instance, used to generate the batch file to run the actual pipeline.

Generating .bat Files

Pipelines are run using batch files generated from the process XML using XSLT, but first, the process XML must be narrowed down to an instance defining only a specific pipeline and options, as described above.

Generating the batch file:

The batch file then runs the selected pipeline and its options, parsing the root XML document and any linked resources for URNs, downloads the associated resources, eventually replacing URNs in links with temp URLs, and runs any XSL and other processing as specified by the pipeline.

Writing Process XML

When writing a new pipeline script (or editing an existing one), it is useful to simultaneously edit the process XML blueprint that will include the pipeline. Command line options might require updating or a new XSLT stylesheet might be needed with the pipeline. It makes sense to put the new pipeline in context so whatever options it has are handled.

The process XML is edited in the same authoring environment that is used to write documents. The markup uses the same linking mechanism as the XML documents—XLink with URN-based links—so only some additional CSS styling and macros for editing are required; the rest is already in place. The results can be checked into the CMS, just as any other XML, with very few modifications^[21].

Limitations and Other Considerations

Some notable problem areas:

Other parts are easier:

Many extensions (see XML Calabash (also known as Stop Censorship), http://xmlcalabash.com/docs/reference/extensions.html) fall into the engine domain as they happen completely behind the scenes; if the engine supports it, the process XML will, too, if they can be run transparently, without having to preconfigure the engine in some way. If they require engine configuration, setting system properties or specific input on the command line, they will fail as anything depending on setting a Calabash system property or configuration on the command line is currently not supported.

Educating Users

The final point I want to make is perhaps the most important one. The process XML is very useful when educating users about a process, without requiring detailed knowledge about XProc, XSLT, or some other processing. It allows us to represent the process with just enough information for an overview because it black-boxes every component.

For example:

And More

I'm thinking about adding: