Balisage Paper: Pipelined XSLT Transformations

Basics

Imagine you need to transform a Microsoft Excel 2013 spreadsheet to DocBook. The input is nothing like your final output; there is no structure and the input's got no clear semantics. The input is a mess of word processor formatting and weird namespaces, and your actual contents are either hidden in multiple t and v elements, or not visible at all on first sight, seemingly placed there at random. And imagine that you'll have to use traditional XSLT development to do it. I'm sure you can do it, if you're a reasonably experienced XSLT developer with some insight into the xlsx format, but you'll experience pain like none you've known before.

Now, imagine you'd instead be able to do it step by step, with one step's output being the input to the next: first, you remove anything that isn't needed, second you include the shared strings into your cells, third, you normalise your hyperlinks, and so on (where and so on means pretty much every actual conversion step). You'd do it all in steps that use the previous one's output as input.

Wouldn't that be wonderful?

Ideally, such a transformation should be describable by a document listing the individual steps in order:

And ideally, it should be possible to change the way the pipeline works, simply by adding a step between existing steps or by rearranging the individual steps:

If allowing any number of steps, it should be possible to isolate concerns, that is, to have each and every step address one single thing, be it to convert lists, add a caption to images or resolve external references.

The idea isn't new, of course (see section “Some Previous Work (That Inspired Me)”), and I don't in any way claim to have originated any of it; I'm simply a happy user, a convert, and hope never to have to go back to a more traditional, monolithic way of developing XSLTs again.

Mechanics

My preferred approach to pipelined XSLT transformations is largely based on Nic Gibson's XProc Tools (see XProc Tools), where an XML-based manifest file lists the XSLT stylesheets to be run, in document order. A simple manifest looks like this:

<manifest xmlns="http://www.corbas.co.uk/ns/transforms/data" xml:base=".">
    
    <group description="XLSX normalisation and cleanup steps" xml:base="../xslt/">
        <item href="step1.xsl" description="To element one">
            <meta name="param1" value="value1"/>
        </item>
        <item href="step2.xsl" description="To element two"/>
        <item href="step3.xsl" description="To element three">
            <meta name="param3A" value="value3A"/>
            <meta name="param3B" value="value3B"/>
        </item>
        <item href="step4.xsl" description="To element four"/>
    </group>
    
</manifest>

Each item element identifies an XSLT stylesheet to be run (@href), in the order they're listed, and everything has a @description attribute that tells us what the step does. Also note that Stylesheets #1 and #3 have input parameters described by meta elements. Changing the order in which the steps run is easy enough: simply move the items about in your favourite XML editor.

The manifest adheres to a Relax NG compact schema, available on Github as part of the XProc Tools repository (see XProc Tools).

Developing

Developing XSLT stylesheets for pipelines is straight-forward and quite a bit easier than those monolithic stylesheets we've all been used to. Given the manifest file above, step1.xsl, for example, looks like this:

<xsl:stylesheet
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    exclude-result-prefixes="xs"
    version="2.0">
    
    <xsl:output method="xml" indent="yes"/>
    
    <xsl:param name="param1"/>
    
    <xsl:template match="/">
        <xsl:apply-templates select="node()" mode="STEP-1"/>
    </xsl:template>
    
    <xsl:template match="*" mode="STEP-1" priority="1">
        <one one="{$param1}">
            <xsl:copy-of select="@*"/>
            <xsl:apply-templates select="node()" mode="STEP-1"/>
        </one>
    </xsl:template>
    
    <xsl:template match="node()" mode="STEP-1">
        <xsl:copy>
            <xsl:copy-of select="@*"/>
            <xsl:apply-templates select="node()" mode="STEP-1"/>
        </xsl:copy>
    </xsl:template>
    
</xsl:stylesheet>

There is a higher-priority template that converts every input element to one elements in the output and adds @one="{$param1}" to each, but the rest is just an identity transform, copying its input verbatim. And if you'd wanted to only convert, say, para elements, we'd obviously not need @priority at all. This is the case with step2.xsl:

<xsl:stylesheet
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    exclude-result-prefixes="xs"
    version="2.0">
    
    <xsl:output method="xml" indent="yes"/>
    
    <xsl:template match="/">
        <xsl:apply-templates select="node()" mode="STEP-2"/>
    </xsl:template>
    
    
    <xsl:template match="one" mode="STEP-2">
        <two>
            <xsl:copy-of select="@*"/>
            <xsl:apply-templates select="node()" mode="STEP-2"/>
        </two>
    </xsl:template>
    
    
    <xsl:template match="node()" mode="STEP-2">
        <xsl:copy>
            <xsl:copy-of select="@*"/>
            <xsl:apply-templates select="node()" mode="STEP-2"/>
        </xsl:copy>
    </xsl:template>
    
</xsl:stylesheet>

Steps #3, and #4 are both very similar to #2, as this is merely an example pipeline demonstrating the basic ideas. It does illustrate the approach, though, most importantly that individual steps should always be as simple as possible. This helps us achieve that all-important separation of concerns; if you're tempted to add an unrelated template to an existing step rather than creating a new, you're adding needless complexity to that step.

In other words, we create simple stylesheets that focus on changing one thing, be it a single element or a group of similar elements. For example, I tend to have a single step for all inline processing if I only have a few inline elements to transform. More complex semantics might require multiple steps. For example, when upconverting a flat structure such as a Microsoft Word docx file, you'll likely want to use several steps to convert docx lists to a structured XML list—the first would be to identify the list items themselves, the second to determine list labels, the third to wrap the items in a list element, and so on^[2]. If not, the steps will become needlessly complex and hard to follow, which is what we're trying to avoid here.

Having said that, how you write XSLT pipelines is up to you; however, you might want to consider future maintenance, the level of documentation, the skills of the developers, etc, when writing. Some XSLTs will be large regardless, simply because it's what is required.

The example pipeline above is available on Github (see XSLT Pipeline Example Repository) for the inquisitive reader.

ari@toddao:~/Documents/projects/findcourses/poc/tmp/debug/Activate_Learning.xml$ ls -lh
total 106M
-rw-r--r-- 1 ari ari 3,5M mar  5 16:16 0-Activate_Learning.xml
-rw-r--r-- 1 ari ari 3,5M mar  5 16:16 1-XLSX-UTIL_remove-empty.xsl.xml
-rw-r--r-- 1 ari ari 6,4M mar  5 16:16 2-XLSX-UTIL_normalisation.xsl.xml
-rw-r--r-- 1 ari ari 6,4M mar  5 16:16 3-XLSX-UTIL_hyperlinks.xsl.xml
-rw-r--r-- 1 ari ari 4,8M mar  5 16:16 4-XLSX-UTIL_cleanup.xsl.xml
-rw-r--r-- 1 ari ari 4,9M mar  5 16:16 5-XLSX2XML_structure.xsl.xml
-rw-r--r-- 1 ari ari 6,2M mar  5 16:16 6-XLSX2XML_courses.xsl.xml
-rw-r--r-- 1 ari ari 5,6M mar  5 16:16 7-XLSX2XML_dates.xsl.xml
-rw-r--r-- 1 ari ari 5,6M mar  5 16:16 8-XLSX2XML_locations.xsl.xml
-rw-r--r-- 1 ari ari 6,0M mar  5 16:16 9-XLSX2XML_fields.xsl.xml
-rw-r--r-- 1 ari ari 5,7M mar  5 16:16 10-EXC2XI_course.xsl.xml
-rw-r--r-- 1 ari ari 5,7M mar  5 16:16 11-EXC2XI_content-fields.xsl.xml
-rw-r--r-- 1 ari ari 5,7M mar  5 16:16 12-EXC2XI_course-links.xsl.xml
-rw-r--r-- 1 ari ari 5,5M mar  5 16:16 13-EXC2XI_categories.xsl.xml
-rw-r--r-- 1 ari ari 5,5M mar  5 16:16 14-EXC2XI_exc-locations.xsl.xml
-rw-r--r-- 1 ari ari 5,6M mar  5 16:16 15-EXC2XI_exc-events.xsl.xml
-rw-r--r-- 1 ari ari 5,3M mar  5 16:16 16-EXC2XI_exc-duration.xsl.xml
-rw-r--r-- 1 ari ari 5,3M mar  5 16:16 17-EXC2XI_exc-email.xsl.xml
-rw-r--r-- 1 ari ari 4,4M mar  5 16:16 18-EXC2XI_xi-dedupe.xsl.xml
-rw-r--r-- 1 ari ari 4,4M mar  5 16:16 19-EXC2XI_xi-cleanup.xsl.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-model href="xproc-tools/schema/manifest.rng" type="application/xml" schematypens="http://relaxng.org/ns/structure/1.0"?>
<manifest 
	xmlns="http://www.corbas.co.uk/ns/transforms/data"
	xml:id="migration.rtf.stair"
	description="migration.rtf.stair"
	xml:base="."
	version="1.0">
	
	<group
		xml:id="word.conversion"
		description="word.conversion"
		xml:base="../xslt/common/"
		enabled="true">
		<item href="word-to-xhtml5-elements.xsl" description="Flat Word to XHTML conversion"/>
		<item href="add-eqn-pi.xsl" description="inserts a PI placeholder for equations"/>
		...
	</group>
	
	<group
		xml:id="word.conversion.cleanup"
		description="word.conversion"
		xml:base="../xslt/common/"
		enabled="true">
		...
	</group>
	
	<!-- Up till now we've done common word conversion components -->
	
	<!-- Preprocess for Kepler (HTML to HTML) -->
	<group
		xml:id="preprocess.word.conversion.stair"
		description="preprocess.word.conversion.stair"
		xml:base="../xslt/stair-halsbury/"
		enabled="true">
		...
	</group>
	
	
	<!-- Basic construction of Kepler block structures and inline markup -->
	<group
		xml:id="structure.inline.word.conversion.stair"
		description="structure.inline.word.conversion.stair"
		xml:base="../xslt/stair-halsbury/"
		enabled="true">		
		...
	</group>
	
	<!-- Xref and cite processing -->
	<group
		xml:id="xrefs.cites.word.conversion.stair"
		description="xrefs.cites.word.conversion.stair"
		xml:base="../xslt/stair-halsbury/"
		enabled="true">
		...
	</group>
	
	<!-- Other Kepler fixes and cleanup -->
	<group
		xml:id="word.conversion.stair"
		description="word.conversion.stair"
		xml:base="../xslt/stair-halsbury/"
		enabled="true">

		...
		<item href="html2kepler_cleanup.xsl"/>
		<item href="html2kepler_namespacecleanup.xsl"/>
	</group>
	
</manifest>

<xsl:copy-of select="@* except @style"/>
<xsl:attribute name="formatting" select="@style"/>

<xsl:template match="para" mode="STEP-2">
    <p role="{name(.)}">
        ...
    </p>
</xsl:template>

<xsl:attribute name="src-styling">
    <xsl:for-each select="(@style,@frame)">
        <xsl:value-of select="concat('@',name(.),'=',.)"/>
        <xsl:value-of select="if (position()!=last()) then (' ') else ()"/>
    </xsl:for-each>
</xsl:attribute>

<para style="1" frame="dash">Test</para>

<p src-styling="@style=1 @frame=dash">...</p>

Testing

What about testing, you say? Well, writing tests is still about writing XSpec tests for each step which is fine, but if you need to write tests that check actual XML input and output, it won't be possible by merely checking the pipeline inputs and outputs in a single XSpec. XSpec acts on a single XSLT applied on a single XML file, not a sequence of them. We must therefore instantiate the XSpec tests for any step that requires using the actual step inputs and outputs.

I've written a set of XSLT and XProc tools that use the debug output and generic step XSpec tests to generate instance XSpecs. The XProc orchestrates this by using an XSpec manifest file, much like the XSLT pipeline manifest, running each XSpec in sequence. Here's an example XSpec manifest from a live customer project:

<tests
    xmlns="http://www.sgmlguru.org/ns/xproc/steps"
    manifest="../../pipelines/poc/poc-xlsx2xml-manifest.xml"
    xml:base="/home/ari/Documents/repos/xlsx2xml/">
    
    <group description="XLSX normalisation steps">
        <test xslt="xslt/common/XLSX-UTIL_normalisation.xsl" xspec="xspec/common/XLSX-UTIL_normalisation.xspec" description="Normalise shared strings."/>
        <test xslt="xslt/common/XLSX-UTIL_hyperlinks.xsl" xspec="xspec/common/XLSX-UTIL_hyperlinks.xspec" description="Normalise referenced hyperlinks."/>
    </group>
    
    <group description="Conversion steps">
        <test xslt="xslt/common/XLSX2XML_structure.xsl" xspec="xspec/common/XLSX2XML_structure.xspec" description="Convert main structures."/>
        <test xslt="xslt/common/XLSX2XML_dates.xsl" xspec="xspec/common/XLSX2XML_dates.xspec" description="Convert ECMA 376 dates to xs:dateTime."/>
        <test xslt="xslt/common/XLSX2XML_locations.xsl" xspec="xspec/common/XLSX2XML_locations.xspec" description="Convert location info to exc, group it in locations wrapper."/>
    </group>
    
</tests>

Here, each test matches a pipeline manifest item, and this manifest actually tests the pipeline I'm illustrating debug output with, above.

Note that there is nothing to stop you from adding more than one XSpec test to a pipeline step. Quite the contrary, in fact, as splitting tests into multiple files will often make for more readable code.

Documenting

An XSLT pipeline, while not self-documenting, should be relatively easy to understand simply by looking at the manifest. Every element in the manifest includes a @description attribute that should contain at least a basic description of what a step does.

From a documentation perspective, the test manifest is possibly even better. It also contains descriptions, of course, but it is easy to write an XSLT that extracts descriptions and XSpec tests to document a step^[4].

In my ever-so-humble opinion, one of the real advantages of this pipelined approach is to be able to write short steps, meaning that each step will be easier to understand^[5].

Implementations

My preferred way to write XSLT pipelines, again, is to use XProc Tools (see XProc Tools) as the basis for my pipelines. XProc Tools is open source (LGPL 3.0 license) and freely available on Github, but they are just that, a set of tools. There is the XProc step that runs XSLT manifests as discussed above, and there's a step for reading directories and subdirectories recursively, plus some other useful steps. They remain an excellent showcase of what XProc can do.

They don't run out of the box, however. They require wrapper scripts that run the pipeline transformation step in context, validate the output, unzip docx archives, and so on. To this end, I've written additional XProc scripts that do all this, from converting entire directories in batch to validating the output with Schematron and applying XSpec tests. These, just as XProc Tools, are open source (LGPL 3.0) and available on Github (see XProc Batch Wrapper Scripts).

XProc pipelines can be run from oXygen XML Editor or from the command line^[6].

<p:xslt>
   <p:with-input port="stylesheet" href="..."/>
</p:xslt>
<p:store use-when="$debug" href="..."/>

...

pipelines:transform-collection($sources,$manifest-uri,$debug,$out)

End Notes

<xsl:template match="something">
    
    <xsl:variable name="step1">
		<xsl:element name="some-element">
			<xsl:for-each select="$something-to-start-with/...">
				...
			</xsl:for-each>
		</xsl:element>
	</xsl:variable>
	
	...
	
	<xsl:variable name="step5">
		<xsl:element name="some-element">
			<xsl:for-each select="$step4/...">
				...
			</xsl:for-each>
		</xsl:element>
	</xsl:variable>
	
	<xsl:result-document href="{concat($filepath, 'aaa_step5.xml')}">
		<xsl:copy-of select="$step5"/>
	</xsl:result-document>
	
	<xsl:variable name="step5aa">
		<xsl:element name="some-element">
			<xsl:for-each select="$step5/...">
				...
			</xsl:for-each>
		</xsl:element>
	</xsl:variable>
	
	<xsl:result-document href="{concat($filepath, 'aaa_step5aa.xml')}">
		<xsl:copy-of select="$step5aa"/>
	</xsl:result-document>
    
    ...
    
</xsl:template>