Balisage Paper: Schematron as XSLT Interface, and Advanced Validation Models

Model A: Only XSLT Variables and Functions

Sometimes we just need some basic XSLT functionality. A project is just starting out, and the needs are minimal. But we want to take advantage of the benefits that come with doing our primary processing in XSLT. In this model, we rely upon the XSLT file for our global variables and functions.

Here is a simple example of a paired Schematron file with supporting XSLT file:

Model A Schematron file widgets.sch

<sch:schema xmlns:sch="http://purl.oclc.org/dsdl/schematron"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform" queryBinding="xslt3">
    <xsl:import href="widgets.xsl"/>
    <sch:ns prefix="f" uri="my-functions"/>
    <sch:pattern>
        <sch:rule context="widget">
            <sch:report test="@id = $duplicate-widget-ids">Duplicate id.</sch:report>
        </sch:rule>
        <sch:rule context="widget/name">
            <sch:report test=". = f:duplicate-strings(../name)">Names should not be
                duplicated.</sch:report>
        </sch:rule>
    </sch:pattern>
</sch:schema>

Model A XSLT file widgets.xsl

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:f="my-functions"
    exclude-result-prefixes="#all" version="3.0">

    <xsl:variable name="duplicate-widget-ids" as="xs:string*"
        select="f:duplicate-strings(//widget/@id)"/>

    <xsl:function name="f:duplicate-strings" as="xs:string*">
        <xsl:param name="strings" as="xs:string*"/>
        <xsl:for-each-group select="$strings" group-by=".">
            <xsl:if test="count(current-group()) gt 1">
                <xsl:sequence select="current-grouping-key()"/>
            </xsl:if>
        </xsl:for-each-group>
    </xsl:function>
</xsl:stylesheet>

In the example above, the XSLT file defines the function f:duplicate-strings, which also drives the binding of the global variable $duplicate-widget-ids. Both the function and the variable are invoked by the Schematron file, which checks every <widget> for duplicate ids, and uses f:duplicate-strings to make sure that no <widget> has a duplicate <name> .

Obviously, both the global variable and the function could have been expressed in the Schematron file. But one can easily imagine functions and variables that need much more complex operations. When such expansion is required, we simply expand our XSLT suite, without complicating our Schematron file.

Model A depends upon a strong asymmetric contract between the XSLT and Schematron files. In writing the Schematron file, one needs to know what the XSLT is creating, and but those who write the XSLT file might not need to know about the Schematron file. This allows the XSLT component to develop into a full-fledged library that may have purposes much larger than the catalyzing Schematron file. In fact, that XSLT library may be driving multiple Schematron processes.

Model A is already in use. It is the approach taken by Matthieu Ricaud-Dussarget in his XSLT Quality project, https://github.com/mricaud/xslt-quality.

Model B: Validandum Transformed

There are more complex situations, where it is wholly more efficient to do some full transformations of the validandum, then evaluate as needed. Here is an example of how that might be handled in a Schematron file and its companion XSLT file. Here we treat the widget XML file as a running series of orders, where each <widget> represents change in inventory, and we wish to flag places where the total inventory level dips below a certain point. We set up the desired reports in the Schematron file:

Model B Schematron file widgets.sch

<sch:schema xmlns:sch="http://purl.oclc.org/dsdl/schematron"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform" queryBinding="xslt3">
    <xsl:import href="widgets.xsl"/>
    <sch:ns prefix="f" uri="my-functions"/>
    <sch:pattern>
        <sch:rule context="widget">
            <sch:let name="self-evaluated" value="key('get-by-path', path(.), $input-evaluated)"/>
            <sch:let name="count" value="xs:integer($self-evaluated/running-widget-count)"/>
            
            <sch:report test="$count lt 10 and $count gt 0" role="warn">We are low on widgets! 
                <sch:value-of select="$self-evaluated/running-widget-count"/></sch:report>
            <sch:report test="$count le 0">We are out of widgets! <sch:value-of
                    select="$self-evaluated/running-widget-count"/></sch:report>
        </sch:rule>
    </sch:pattern>
</sch:schema>

Now, in the companion XSLT file, we use an accumulator to keep track of the running inventory.

Model B XSLT file widgets.xsl

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:f="my-functions"
    expand-text="yes"
    exclude-result-prefixes="#all" version="3.0">
    
    <xsl:key name="get-by-path" match="*" use="@_path"/>
    
    <xsl:variable name="input-prepped" as="document-node()">
        <xsl:apply-templates select="/" mode="prep"/>
    </xsl:variable>
    <xsl:variable name="input-evaluated" as="document-node()">
        <xsl:apply-templates select="$input-prepped" mode="eval"/>
    </xsl:variable>
    
    <xsl:mode name="prep" on-no-match="shallow-copy"/>
    <xsl:template match="*" priority="1" mode="prep">
        <xsl:copy>
            <xsl:copy-of select="@*"/>
            <xsl:attribute name="_path" select="path(.)"/>
            <xsl:apply-templates mode="#current"/>
        </xsl:copy>
    </xsl:template>
    
    <xsl:accumulator name="running-widget-count" as="xs:integer" initial-value="0">
        <xsl:accumulator-rule match="widget/count" select="$value + xs:integer(.)"/>
    </xsl:accumulator>
    <xsl:mode name="eval" on-no-match="shallow-copy" use-accumulators="#all"/>
    
    <xsl:template match="widget" mode="eval">
        <xsl:copy>
            <xsl:copy-of select="@*"/>
            <running-widget-count>{accumulator-after('running-widget-count')}</running-widget-count>
            <xsl:apply-templates mode="#current"/>
        </xsl:copy>
    </xsl:template>
    
</xsl:stylesheet>

Despite the trim Schematron file, the underlying process is busy. The bulk of the evaluation takes place in the XSLT file, through a two-pass transformation. A copy of the validandum is made, with a temporary @_path attribute added to each element, to create a dereferencable unique id, and the results are bound to $input-prepped. In the second pass, bound to global variable $input-evaluated, an accumulator is consulted, and a running inventory is added to each <widget>.

Back in the Schematron file, every <widget> fires a rule. The local variable $self-evaluated is bound to its counterpart from the global variable $input-evaluated, i.e., the context node transformed by the main XSLT file, which has been looking for changes in inventory. The rule then evaluates the local variable, in this case checking for a running count of widgets, warning the user when it drops below 10 and raising an error when negative.

Model B requires a strong contract between the XSLT and Schematron files. In writing the Schematron file, one needs to know what the XSLT is creating, and in writing the XSLT file, one needs to know what patterns and analysis are of interest.

A benefit to this model is that it is quite flexible and extensible: you can create as many global variables as you like, with the validandum transformed in different ways. Such processing is straightforward in XSLT, and the results are immediately accessible to any <sch:report> or <sch:assert>.

I am uncertain if any public projects use Model B, but I have used it for a number of projects for the Federal Government. One such project is the validation of XML files for the Federal Register. The Federal Register has complex requirements about where footnote references and the footnotes themselves must be placed. Determining whether footnote references and footnotes have been placed accurately, and making sure that the footnote numerals are sequential and pairwise correct requires complex analysis of the entire document. It is definitely not a job for Schematron. But XSLT can do this analysis efficiently, and provide the results to the Schematron file to report anything of concern or interest.

Model C: Context-Agnostic Schematron

It does not take long for a Model B XSLT-Schematron project to grow to the point where we want the XSLT file to simply do all the evaluation, and use Schematron simply to report the results. A common motivation for taking this extra step is the discovery that the logic being put into a traditional Schematron file needs to be leveraged and repurposed outside the Schematron file. For example, the Schematron reports and asserts may be giving error codes, but we get to the point where we need to maintain an inventory of error codes separately, to integrate with documentation or a regression test suite. Another reason might be the need to take advantage of XSLT’s template-based approach to shadowing, which is more sophisticated than Schematron’s. And yet another reason might be that our Schematron Quick Fix library is getting overly large, and needs to be simplified and refactored into fix typologies.

With these high-level needs, we abandon the thought of making a <sch:rule> for every context. We start by making a very simple and small Schematron file with one rule, and one context, firing for every element. The XSLT’s job is to reduce the validandum to an arbitrarily constructed error and warning report, integrated with possible fixes. The Schematron file simply checks an XSLT-created global variable to see if that particular element has any warnings, errors, or quick fixes.

Here is an example of Model C, beginning with the Schematron file:

Model C Schematron file widgets.sch

<sch:schema xmlns:sch="http://purl.oclc.org/dsdl/schematron"
    xmlns:sqf="http://www.schematron-quickfix.com/validator/process"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform" queryBinding="xslt3">
    <xsl:import href="widgets.xsl"/>
    <sch:ns prefix="f" uri="my-functions"/>
    <sch:pattern>
        <sch:rule context="*">
            <sch:let name="errors" value="key('get-by-path', path(.), $all-errors)"/>
            <sch:report test="exists($errors[1])" sqf:fix="replace-element">
                <sch:value-of select="$errors/message"/></sch:report>
            
            <sqf:fix id="replace-element" use-for-each="$errors/fix[@type = 'replace-element']">
                <sqf:description>
                    <sqf:title><sch:value-of select="$sqf:current/@message"/></sqf:title>
                </sqf:description>
                <sqf:replace select="$sqf:current/*"/>
            </sqf:fix>
        </sch:rule>
    </sch:pattern>
</sch:schema>

Before presenting the XSLT file, which is longer than previous examples, it is worth noting that the Schematron file is now context-agnostic. The Schematron file is not looking or specifically named elements. It simply fires at every element, and checks the XSLT for a list of relevant errors, and reports them. If the error includes any fixes, they are passed to the appropriate SQF construct. For the sake of simplicity, I have used only one SQF fix type here, but the types of SQFs can and do grow. The example also anticipates only the first error for a given element. If there are multiple errors, there are creative ways these can be administered in the Schematron file, depending upon what is desired.

The XSLT that drives the whole process is as follows:

Model C XSLT file widgets.xsl

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:f="my-functions"
    expand-text="yes"
    exclude-result-prefixes="#all" version="3.0">
    
    <xsl:key name="get-by-path" match="*" use="@_path"/>
    
    <xsl:variable name="input-prepped" as="document-node()">
        <xsl:apply-templates select="/" mode="prep"/>
    </xsl:variable>
    <xsl:variable name="all-errors" as="document-node()">
        <xsl:document>
            <errors>
                <xsl:apply-templates select="$input-prepped" mode="errors"/>
            </errors>
        </xsl:document>
    </xsl:variable>
    
    <xsl:mode name="prep" on-no-match="shallow-copy"/>
    <xsl:template match="*" priority="1" mode="prep">
        <xsl:copy>
            <xsl:copy-of select="@*"/>
            <xsl:attribute name="_path" select="path(.)"/>
            <xsl:apply-templates mode="#current"/>
        </xsl:copy>
    </xsl:template>
    
    <xsl:accumulator name="running-widget-count" as="xs:integer" initial-value="0">
        <xsl:accumulator-rule match="widget/count" select="$value + xs:integer(.)"/>
    </xsl:accumulator>
    <xsl:mode name="errors" on-no-match="shallow-skip" use-accumulators="#all"/>
    
    <xsl:template match="widget[accumulator-after('running-widget-count') lt 0]/count" mode="errors">
        <error n="1">
            <xsl:copy-of select="@_path"/>
            <message>We are out of widgets! {accumulator-after('running-widget-count')}</message>
            <fix type="replace-element" message="Increase count so running count is zero.">
                <xsl:copy>
                    <xsl:copy-of select="@* except @_path"/>
                    <xsl:value-of select="xs:integer(.) - accumulator-after('running-widget-count')"/>
                </xsl:copy>
            </fix>
            <fix type="replace-element" message="Add 1,000 to count.">
                <xsl:copy>
                    <xsl:copy-of select="@* except @_path"/>
                    <xsl:value-of select="xs:integer(.) + 1000"/>
                </xsl:copy>
            </fix>
        </error>
        
        <xsl:next-match/>
    </xsl:template>
    
</xsl:stylesheet>

As with Model B, we take the preliminary step of adding the temporary attribute @_path to each element. The next pass, however, does its best to reduce the original input to a simple set of elements. The errors template mode is set to shallow-skip (not shallow-copy, as in Model B), to reduce the validandum to a (hopefully small) list of errors. For a long validandum with few errors, the query time in each <sch:rule> is minimal. The bulk of work (and therefore performance concerns) are restricted to the XSLT tier.

In Model C, we customize our own error report, along with potential quick fixes, using whatever structure makes sense to us. In the example above, there is an <error> element that takes an error code, a message, and zero or more <fix>es. It has been my experience that some knotty SQFs benefit from sophisticated XSLT adjustments. Because the SQF process has been moved into the XSLT tier, we can provide some creative fixes.

Model C depends upon a symmetric but loose contract between the XSLT and Schematron files. In writing the Schematron file, one does not need to much about what the XSLT is creating, aside from the structure of the errors and fixes. Those who write the XSLT file might need to know how those errors will be reported and those fixes will be applied, but that is all, because the Schematron file is generic. The XSLT tier essentially subsumes the traditional responsibility of Schematron.

The error or warning report created by the XSLT is of an arbitrary structure. How that report is designed is a private affair, exclusively between the XSLT file and the Schematron handler.

This particular approach is eminently expansible. A project can set up error codes and messages in a stand-off file, and create a suite of private functions to more concisely build errors, warnings, and fixes.

I have adopted Model C widely, the best public example being the validation system behind the Text Alignment Network, https://textalign.net/. In a project I am developing for the Government Publishing Office, pertaining to marking up legislation that comes from the U.S. Congress, I have used Model C to do some very nuanced validation, such as checking to see whether conjunctions (and / or) have been properly placed in the complex hierarchy. Each error or warning is assigned a code. If a user wishes to ignore a particular error, they can squelch specific warnings or errors through the Oxygen interface, which writes the preferences to a stand-off configuration file for the user. The Schematron file does not need to worry about these matters. The XSLT tier checks external resources and makes the decisions about whether a warning or error should be fired.

Model D: Push-Me Pull-You

Models A, B, and C cover common situations. Model D handles a rarer, but significant, use case. It should appeal to those who appreciate XSLT recursion.

Let us suppose we have adopted model A or B, and it is dependent upon an underlying XSLT library. The two become tightly coupled. The XSLT library matures and begins to serve a variety of needs, not just the Schematron process. The library grows to the point where it needs to validate a file and incorporate the report into other processes. That is, the XSLT library needs to get validation information on an XML file and do something with it.

We have this feature already, in the case of other schemas. For example, one can use Charles Foster’s Saxon Jing project (https://www.cfoster.net/saxon-jing/) within XSLT code to validate any XML file against a given RELAX-NG schema. Or in the case of XML Schema one can use saxon:evaluate() (https://www.saxonica.com/documentation12/index.html#!functions/saxon/evaluate) or <xsl:evaluate> to validate any XML file. Once bound to an XSLT variable, such error reports can be used very creatively. And of course Skeleton or SchXslt could be leveraged to convert a Schematron into SVRL-making XSLT.

But what if the XSLT library that needs the SVRL report is the very same library upon which the Schematron file depends? We run up against the constraint of mutual dependence. Two XSLT files may not include each other (although that may change in XSLT 4.0). We want to be able to use the Schematron in its own right. But we also want to compile it to an SVRL-making XSLT file that can be grafted within the main XSLT project.

This model, although the most sophisticated of the four we have so far examined, has the most in common with Model A. That is, the Schematron file expresses all our rules, and draws from the XSLT tier only for variables and functions. The Schematron file might be quite lengthy, with many patterns and rules.

There are solutions to the problem of mutual inclusion, and the one I have opted for here requires introducing to the Schematron file a static XSLT parameter. <xsl:param> is supported by the Skeleton implementation, but not by SchXslt (at this writing). But SchXslt’s MIT license allows us to make the necessary modifications.

We also make sure that we use <xsl:import> (not include), because we are going to declare the same static parameter in the XSLT library, which will import the XSLT that is built by Skeleton or SchXslt.

The abstract idea is made clear with some example code, with key sections set in boldface.

Model D Schematron file widgets.sch

<sch:schema xmlns:sch="http://purl.oclc.org/dsdl/schematron"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:f="my-functions" queryBinding="xslt3">
    
    <xsl:param name="schematron-is-main-point-of-entry" as="xs:boolean" select="true()" static="yes"/>
    <xsl:import href="widgets.xsl" use-when="$schematron-is-main-point-of-entry"/>
    
    <sch:ns prefix="f" uri="my-functions"/>

    <sch:pattern>
        <sch:rule context="widget">
            <sch:report test="@id = $duplicate-widget-ids">Duplicate id.</sch:report>
        </sch:rule>
        <sch:rule context="widget/name">
            <sch:report test=". = f:duplicate-strings(../name)">Names should not be
                duplicated.</sch:report>
        </sch:rule>
    </sch:pattern>
</sch:schema>

In the above code sample, note the use of the static parameter and the import statement, which allows our parameter to shadow the one in the XSLT file. Note also @use-when, which will determine whether this XSLT element will even be used.

Now the XSLT library:

Model D XSLT file widgets.xsl

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:f="my-functions"
    exclude-result-prefixes="#all" version="3.0" default-mode="widgets">
    
    <xsl:param name="schematron-is-main-point-of-entry" as="xs:boolean" select="false()" static="yes"/>
    
    <xsl:import href="widgets-svrl-maker_thank-you-skeleton.xsl" 
        use-when="not($schematron-is-main-point-of-entry)"/>
    <xsl:import href="widgets-svrl-maker_thank-you-schxslt.xsl" 
        use-when="not($schematron-is-main-point-of-entry)"/>
    
    <xsl:variable name="duplicate-widget-ids" as="xs:string*"
        select="f:duplicate-strings(//widget/@id)"/>
    
    <xsl:function name="f:duplicate-strings" as="xs:string*">
        <xsl:param name="strings" as="xs:string*"/>
        <xsl:for-each-group select="$strings" group-by=".">
            <xsl:if test="count(current-group()) gt 1">
                <xsl:sequence select="current-grouping-key()"/>
            </xsl:if>
        </xsl:for-each-group>
    </xsl:function>
    
    <xsl:variable name="skeleton-svrl-report" as="item()*">
        <xsl:apply-templates select="/" mode="#unnamed"/>
    </xsl:variable>
    
    <xsl:variable name="schxslt-svrl-report" as="item()*">
        <xsl:apply-templates select="/" mode="d8e9"/>
    </xsl:variable>
    
    <xsl:template match="/" mode="widgets">
        <report>
            <skeleton>
                <xsl:copy-of select="$skeleton-svrl-report"/>
            </skeleton>
            <schxslt>
                <xsl:copy-of select="$schxslt-svrl-report"/>
            </schxslt>
        </report>
    </xsl:template>
    
</xsl:stylesheet>

The <xsl:import> statements presume access to the SVRL-creating XSLT, courtesy Skeleton or SchXslt. The names of the files referenced by @href attempt to make clear this subtle point. That is, we need some system (manual or automated) that can convert the Schematron, on a whim’s notice, to the appropriate XSLT package.

Just like the Schematron file, the XSLT library invokes the same static parameter (albeit with the opposite value), and it points with <xsl:import> declarations to two possible XSLT files that create SVRL reports, one generated by Skeleton and the other generated by SchXslt.

The @use-when attribute is key. The <xsl:import> declarations are retained if the file is the main point of entry, and dropped otherwise. The static parameters ensure that each set of XSLT files can invoke the other – the push-me pull-you aspect of the model – without triggering a fatal XSLT error.

The widgets.xsl file presented above does not do very much of interest. It simply returns the two different SVRL reports, but the global variables $skeleton-svrl-report and $schxslt-svrl-report could be used in creative ways in a more developed form of the project.

Model D depends upon a strong symmetric contract between the XSLT and Schematron files. In writing the Schematron file, one needs to know what the XSLT is creating, and those who write the XSLT need to know about the Schematron file, or, rather, they need to know about its SVRL-making XSLT representation. Skeleton takes over the unnamed template mode as its primary mode, but SchXslt uses a hexadecimal name for its primary mode. The XSLT tier needs to accommodate the decisions made by either implementation about how to build the template modes.

I am unaware of any public project that uses Model D. In private development for the Federal Government, I have used the push-me pull-you method in the GPO XML Toolkit, where validation of the Federal Register needs to be performed and presented in different contexts. It needs to be fired directly in an XML editor on the command line, but it is also necessary to synthesize the validation report into the validandum to create a visually appealing HTML file. I do not expect Model D to be used commonly, because it is tailored for projects that have tightly coupled their Schematron files with an advanced XSLT function library, and this does not happen often.