Balisage Paper: Stretching XPath: Three Testing Tales

Example of (A), Tests Passing Despite Mistake

Consider an XSpec scenario that calls an XSLT or XQuery function that produces a topic document having sections and paragraphs. XSpec stores the document in a variable named x:result. Suppose you expect at least three sections, and you want to verify that the third section does not contain any paragraphs. You can use XSpec syntax like one of these two <x:expect> elements:

                           
<x:expect label="The third section has no paragraphs"
  test="empty($x:result/topic/section[3]/para)"/>

<x:expect label="The third section has no paragraphs"
  test="$x:result/topic/section[3]/para" select="()"/>

The first syntax expresses a true/false condition using the empty function, and the verification passes if the condition is true. The second syntax uses the test attribute to filter the document down to a hypothetical paragraph in the third section of the topic and verifies that the filtering didn’t find anything—that is, the path expression led to the empty sequence that equals the select attribute.

When constructing these <x:expect> elements, presumably you thought $x:result/topic/section[3] would return at least one section. The label certainly describes the third section as if it exists. If it doesn’t exist, the verifications pass for the wrong reason, as follows:

Maybe the test really should have used a path like $x:result/topic/section[2]/para or $x:result/topic/descendant::section[3]/para, or a different value of the input argument to the function being tested. Maybe the XSLT or XQuery code has a bug that causes the third section to be missing. In any case, the lack of feedback makes it harder to discover the mistake.

Example of (B), Mistake Hindering Failure Investigation

Working with the actual result of a test scenario is not the only place where path expressions can have mistakes. Consider an XSpec scenario for testing an XSLT or XQuery function named f:proto-list that produces list markup with one list item for each node in an input parameter. The scenario loads an XML document from an external file named test-document.xml and uses an XPath expression to select nodes as the value of the parameter.

                           
<x:scenario label="Generate list of tables">
  <x:call function="f:proto-list">
    <x:param name="items" href="test-document.xml"
      select="topic/section/table"/>
  </x:call>
  <x:expect label="List with one item per table">
    <itemizedlist>
      <listitem>
        <para>Table 1. Size chart</para>
      </listitem>
      <listitem>
        <para>Table 2. Color choices</para>
      </listitem>
    </itemizedlist>
  </x:expect>
</x:scenario>

Suppose the document in test-document.xml has no <table> elements having the specified path, such as because the tables are all in subsections rather than first-level sections. In this case, the function parameter named items is an empty sequence, and the test probably fails. While troubleshooting, you might spend a lot of time investigating the XSLT or XQuery code in the function before realizing that the problem is either in the test’s <x:param> element or in the test document.

In Search of Prevention

The XSpec vocabulary is fairly small and does not have dedicated features to alert you when a path expression you provide evaluates to an empty sequence. Using the XSpec vocabulary, you have these options for detecting such an empty sequence:

What if you want an unobtrusive way to ensure that an intermediate result isn’t empty, without adding extra elements to the test or extra rows in the test report?

XPath Syntax Options

It turns out that XPath already has concise syntaxes that provide a way to say, Alert me when this path evaluates to an empty sequence. You can slip them into any XPath expression that you use in XSpec. You can even use them with an intermediate result like a portion of a path expression, without having to define a separate XSpec variable.

Outside XSpec, these syntaxes are useful when an XPath processor performs static type checking and you want to promise the processor up-front that a sequence will satisfy a certain data type condition at run time. Making a strict processor relax its vigilance is a use case that comes up in the specification [KXP] and in authoritative books like those by Michael Kay [K2] and Priscilla Walmsley [W].

In the XSpec situation that this tale is about, avoiding a static type error from a strict processor is not what’s going on. Instead, it’s the other way around, where you want a lax processor to be vigilant and tell you when something doesn’t have the cardinality or data type you expect. Some references for these syntaxes, such as [K2], mention this other use case, and you might encounter non-testing situations where these syntaxes help you construct XPath expressions that alert you to severe missing-data problems. I learned about the testing value of these syntaxes from the XSpec lead developer, GitHub user AirQuick.

XPath Syntax in the XSpec Examples

Returning to the earlier code examples, you can insert a call to the exactly-one function to make sure you find out if the third section fails to exist:

                           
<x:expect label="The third section has no paragraphs"
  test="empty(exactly-one($x:result/topic/section[3])/para)"/>

<x:expect label="The third section has no paragraphs"
  test="exactly-one($x:result/topic/section[3])/para" select="()"/>

Also, you can insert a call to the one-or-more function to make sure you find out if the function parameter turns out to be empty:

                           
<x:scenario label="Generate list of tables">
  <x:call function="f:proto-list">
    <x:param name="items" href="test-document.xml"
      select="one-or-more(topic/section/table)"/>
  </x:call>
  <x:expect label="List with one item per table">
    <itemizedlist>
      <listitem>
        <para>Table 1. Size chart</para>
      </listitem>
      <listitem>
        <para>Table 2. Color choices</para>
      </listitem>
    </itemizedlist>
  </x:expect>
</x:scenario>

With these changes, if the exactly-one or one-or-more function produces an error message when the test runs, the error gives you clear and valuable information that addresses the two categories of testing problems in this tale.

Compact, Scalable Test with Neighboring Contexts

We can reduce the test code to one 19-line scenario, by putting all 11 input items in a single <x:context> element, as follows:

                           
<x:scenario label="as-type attribute with 11 values listed">
  <x:context mode="assign-json-type" select="/*/@as-type">
    <value as-type="uuid"/>
    <value as-type="markup-line"/>
    <value as-type="dateTime-with-timezone"/>
    <value as-type="string"/>
    <value as-type="token"/>
    <value as-type="uri"/>
    <value as-type="markup-multiline"/>
    <value as-type="uri-reference"/>
    <value as-type="email"/>
    <value as-type="empty"/>
    <value as-type="base64Binary"/>
  </x:context>
  <x:expect label="Each attribute of the context maps to in-json='string'"
    select="for $i in ($x:context) return /*/@in-json">
    <any-element in-json="string" xmlns=""/>
  </x:expect>
</x:scenario>

In this scenario, the context is a sequence of 11 as-type attribute nodes, because the select="/*/@as-type" attribute is selecting the attribute nodes from the child elements of <x:context>. XSpec loops over the 11 nodes and applies templates to each. XSpec gathers the actual results together into a sequence of 11 in-json attributes, and that sequence is what the <x:expect> element uses for verification. If verification fails, the report would show the full sequences of actual and expected results, and we would need to determine which of the 11 comparisons failed; that extra labor is one reason I do not use multiple-item contexts in that many situations.

In this case, the expected value is identical for all the context nodes; it’s an in-json="string" attribute node. The <x:expect> provides this node in one child element and uses select="for $i in ($x:context) return /*/@in-json" to replicate the attribute node once per context item. The notation $x:context refers to a variable that XSpec populates with the 11-item context. (In XPath 4.0, an alternative could be select="replicate(/*/@in-json,count($x:context))".)

Now the scenario is fairly compact and would scale well if we needed to add a few more input values. This tale isn’t complete, though, because of an aspect of the scenario that might be problematic or seem philosophically questionable: the potential for the different items in the context to interfere with each other as the XSLT code accesses parts of the tree. Having the different items mingling in the XSpec markup makes the test code concise, but having them mingle when the XSLT runs is not desirable.

What makes interference during the XSLT execution a possibility is that the context is a sequence of 11 attribute nodes within a tree that includes all 11 elements as siblings of each other. To see evidence of this tree relationship, add the following message to the XSLT template and watch the console output from running the test.

                           
<xsl:message expand-text="1">Preceding {
  local-name(..)} element has as-type={
  ../preceding-sibling::*[1]/@as-type/string()
  }</xsl:message>

Preceding value element has as-type=
Preceding value element has as-type=uuid
Preceding value element has as-type=markup-line
Preceding value element has as-type=dateTime-with-timezone
Preceding value element has as-type=string
Preceding value element has as-type=token
Preceding value element has as-type=uri
Preceding value element has as-type=markup-multiline
Preceding value element has as-type=uri-reference
Preceding value element has as-type=email
Preceding value element has as-type=empty

Preventing Interference Among Contexts

How might a test make the attributes isolated, if they don’t start that way? XSpec lacks syntax for constructing new attribute nodes. XSpec does support helper functionality, such as an XSLT function that takes an attribute within a tree and creates a new, isolated attribute node.

However, a solution is even easier than that! In the XSLT 3.0 specification, the Streaming section lists functions named copy-of and snapshot that isolate parts of trees. While snapshot preserves the subtree’s ancestors and their attributes, copy-of returns the subtree only. Neither function includes siblings in its output. These functions are useful during stream processing, which imposes restrictions on access to nodes of a tree. Buffering a copy of a subtree in memory, with or without the ancestry of the subtree, enables freer access to it. The specification notes that each of these two functions is available for use (and is primarily intended for use) when a source document is processed using streaming. It can also be used when not streaming [K3].

This XSpec situation does not involve stream processing, and the two functions are useful not because of access restrictions or memory usage but for isolation and test cleanliness. If the <x:context> element in Figure 8 changes select="/*/@as-type" to select="/*/@as-type/copy-of()", the XSLT code sees isolated attributes instead of attributes of elements in a tree. The console messages from running the test no longer show the name of the element or its preceding sibling’s attribute value, because there is no element and hence no sibling element.

Preceding  element has as-type=
Preceding  element has as-type=
Preceding  element has as-type=
Preceding  element has as-type=
Preceding  element has as-type=
Preceding  element has as-type=
Preceding  element has as-type=
Preceding  element has as-type=
Preceding  element has as-type=
Preceding  element has as-type=
Preceding  element has as-type=

As a variation, changing select="/*/@as-type" to select="/*/@as-type/snapshot()" causes the XSLT code to see attributes that are attached to elements, but each element has no siblings. The console messages from running the test show the name of the element but not a preceding sibling’s attribute value, because there is no sibling element.

Preceding value element has as-type=
Preceding value element has as-type=
Preceding value element has as-type=
Preceding value element has as-type=
Preceding value element has as-type=
Preceding value element has as-type=
Preceding value element has as-type=
Preceding value element has as-type=
Preceding value element has as-type=
Preceding value element has as-type=
Preceding value element has as-type=

The point is that if siblings affected the XSLT template’s behavior in a way that could affect the test, using copy-of or snapshot in the XSpec code would take those siblings out of the view of the XSLT template. As a result, a test author would be able to write a compact scenario having a multiple-item context while preventing interference among the different items.

Creating the Variations

Here are two examples of XQuery Update code for creating variants as persistent files, where the file:write and file:base-dir functions are specific to BaseX.

                           
copy $s := doc('original/valid-123.xml')
  modify (
  rename node $s/d:article//d:code
    as 'd:literal'
  )
  return file:write(
    file:base-dir() || "gen/invalid-123-literal.xml", $s
)

                           
copy $s := doc('original/invalid-456.xml')
  modify (
  insert node ($s/d:article/d:abstract)
    before $s/d:article/d:info/d:author,
  delete node $s/d:article/d:abstract
  )
  return file:write(
    file:base-dir() || "gen/valid-456-abstract-moved.xml", $s
)

Both examples use this three-step procedure:

If you don’t mind using even more BaseX-specific functionality, you can streamline the syntax using the BaseX convenience operator, update [XB]. The three steps are the same, but they look a bit different in this syntax. Here is how the expressions above look using the update operator:

                           
file:write(
  file:base-dir() || "gen/invalid-123-literal.xml",
  doc('original/valid-123.xml') update {
    rename node /d:article//d:code
      as 'd:literal'
  }
)

                           
file:write(
  file:base-dir() || "gen/valid-456-abstract-moved.xml",
  doc('original/invalid-456.xml') update {
    insert node (/d:article/d:abstract)
      before /d:article/d:info/d:author,
    delete node /d:article/d:abstract
  }
)

Using the Variations

If you have a module of functions that each use this three-step procedure, you can run all the functions in the module to create all the variations you need. With the variation files in hand, you are ready to run validation tests against the set of original documents and generated documents. The validation tests themselves can use whatever testing functionality you like, such as the Schematron support in XSpec or the BaseX modules for validation and unit testing.

Advantages of separating the variant creation from the validation testing include:

On the other hand, disadvantages include:

If you think the disadvantages outweigh the advantages, you can combine the XQuery Update expressions with the validation tests themselves. Here, we show an example that illustrates the combination approach, using two BaseX modules: validation and unit testing.

First, we declare namespaces and a global variable that stores the path to a RelaxNG schema.

                           
module namespace mytest = "my-module-namespace-uri";
declare namespace d = "http://docbook.org/ns/docbook";
declare variable $mytest:rng-schema := 'balisage-1-5.rng';

Although it is not required, we find it useful to define helper functions that perform repeated tasks. These helper functions validate documents using the validate:rng-info function [V] and make test assertions using the unit:assert function [UM]. The validate:rng-info function returns a string sequence containing warnings and errors, if any. The test assertion hinges on whether this string sequence is empty (that is, the document is valid) or non-empty. In the unit:assert element, the second parameter is output in case of a test failure, so it should be something that helps you investigate the failure.

                           
declare function mytest:expect-valid(
$to-validate as item()
) as empty-sequence() {
  let $val := $to-validate => validate:rng-info($mytest:rng-schema)
  return
    unit:assert(empty($val), $val)
};

declare function mytest:expect-invalid(
$to-validate as item()
) as empty-sequence() {
  let $val := $to-validate => validate:rng-info($mytest:rng-schema)
  return
    unit:assert(exists($val), 'Valid when it should have been invalid')
};

If you start with a valid document and use XQuery Update to make it invalid, your test functions look like the following pair. The second function in the pair uses XQuery Update code identical to Figure 13. Each function uses one of the helper functions defined above.

                           
declare %unit:test function mytest:valid-sample-doc() {
  let $s := doc('original/valid-123.xml')
  (: As is, file is valid :)
  return
    mytest:expect-valid($s)
};

declare %unit:test function mytest:xqupdate-makes-invalid-literal() {
  copy $s := doc('original/valid-123.xml')
    modify (
    (: Renaming <code> as <literal> makes the file invalid :)
    rename node $s/d:article//d:code
      as 'd:literal'
    )
    
    return
      mytest:expect-invalid($s)
};

Going in the other direction, if you start with an invalid document and use XQuery Update to make it valid, your test functions look like the following. The second function in the pair uses XQuery Update code identical to Figure 14.

                           
declare %unit:test function mytest:invalid-sample-doc-abstract() {
  let $s := doc('original/invalid-456.xml')
  (: As is, file is invalid :)
  return
    mytest:expect-invalid($s)
};

declare %unit:test function mytest:xqupdate-makes-valid-abstract() {
  copy $s := doc('original/invalid-456.xml')
    modify (
    (: Moving <abstract> makes the file valid :)
    insert node ($s/d:article/d:abstract)
      before $s/d:article/d:info/d:author,
    delete node $s/d:article/d:abstract
    )
    
    return
      mytest:expect-valid($s)
};

If you run the test module in BaseX, the result looks like this:

                           
<testsuites time="PT0.049S">
  <testsuite name="file:/.../generate-in-memory-and-then-test.xqm"
    time="PT0.049S" tests="4" failures="0" errors="0" skipped="0">
    <testcase name="valid-sample-doc" time="PT0.007S"/>
    <testcase name="xqupdate-makes-invalid-literal" time="PT0.01S"/>
    <testcase name="invalid-sample-doc-abstract" time="PT0.007S"/>
    <testcase name="xqupdate-makes-valid-abstract" time="PT0.01S"/>
  </testsuite>
</testsuites>

If you cause deliberate test failures by interchanging mytest:expect-invalid with mytest:expect-valid, the failures produce <failure> elements like the following. The line and column numbers point to the helper functions (not extremely useful when all the functions use the same helper functions), while the child <info> elements provide some useful information.

                           
<testsuites time="PT0.048S">
  <testsuite name="file:/.../generate-in-memory-and-then-test.xqm"
    time="PT0.048S" tests="4" failures="2" errors="0" skipped="0">
    <testcase name="valid-sample-doc" time="PT0.007S"/>
    <testcase name="xqupdate-makes-invalid-literal" time="PT0.01S">
      <failure line="13" column="16">
        <info>file:/.../original/valid-123.xml, 20:98:
          element "d:literal" not allowed anywhere; expected the
          element end-tag, text or element "blockquote", "citation",
          "code", "email", "emphasis", "equation", "figure", "footnote",
          "informaltable", "inlinemediaobject", "itemizedlist", "link",
          "mediaobject", "note", "orderedlist", "phrase", "programlisting",
          "quote", "subscript", "superscript", "table", "trademark",
          "variablelist" or "xref"
        </info>
      </failure>
    </testcase>
    <testcase name="invalid-sample-doc-abstract" time="PT0.007S"/>
    <testcase name="xqupdate-makes-valid-abstract" time="PT0.009S">
      <failure line="19" column="16">
        <info>Valid when it should have been invalid</info>
      </failure>
    </testcase>
  </testsuite>
</testsuites>