Balisage Paper: Saxon-JS Meets XSpec Unit Testing

Role of Unit Testing in a Web Application

Suppose you are creating a web application that implements a significant feature set in its initial release, and suppose your design team anticipates growing and improving the features in successive releases. You want to build quality in from the start. Looking ahead, you want to be able to refactor code fearlessly as the design evolves. Unit testing can help you accomplish those objectives. This paper discusses ways to resolve incompatibilities between a particular testing tool and a particular web application development tool.

What Is Saxon-JS?

Saxon-JS is a product from Saxonica that runs XSLT in a browser^[1] using JavaScript. In a typical architecture of a web application for use with Saxon-JS, your source code comes in two parts: an XSLT transform and an HTML file. You use another Saxonica product called Saxon-EE to compile your XSLT code to a separate file that is known as a stylesheet export file. In your HTML file, you include JavaScript code that directs Saxon-JS to that stylesheet export file. When the browser renders the HTML page, it calls Saxon-JS, which implements your XSLT transform by running JavaScript code that is part of the Saxon-JS product. For a diagram of this architecture, see [S1].

To help you create interactive web applications, Saxon-JS includes a suite of what it calls interactive XSLT extensions. These extensions help you connect your XSLT code with the larger context of the web application. For example, you can use interactive XSLT extensions in your XSLT code to respond to the user's mouse clicks, obtain the URL query string, schedule asynchronous calls to named XSLT templates, execute custom JavaScript code, or change the values of HTML attributes currently being rendered in the browser.

What Is XSpec?

XSpec is an open-source software product for testing XSLT, XQuery, and Schematron code. An XSpec test suite is an XML document that conforms to the XSpec schema. When writing an XSpec test suite, you set up test scenarios (or "tests") and express the results you expect. When you run an XSpec test suite, the software evaluates the test scenarios, gathers results, and generates a report that compares the actual results to the expected ones. You can view the report as HTML in a browser, or you can programmatically operate on the same information in textual or XML format. An XSpec test suite can include multiple test scenarios, each of which can make multiple assertions about expected results.

Here is a bit more detail about what happens when you run an XSpec test suite to test XSLT code. First, the XSpec infrastructure uses XSLT to compile your XSpec test suite into a new XSLT transform. This compiled test suite incorporates all the logic from your test suite and also uses <xsl:import> to import the XSLT you are testing. Next, the compiled test suite runs and produces an XML report that is further transformed into an HTML report. Execution of the compiled test suite uses Saxon on the Java platform, not the JavaScript platform running in a Web browser.

Figure 1: XSpec Compilation

The XSpec package also includes functionality for computing and reporting on code coverage.

Case Study for This Paper

While we expect the techniques and commentary in this paper to have broader applicability, we developed them while working on a specific Saxon-JS (version 1.2) web application at a software company. The application displays lists of software language items (functions, methods, and so on) in a suite of approximately 100 products. Each item in the list shows the hyperlinked name of a function or other language item next to a one-line summary. The application enables the end user to:

Any choices that the user can make interactively are also reflected in the URL query string, making the choices easy to access programmatically during testing.

In this application, selecting the correct data to display based on those choices is critical. The graphical controls are important, too, but animation is neither flashy nor the primary focus of the application. These aspects of the application's nature influenced the unit-testing strategy, as later sections describe.

The next screen capture shows a sample in which the application renders a categorized view of a subset of functions within one product [M].

Figure 2: Sample View of Case Study Application

Types of Incompatibilities

During development of the web application, we soon discovered that certain key traits of a Saxon-JS web application cause XSpec to issue error messages. For example:

Having successfully used XSpec for testing other parts of our XSLT code base, we wanted to have unit testing for the web application, too. The rest of this paper describes the approach we used for testing this Saxon-JS application using XSpec, an alternative approach that we explored but did not use, and the reasons for our choice.

High-Level File Structure to Enable Substitution

XSLT import precedence provides a straightforward architecture for many of the substitutions we needed. Instead of making the XSpec file directly test the production XSLT transform, we used the following set of files:

Figure 3: High-Level File Structure

Alternatively, an XSpec element named <x:helper>, implemented but not yet released as of this writing, will enable you to achieve the same result without requiring a test harness as a separate file.

Case 1: Creating Result Documents

As stated earlier, Saxon-JS applications commonly use <xsl:result-document> instructions to replace or augment the content of an element in the HTML page, but XSpec has no HTML page at its disposal. We created a named template that, in production, does nothing but execute <xsl:result-document> instructions. The production XSLT transform consistently calls this named template instead of executing <xsl:result-document> instructions directly. The purpose of this named template is to provide a level of indirection that enables substitution for testing purposes.

  <!-- Populate specified portion of the HTML page with specified content.
       The method param is 'ixsl:replace-content' or
       'ixsl:append-content'. -->
  <xsl:template name="mt:result-document" as="empty-sequence()">
    <xsl:param name="href" as="xs:string" required="yes"/>
    <xsl:param name="content" as="item()*" required="yes"/>
    <xsl:param name="method" as="xs:string" select="'ixsl:replace-content'"/>
    <xsl:result-document href="{$href}" method="{$method}">
      <xsl:sequence select="$content"/>
    </xsl:result-document>
  </xsl:template>

Our chosen substitute behavior is to emit a processing instruction followed by the content that was passed in. The processing instruction describes what would happen in production. The next code excerpt shows the test utility file's code for the mt:result-document template.

  <!-- Output a message and the content of the result document. -->
  <xsl:template name="mt:result-document" as="item()*">
    <xsl:param name="href" as="xs:string" required="yes"/>
    <xsl:param name="content" as="item()*" required="yes"/>
    <xsl:param name="method" as="xs:string" select="'ixsl:replace-content'"/>
    <xsl:processing-instruction name="tst" expand-text="yes">In production, would execute
      xsl:result-document href={$href}
      method={$method}
    </xsl:processing-instruction>
    <xsl:sequence select="$content"/>
  </xsl:template>

During XSpec testing, any XSLT code that calls the mt:result-document template executes the testing version. This accomplishes two things: it avoids an XSpec error and it provides information that XSpec can verify. The following two <x:expect> elements illustrate how XSpec can confirm our expectations about content that the production XSLT transform inserts into the HTML page.

<x:expect label="h1 gets populated" test="$x:result[1]">
  <?tst In production, would execute
    xsl:result-document href=#listpage_h1
    method=ixsl:replace-content
  ?>
</x:expect>
<x:expect label="with correct text."
  test="$x:result[2]" select="'Functions'"/>

These <x:expect> elements do not directly check that the HTML <h1> element in the web application gets populated with the correct text. Instead, they check for indirect indications that the web application behaves as expected. Designing indirect indications that give you confidence about the production behavior is part of how you apply mocking techniques.

Still, it is possible to have defects in the <h1> element that these XSpec assertions would not detect. For example, if the HTML page had no element with id="listpage_h1" due to a bug, Saxon-JS would not find the spot on the page to put the text content. XSpec would report that the XSLT code exhibited the expected behavior, but the web application would still have a defect. Just because a defect can go undetected in a unit test, that does not mean the test is not worthwhile. When you design your testing strategy, factors you typically consider include:

Case 2: Calling an Extension Function

The test utility file contains substitute definitions of these interactive XSLT functions that we use in production: ixsl:apply, ixsl:call, ixsl:eval, ixsl:location, ixsl:page, ixsl:query-params, and ixsl:window. Each substitute definition has the same signature as the original function, documented at [S3]. We added each of these substitute definitions to the test utility file as the need arose.

Our substitute behaviors for Interactive XSLT extension functions have these goals:

For each goal, we offer an example showing where the goal applied and how we used code in the test utility file and XSpec file to accomplish the goal.

  <xsl:function name="ixsl:location" as="xs:string">
    <xsl:message terminate="yes">
      We should not get here, because calls to
      ixsl:location are done only in globals that the XSpec file overrides
    </xsl:message>
    <xsl:value-of select="''"/>
  </xsl:function>

  <xsl:try>
    <xsl:sequence select="ixsl:window() => ixsl:call(
      'history.pushState',[$new_url,0,$new_url])"/>
    <xsl:catch/>
  </xsl:try>

  <xsl:function name="ixsl:apply" as="item()?">
    <xsl:param name="function" as="item()"/>
    <xsl:param name="arguments" as="array(*)"/>
    <xsl:processing-instruction name="tst">In production, would call JavaScript function</xsl:processing-instruction>
  </xsl:function>

  <x:expect
    label="Left nav scrolls to the correct product via JavaScript call."
    test="$x:result[last()]">
    <?tst In production, would call JavaScript function?>
  </x:expect>

  <!-- Mark the default active radio button. -->
  <!-- The ixsl:page() output reflects:
  - In the browser, the result of the mt:result-document call earlier
  - In XSpec, the fixed $html_el value. -->
  <xsl:for-each select="ixsl:page()//input
    [@name='blocktype']
    [@value=$configuration?blocktype]">
    <xsl:call-template name="mt:set-attribute">
      <xsl:with-param name="attrname" select="'checked'"/>
      <xsl:with-param name="attrvalue" select="''"/>        
    </xsl:call-template>
  </xsl:for-each>

  <xsl:param name="html_el" as="element()"/>

  [...]

  <xsl:function name="ixsl:page" as="node()">
    <xsl:sequence select="root($html_el)"/>
  </xsl:function>

  <x:expect label="The 'All' radio button has 'checked' attribute"
    test="$x:result[3]">
    <input checked="" type="..." name="blocktype"
    value="all" class="..."/>
  </x:expect>

Case 3: Executing an Extension Instruction

Substituting for extension instructions requires a little more work than substituting for extension functions. The goals are similar, however:

The extension instruction that schedules asynchronous calls to a named template is particularly challenging to integrate with XSpec, and we discuss that in this section.

  <!-- Set specified object to specified style. -->
  <xsl:template name="mt:set-style" as="empty-sequence()">
    <xsl:param name="stylename" as="xs:string"/>
    <xsl:param name="stylevalue" as="xs:string"/>
    <xsl:param name="object"/>
    <ixsl:set-style name="{$stylename}" select="$stylevalue"
    object="$object"/>
  </xsl:template>

  <xsl:template name="mt:set-style">
    <xsl:param name="stylename" as="xs:string"/>
    <xsl:param name="stylevalue" as="xs:string"/>
    <xsl:param name="object"/>
    <xsl:processing-instruction name="tst" expand-text="yes">Set object's
      {$stylename} style to {$stylevalue}</xsl:processing-instruction>
  </xsl:template>

  <x:expect label="Doc survey was hidden temporarily."
    test="$x:result[self::processing-instruction()][1]">
    <?tst Set object's visibility style to hidden?>
  </x:expect>

  <!-- In context node (presumed to be in HTML DOM),
  set specified attribute to specified value. -->
  <xsl:template name="mt:set-attribute" as="empty-sequence()">
    <xsl:param name="attrname" as="xs:string"/>
    <xsl:param name="attrvalue" as="xs:string"/>
    <!-- XSpec override also has html_el parameter. -->
    <ixsl:set-attribute name="{$attrname}" select="$attrvalue"/>
  </xsl:template>

  <!-- Mimic setting the specified attribute on the HTML element passed in. -->
  <xsl:template name="mt:set-attribute">
    <xsl:param name="attrname" as="xs:string"/>
    <xsl:param name="attrvalue" as="xs:string"/>
    <xsl:param name="html_el" select="." as="element()"/>
    <xsl:if test="exists($html_el)">
      <xsl:element name="{local-name($html_el)}"
        namespace="{namespace-uri($html_el)}">
        <xsl:attribute name="{$attrname}" select="$attrvalue"/>
        <xsl:copy-of select="$html_el/@*[not(name()=$attrname)]"/>
        <xsl:copy-of select="$html_el/node()"/>
      </xsl:element>
    </xsl:if>
  </xsl:template>

  <x:call template="mt:top_nav_targets">
    <x:param name="html_el" select="[...]"/>
    <x:param name="configuration" select="[...]"/>
  </x:call>
  <!-- The mt:set-attribute template's XSpec override
  returns the modified li element. -->
  <x:expect label="Functions button has class attr to mark it as active"
    test="exists(//li[@class = 'crux_resource active']/a[. = 'Functions'])"/>

  <!-- Add capability filter to left navigation area. -->
  <xsl:try>
    <ixsl:schedule-action wait="1">
      <xsl:call-template name="mt:extcap_filter">
        <xsl:with-param name="configuration" select="$configuration"/>
      </xsl:call-template>
    </ixsl:schedule-action>
    <xsl:catch>
      <xsl:call-template name="mt:extcap_filter">
        <xsl:with-param name="configuration" select="$configuration"/>
      </xsl:call-template>
    </xsl:catch>
  </xsl:try>

  <!-- Schedule processing for next product -->
  <xsl:try>
    <ixsl:schedule-action wait="1">
      <xsl:call-template name="mt:content_area_aboveproduct_block">
        <xsl:with-param name="configuration" select="$configuration"/>
        <xsl:with-param name="index" select="$index + 1"/>
      </xsl:call-template>
    </ixsl:schedule-action>
    <xsl:catch>
      <xsl:processing-instruction name="tst">In production, would schedule next call to
      mt:content_area_aboveproduct_block, for product
      '<xsl:value-of select="$product_html_paths[$index +1]"/>'</xsl:processing-instruction>
    </xsl:catch>
  </xsl:try>

Simple Example

Consider a simple Saxon-JS application that operates on an HTML page whose body contains an unordered list.

<body>
  <h1>My App</h1>
  <p>Here is a list.</p>
  <ul>
    <li><span>First item</span></li>
    <li><span>Second item</span></li>
    <li><span>Third item</span></li>
  </ul>
</body>

Suppose you have an XSLT template that highlights one of the list items by coloring its text red and its background yellow. To demonstrate multiple approaches, the template uses both the <ixsl:set-style> and <ixsl:set-attribute> extension instructions.

  <xsl:template name="mt:adjust_color">
    <xsl:param name="index" as="xs:integer" select="xs:integer(1)"/>
    <xsl:for-each select="ixsl:page()//ul/li[position()=$index]/span">
      <ixsl:set-style object=".." name="background-color" select="'yellow'"/>
      <ixsl:set-attribute name="style" select="'color:red;'"/>
    </xsl:for-each>
  </xsl:template>

Consider the following XSpec test scenarios for the mt:adjust_color template. The last <x:expect> element's assertion is false, because the text turns red, not blue.

  <x:scenario label="mt:adjust_color template">
    <x:call template="mt:adjust_color"/>
    <x:expect label="turns first li red"
      test="ixsl:style(ixsl:page()//ul/li[1]/span)?color"
      select="'rgb(255, 0, 0)'"/>
    <x:expect label="with a yellow background"
      test="ixsl:style(ixsl:page()//ul/li[1])?background-color"
      select="'rgb(255, 255, 0)'"/>
  </x:scenario>
  <x:scenario label="mt:adjust_color template with index=2">
    <x:call template="mt:adjust_color">
      <x:param name="index" select="2"/>
    </x:call>
    <x:expect label="turns second li blue"
      test="ixsl:style(ixsl:page()//ul/li[2]/span)?color"
      select="'rgb(0, 0, 255)'"/>
  </x:scenario>

In the existing architecture for an XSpec test suite for an XSLT transform, the XSpec infrastructure compiles your XSpec file into a generated XSLT file that imports the XSLT stylesheet you are testing, runs your code according to the test scenarios, and compares the results with your expectations. The XSpec infrastructure also formats the results into an HTML report. In this example, we envision an enhanced XSpec infrastructure using multiple files:

While developing this example, we wanted to explore how closely we could align with the existing XSpec infrastructure. We found that the mapping of basic XSpec code units (scenarios and assertions) to compiled code units could be the same, but the way the units of compiled code call each other had to be different. For a simple XSpec file structured as in this example, the existing XSpec compiler produces one named template per scenario and one additional named template per <x:expect> assertion. Such a mapping works well in this example. However, whereas the existing XSpec compiler checks the assertions immediately after running the scenario, the test for the Saxon-JS application should let the browser regain control and render the scenario behavior before the test starts to check assertions. In other words, you should not run the code being tested and check assertions in the same transform operation.

In Saxon-JS version 1.2, the SaxonJS.transform method, which invokes Saxon-JS, lets you specify a callback function that executes when the browser regains control after processing an XSLT transform. Typically, this callback function is a suitable vehicle for calling Saxon-JS again to check the assertions (although scheduled template calls would pose challenges, just as in the mocking approach). In the JavaScript code sample below, the functions named scenario1assertions, scenario2, scenario2assertions, and scenario2done are all callback functions for invocations of SaxonJS.transform. Note that SaxonJS.transform does not support this callback syntax in Saxon-JS version 2; the same underlying requirement can probably be satisfied using different syntax, but we did not pursue that.

Another question that arose during development of this example was how different parts of the process would transfer data, without writing intermediate files for a subsequent step to read. What data does the transform that checks assertions need to obtain from the transform that executed the parent scenario? How do we store results of each assertion so we can later format the report of the results? Or can we avoid the need for storage by incrementally building the report? Our approach in this example uses an XSLT global parameter ( slabel, for scenario label) to transfer data from the scenario transform to the corresponding assertion transform. Also, the example uses the <xsl:message> instruction to transfer data from XSLT back to JavaScript. When running each assertion, the XSLT outputs the results as a message that the JavaScript code captures using the deliverMessage option that is part of the SaxonJS.transform method. The messages from various assertions accumulate in a JavaScript variable named alltestresults, whose content is eventually passed to the code that formats the report.

var alltestresults = ""; // Results from xsl:message in assertion transforms
var testsef = "unittest/browser_based_attribute_test-generated.sef";
window.onload = function scenario1() { // Run scenario 1.
  SaxonJS.transform(
    {
     stylesheetLocation: testsef,
     initialTemplate: "Q{http://www.jenitennison.com/xslt/xspec}sc1",
     deliverMessage: function(msg) { slabel = msg + '...'; }
    },
   function scenario1assertions() { // Check assertions from scenario 1.
     SaxonJS.transform(
       {
        stylesheetLocation: testsef,
        initialTemplate: "Q{http://www.jenitennison.com/xslt/xspec}sc1ex1",
        templateParams: {
          "Q{http://www.jenitennison.com/xslt/xspec}slabel": slabel
        },
        deliverMessage: function(msg) { alltestresults += msg + '\n'; }
       },
       function scenario2() { // Run scenario 2, including setup.
         SaxonJS.transform(
          {
           stylesheetLocation: testsef,
           initialTemplate: "Q{http://www.jenitennison.com/xslt/xspec}sc2",
           deliverMessage: function(msg) { slabel = msg + '...'; }
          },
         function scenario2assertions() { // Check assertions from scenario 2.
           SaxonJS.transform(
             {
              stylesheetLocation: testsef,
              initialTemplate: "Q{http://www.jenitennison.com/xslt/xspec}sc2ex1",
              templateParams: {
                "Q{http://www.jenitennison.com/xslt/xspec}slabel": slabel
              },
              deliverMessage: function(msg) { alltestresults += msg + '\n'; }
             },
             function scenario2done() { // Display all results in HTML body.
               mytext = "<root>" + alltestresults + "</root>";
               modifiedformatter = "framework_xspec_modified/" +
                  "src/reporter/format-xspec-report.sef";
               SaxonJS.transform(
                 {
                  stylesheetLocation: modifiedformatter,
                  sourceText: mytext,
                  initialTemplate: "xspec_for_saxonjs"
                 }
               ) // end of SaxonJS.transform() for report formatting
             } // end of scenario2done fcn
           ); // end of SaxonJS.transform() for assertions from scenario 2
         } // end of scenario2assertions fcn
        ); // end of SaxonJS.transform() for scenario 2 
       } // end of scenario2 fcn
     ); // end of SaxonJS.transform() for assertions from scenario 1
   } // end of scenario1assertions fcn
  ); // end of SaxonJS.transform() for scenario 1
} // end of scenario1 fcn

When you open the HTML file for this example in a browser, it briefly flashes the first two screens below to run the two scenarios, and then settles on the report.

Figure 26: Running Scenario 1

Figure 27: Running Scenario 2

Figure 28: Report of Results

Some Ingredients for Browser-Based Approach

Extrapolating from the example, we conjecture that browser-based XSpec testing for Saxon-JS applications might require the following ingredients. This list is not comprehensive.

Benefits and Costs of Mocking Approach

In our experience, benefits of the mocking approach include:

Costs of the mocking approach include:

Benefits and Costs of Browser Approach

Benefits of the browser approach include:

Costs of the browser approach include: