Balisage Paper: XML Pipeline Processing in the Browser

XProc Compliance

A large part of the code base is shared between the Java and GWT versions. This not only reduces code duplication, but also ensures that both versions of Calumet are aligned in terms of functionality and the level of XProc support. As of July 2010, the GWT version supports 34 out of the total 41 steps from the standard XProc step library W3C XProc (the Java version supports all steps); Table I provides an overview of the missing functionality.

The GWT version has most of the features of the original Java version: from high compliance with the XProc specification to extensibility and customizability. For instance, it is possible to register extension steps or custom URI handlers with both the Java and GWT versions.

Table II below summarizes the current results of running the GWT version of Calumet against the XProc Test Suite XProc Test Suite. The XProc Test Suite comprises four main categories of tests:

As can be seen from Table II, GWT Calumet scores relatively well with the required tests (passing over 80% of the tests), as well as with the serialization and extension tests. The biggest gap is in the optional tests that depend on functionality that is difficult to implement in JavaScript (for instance, XQuery or schema validation) and some of which will probably not be supported in the GWT version of Calumet any time soon. However, this limitation can be overcome easily in the web browser environment by using the standard XProc HTTP Request step and invoking a server-side service that provides the missing functionality. Alternatively, the missing steps themselves can be implemented as callbacks to the server-side if necessary.

Preliminary Performance Results

The GWT version of Calumet has been tested successfully with most of the major browsers. Work is currently underway on implementing the gaps in the standard XProc step library support and also on improving the performance and reducing the size of generated JavaScript. The current size of the JavaScript code (without compression) is about 550 KiB, but we are confident that this can be brought down - the original Java code was not written with GWT in mind and many of the programming constructs used in the code are perhaps too generic for efficient translation to JavaScript.

Regarding the performance of the GWT version of the processor, there are noticeable (and expected) differences between different browsers, especially with more complex pipelines that involve evaluating many XPath expressions or creating large numbers of intermediate XML documents. But surprisingly, the most expensive part of running an XProc pipeline turned out to be the initial phase: parsing the pipeline source, resolving imports, performing the static analysis, and establishing the evaluation order of the steps in the pipeline. While executing the pipeline itself generally requires only milliseconds or tens of milliseconds (in Gecko- or WebKit-based browsers on a 2.33 GHz dual-core workstation with 4 GB of RAM), preparing and statically checking the pipeline often takes considerably more time (from 50 to 300 milliseconds depending on the web browser and the complexity of the pipeline); a clear area for further performance optimizations.

Having the possibility to run - and test - the same code base in two completely different environments (Java and in-browser JavaScript) actually resulted in an interesting synergistic effect between the two versions of the processor: code that performs reasonably well in Java may prove to be a performance bottleneck in JavaScript (or the other way around); but fixing the issue usually has a positive effect in both environments.

AJAX and Dynamic Web Applications

Modern dynamic web applications often rely on heavy use of JavaScript and AJAX for interacting with the server-side. In the XML-based model, the result of an AJAX request is an XML document that needs to be processed in some way; most often by traversing the XML structure using the DOM API. For example, in a web shop application, an AJAX callback might be used to call a productlist service and to display the results in a dynamically constructed HTML table. This would typically involve iterating over the result elements in the returned XML document and creating a table row for each result.

While this approach works, hard-coding the AJAX XML response processing in JavaScript may not always be the best option. First, it requires knowledge of the AJAX and DOM APIs as well as awareness of various browser quirks, and second, the processing model becomes set in stone, often hard to change or even understand. This may be critical for larger-scale enterprise applications, or in any application in general that requires a flexible (and maintanable) processing model the developers can build on.

The example below attempts to translate the usual AJAX request - process response - update host page pattern to XProc. The XProc pipeline starts with an HTTP request to the productlist service. The XML document returned by the service is then processed by creating an HTML table row for each product element in the document. After that, all table rows are inserted into a table wrapper which is then injected into the host page. In the example, a custom extension is used that makes it possible to address elements of the host page using their ID. In our case, the p:store step effectively replaces the element with the ID search-results by the generated HTML table.

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
                xmlns:c="http://www.w3.org/ns/xproc-step"
                xmlns:xhtml="http://www.w3.org/1999/xhtml"
                version="1.0">

  <p:http-request>
    <p:input port="source">
      <p:inline>
        <c:request method="GET" href="/productlist"/>
      </p:inline>
    </p:input>
  </p:http-request>

  <p:for-each name="for">
    <p:iteration-source select="//product"/>
    <p:output port="result" sequence="true"/>

    <p:variable name="product-name" select="/product/@name"/>

    <p:string-replace match="xhtml:td/text()"
                      replace="concat('&quot;', $product-name, '&quot;')">
      <p:input port="source">
        <p:inline>
          <xhtml:tr>
            <xhtml:td>tmp</xhtml:td>
          </xhtml:tr>
        </p:inline>
      </p:input>
    </p:string-replace>
  </p:for-each>

  <p:insert match="xhtml:tbody" position="first-child">
    <p:input port="source">
      <p:inline>
        <xhtml:table id="search-results">
          <xhtml:tbody/>
        </xhtml:table>
      </p:inline>
    </p:input>
    <p:input port="insertion">
      <p:pipe step="for" port="result"/>
    </p:input>
  </p:insert>

  <p:store href="domid:search-results"/>

</p:declare-step>

XML is not the only format that can be used in AJAX environments. JSON, or JavaScript Object Notation JSON, is an increasingly popular, text-based format for data interchange that is both light-weight and very easy to work with in JavaScript (on the syntactic level, JSON is just a subset of JavaScript). In the web applications world, JSON and XML can be seen as competing formats where JSON is often presented as the simpler and more efficient alternative to XML. Leaving aside the (often emotional) arguments between the JSON and XML proponents as to which format is better, it is obvious that both JSON and XML have much in common: they are both open formats, readable by machines and humans; they are self-describing; and they are well supported by a wide variety of tools. The strength of JSON lies mainly in representing simple data structures, where XML is often seen as too heavy-weight; XML, on the other hand, is generally better suited for semistructured data.

For the use case discussed in this section, one could argue that using AJAX and JSON would be a much better fit than XProc. Indeed: the 40 lines of XProc code (plus the necessary GWT or JavaScript code for actually executing the pipeline) could probably be replaced with only a handful of lines of simple JavaScript. The overhead of an XProc solution is also likely to be much higher compared to plain JavaScript: the XProc pipeline needs to be parsed and analyzed first, and the data is then processed using relatively expensive (at least in the browser) XML manipulations.

So is there a benefit in adopting the inherently heavier XProc approach where a simpler and more efficient alternative exists? We argue there is, although it depends strongly on the particular use case. XProc is not a hammer for everything: it is first and foremost an XML processing language, and it should be used as such. Client-side XProc therefore makes most sense in user interfaces that are XML-driven, consume or produce XML data, or require non-trivial XML processing. In other situations, other approaches may be more appropriate.

The XProc pipeline above is admittedly very minimal, and from the dynamic web applications perspective it does not show anything new that could not be done with existing approaches. The interesting part lies in what XProc can offer beyond this point. While the example pipeline may seem as unnecessary overhead for the simple problem (populating an HTML table based on data returned from the server), the perspective begins to shift when further processing of the server-side results is needed. The XProc pipeline can query or transform the data easily, enrich it, or combine it with data obtained from other services; all using a unified, declarative, and flexible XML processing model.

Client-side XML Presentation

A common task in XML-based web applications is presenting the XML data (either static or dynamically generated) to the user. Typically, this is done by transforming the XML data on the server-side to HTML or some other format understood by the browser. Most of these transformations can be expressed in the form of XProc pipelines, and with a reasonably compliant client-side XProc implementation, it should be possible to move the processing to the client-side completely. Delegating the rendering to the client-side can not only reduce the load on the server, but in many cases, it can also simplify the server-side functionality in general and make it less coupled with the front-end technology.

The pipeline below takes an XML document, resolves possible XInclude references, and applies an XSLT stylesheet to the resolved document; all with standard XProc and completely in the browser.

<p:pipeline xmlns:p="http://www.w3.org/ns/xproc"
            version="1.0">

  <p:xinclude/>

  <p:xslt>
    <p:input port="stylesheet">
      <p:document href="style.xsl"/>
    </p:input>
  </p:xslt>

</p:pipeline>

The previous pipeline is obviously a very simple one, but the potential of using XProc as a client-side XML rendering tool is obvious: one can imagine using XProc for bringing popular XML vocabularies like, for instance, DocBook DocBook or DITA DITA to the browser.

The example below shows a pipeline that takes a DITA topic and returns its HTML rendition. While the pipeline itself is trivial, the dita:topic-to-xhtml step will most likely perform rather complex XML processing: from resolving the various DITA link types to content filtering to applying an XSLT stylesheet. Or... maybe not: the black-box nature of XProc steps provides great freedom by allowing different implementations of the same step interface - which is not only convenient when writing (and testing) the pipelines, but it also makes it possible to adapt the pipelines to the needs of a particular user audience or to different browser environments. Thus, the pipeline below can, for instance, do full-blown client-side DITA processing in web browsers that are known to be fast (or compliant) enough, and delegate to the server-side in other cases.

<p:pipeline xmlns:p="http://www.w3.org/ns/xproc"
            xmlns:dita="http://example.org/ns/dita"
            version="1.0">

  <p:import href="dita-lib.xpl"/>

  <dita:topic-to-xhtml/>

</p:pipeline>

Interactive Processing

An interesting application of XProc pipelines in web browser is using them as a driver for interactive client-side procedures. In such an obvious user interface-oriented platform as a web browser, it is possible to imagine XProc extension steps that would add interaction with the end-user to the XML processing logic - for example, by displaying dynamically generated dialogs or messages on the screen. This would make XProc a simple (yet powerful) alternative to other approaches for representing interactive processes: from simple data collection procedures to complex and often highly dynamic maintenance and diagnostic procedures found, for instance, in the military and in the aerospace industry.

In the area of interactive processing, client-side XForms is a technology that combines exceptionally well with XProc: as an XML-based standard, it is very easy to load - or even dynamically generate - XForms-based dialogs in XProc pipelines. Similarly, the XForms submissions, which are XML documents as well, can be processed naturally in XProc.

The pipeline below shows how XForms could be used with XProc in an imaginary aircraft maintenance system; in this case, the example is a simple circuit breaker check procedure. The pipeline has two options, an aircraft model number and its variant. Depending on a specific combination of the model and the variant, the maintenance mechanic is presented with a dialog (an XForm) that allows him to enter information about the state of the circuit breaker. When the mechanic submits the dialog, the pipeline displays a message that tells the mechanic to switch on the breaker if it was in the OFF position.

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
                xmlns:am="http://example.org/ns/aircraft-maintenance"
                xmlns:xf="http://www.w3.org/2002/xforms"
                version="1.0">

  <p:option name="model" required="true"/>
  <p:option name="variant" required="true"/>

  <p:import href="aircraft-maintenance-lib.xpl"/>

  <p:choose>
    <p:when test="$model = '777' and ($variant='200' or $variant='200ER')">
      <am:dialog href="forms/777-200-circuit-breaker-check.xfm"/>
    </p:when>
    <p:when test="$model = '777'">
      <am:dialog href="forms/777-circuit-breaker-check.xfm"/>
    </p:when>
    <p:when test="$model = '787'">
      ...
    </p:when>
    ...
  </p:choose>
  
  <p:choose>
    <p:when test="/state = 'OFF'">
      <am:message>
        <p:input port="source">
          <p:inline>
            <p xmlns="http://www.w3.org/1999/xhtml">                                                              
              Switch the circuit breaker to the ON position.
            </p>
          </p:inline>
        </p:input>
      </am:message>
    </p:when>
    <p:otherwise>
      <p:sink/>
    </p:otherwise>
  </p:choose>

</p:declare-step>