Balisage Paper: Web-Based Job Aids Using RESTXQ and JavaScript for Authors of XML Documents

Web Form and Step-by-Step Procedure

To use this application, authors submit a form in a web browser that indicates the file to move, the new location, and which of two computing clusters they use. In the browser, the application displays a report describing a procedure for moving the file and addressing all its dependencies. The following sample report indicates in step 4 that moving the XML file necessitates moving its referenced graphics files to comply with business rules. In step 5, the report indicates which references to the relocated graphics files also need to be updated in other XML source files.

The web interface was more appealing than alternatives for two reasons:

The next three sections discuss the technical details of how RESTXQ turns XQuery functions into web applications, ways to separate analysis code from formatting, and unit testing techniques.

RESTXQ Usage in XQuery Module

The file reorganization application is a simple web application, with a single form that leads to a static report. This application was a good way to start exploring the RESTXQ web API, building on our group's prior experience with XQuery. RESTXQ is a good fit for the technical requirements of this application, providing a way to make a web browser call XQuery code when the author accesses a URL associated with the code. The XQuery code can access XML files in a file system, or in a database compatible with the RESTXQ implementation. In this case, there was already a BaseX database associated with the XML source files.

Using the BaseX [B1] engine for the XQuery language, we developed a single module file that defines a series of XQuery functions. Here are how the functions correspond to the author's actions in the browser:

We use the same BaseX database to do bulk reporting on file dependencies, chiefly to help identify files that are shared or unused. Instead of using a web application for the task, the bulk reporting uses cron jobs that create Excel reports. We chose this approach because analyzing the whole file set is time-consuming and because the requirements for bulk reporting do not make a compelling argument for a web application.

Separation of Analysis and Formatting

If you work with XML documents or HTML5, you are familiar with the idea of separating semantics or content from presentation or styling. By design, the XQuery module for this application mostly separates the data analysis, such as looking for file dependencies in the XML documents, from the formation of HTML markup for the browser. Separating these concerns makes it easier to write unit tests for the analysis code and easier to change the appearance or organization of the HTML report.

For example, one step in the report instructs authors to update XInclude references from the XML file they are moving to other XML files. Behind this step are these XQuery functions:

The unit test module has these corresponding test functions:

Unit Testing for Analysis Functions

Multiple unit testing frameworks exist for XQuery, some of which are vendor-specific [W1][X1][V1]. Because we use BaseX to develop and deploy the web application, we use the BaseX unit testing module for testing [U1]. We have unit tests for all the analysis functions and most other functions. The unit tests reside in a separate XQuery module that imports the main XQuery module for the file reorganization application.

For some functions, unit testing is straightforward; you call the function with representative but simple input arguments, and use unit:assert() or unit:assert-equals() to check that the function returns an expected result.

For other functions, we use a set of XML files with known dependencies on other XML files to be able to check that the analysis functions find the correct set of dependencies. Basing tests on production XML documents would make the tests unstable because the XML documents change over time. Instead, we use a small set of test documents. The test module includes a setup function that creates an XQuery database with the test documents. In the declaration of this setup function:

After the setup function executes, unit tests can use the testing database when calling the functions under test.

Small Multiples and Iterative Behavior

The current prototype of this application uses the concept of small multiples [T1] to depict author-coded links among topics as geometric icons. Icons in the application vary in their characteristics: shape, color, and fill. The icon characteristics depend on:

The following legend shows the combinations of icon characteristics that can occur and what they mean.

For example, the following row of icons shows that authors coded 25 links in a certain topic.

The fill, shape, and color of the icons indicate that, of those 25 links:

The idea is that in the web application, authors can start with a topic of interest, click Submit to display the set of icons that depict topic links, and use the depiction to learn about topic connectivity in the network of topics. They can hover over any icon to see a tooltip that indicates which topic the icon represents. They can click an icon to display the corresponding set of icons for that topic, as a new row at the bottom. By iterating, they can follow an end user's possible clickstream in the HTML deliverables or cover a desired subset of their topics. Using their knowledge of the subject matter and relative importance of each topic, they can assess whether the connectivity characteristics of a given topic are appropriate. After iterating a few times, the results might look like this:

This application traverses a large set of XML documents at the back end, while supporting graphics and dynamic behavior at the front end. Using a web-based front end lets us take advantage of third-party JavaScript libraries.

Boundaries Between XQuery and JavaScript

The application relies on XQuery to find links in the XML documentation and identify their characteristics. We integrated the D3 library, a popular JavaScript library for visualizing data and manipulating documents based on data [B3], with XQuery in a single web application. We also simplified the code using the jQuery JavaScript library [J1]. In this application, JavaScript is the main driver, and it interacts with XQuery at these discrete points:

This diagram summarizes how JavaScript, XQuery, and the author's mouse operations transfer control back and forth in this application.

Calling from JavaScript to XQuery

One way to call XQuery from JavaScript is to use the AJAX .load() method in jQuery [J2]. When this method accesses a URL associated with a RESTXQ path, the XQuery engine invokes the XQuery function that maps to that URL. The XQuery function returns markup. The .load() method regains control and inserts the returned markup into the page.

For example, suppose the HTML page contains the following markup.

<div class="xmlcontainer">
   <!–- XQuery will populate this part of the page. -->
</div>

The following JavaScript code selects the <div> element using a jQuery selector, and loads a URL.

$(".xmlcontainer")
   .load("my-restxq-path/mydocID/mytargetID");

Through the RESTXQ path, "my-restxq-path/mydocID/mytargetID", the .load() method calls the XQuery function associated with that path. The XQuery module contains a function similar to the next excerpt, receiving two function arguments via the {$docID} and {$targetID} path templates in the %rest:path annotation.

declare
  %rest:path("my-restxq-path/{$docID}/{$targetID}")

   function linkvis:linkinfo(
      $docID as xs:string,
      $targetID as xs:string)
      as element()
{  
   (: Body of XQuery function goes here :)
};

When JavaScript regains control, it inserts the return value of the XQuery function into the HTML <div> element.

If further JavaScript processing is necessary after the XQuery function has finished, the extra JavaScript code can go in a callback function that becomes the third argument of .load(). The callback executes when the AJAX request completes.

$(".xmlcontainer")
   .load("my-restxq-path/mydocID/mytargetID", {}, function() {

      // JavaScript code to execute after XQuery function finishes

   });

Passing Data from XQuery to JavaScript

In this case, the return value of the XQuery function is not the HTML markup for the graphics we want the application to display. It is the job of D3 to create that HTML markup. XQuery gathers data about links in documents and passes the data back to JavaScript, enabling D3 to create HTML markup based on the data.

<script>
var mydata =
   [
      {
         "docID":"import_export",
         "targetID":"buh9wft",
         "topictitle":"Getting Started with Datastore"       
      },
      {
         "docID":"import_export",
         "targetID":"bhg4l7w-1",
         "topictitle":"Getting Started with MapReduce"         
      }
   ];
</script>

<script>{
   "var mydata = " ||
   serialize(arguments for serializing data...)
   || ";"
}</script>

declare
  %rest:path("my-restxq-path/{$docID}/{$targetID}")

   function linkvis:linkinfo(
      $docID as xs:string,
      $targetID as xs:string)
      as element()
{  
   (: XQuery computations go here. :)

   return
      <script>{
         "var mydata = " ||
         serialize(arguments for serializing data...)
         || ";"
      }</script>
};

serialize(
   <json type="array">{
      for $thislink in $linkseq (: Define $linkseq elsewhere :)
      return
         <_ type="object">
            <docID>{ (: Query logic, such as $thislink/@docID :) }</docID>
            <targetID>{ (: Query logic... :) }</targetID>
            <topictitle>{ (: Query logic... :) }</topictitle>            
         </_>
   }</json>,
   <output:serialization-parameters
      xmlns:output="http://www.w3.org/2010/xslt-xquery-serialization">
      <output:method value="json"/>
   </output:serialization-parameters>
)

// Start a new row of data.
var svg = d3.select(".new").append("svg")
   .attr("height",rowHeight)
   .attr("width","100%");
var symdiv = svg.append("g").attr("class","symdiv");

// Insert <g> elements with data bound to them.
var symbolgroups = symdiv.selectAll("g").data(mydata).enter()
  .append("g")
    .attr("transform","translate(" + labelWidth + ",0)")
    .attr("class","gsymbols")
    .on("click",symbolclick);

// Further code inserts symbols, labels, and tooltips.