Balisage Paper: XQuery is not (just) a query language: Web application development with XQuery

Model

The model is responsible for interacting with the database; no other component should create, read, update or delete data (think “no CRUD outside the model”). In this sense it acts as a kind of API (or like an encapsulated object, in object-oriented terms), which other modules call upon without needing to know anything about the database or how to interact with it. Ideally this separation of concerns is enforced to the point that one could swap out the underlying database server for a different one without touching any code in the controller(s) or views. (Conversely, we should be able to overhaul the presentation or adjust business logic without touching any code in the model(s).)

However, in the context of this discussion, in which data is in XML form, there is an important additional consideration. We have to decide whether the model should not only handle all database interaction but also handle all navigation of the structure of the XML documents, which is to say, whether only the model should contain XPath expressions. This decision is essentially a question of how strictly we want to separate knowledge of the document structure from the knowledge of how the document content should be formatted and presented. For example, let’s say we have this kind of markup, containing information about a book:

<?xml version="1.0" encoding="UTF-8"?>
<doc>
  <metadata>
    <id>ab1geschichtedes03gind</id>
    <name>Gindely, Anton, 1829-1892</name>
    <title>Geschichte des dreissigjährigen Krieges</title>
    <date>1869</date>
    <language code="de">German</language>
    <format>Book</format>
    <callNumber source="oclc">D258 .G49</callNumber>
    <class>History</class>
    <topics>
      <topic>Thirty Years' War, 1618-1648</topic>
    </topics>
  </metadata>
</doc>
          

import module namespace m = "http://balisage.net/ns/Bal2016murr0319/model" at ... ;
...
let $doc := m:get-xml-doc("ab1geschichtedes03gind")
let $title := fn:string($doc/doc/metadata/title)
let $name := fn:string($doc/doc/metadata/name)
          

module namespace m = "http://balisage.net/ns/Bal2016murr0319/model";

(:~ Returns the book title as a string. :)
declare function m:get-title($doc as document-node())
as xs:string
{
  fn:string($doc/doc/metadata/title)
};

(:~ Returns the author name as a string. :)
declare function m:get-name($doc as document-node())
as xs:string
{
  fn:string($doc/doc/metadata/name)
};
          

import module namespace m = "http://balisage.net/ns/Bal2016murr0319/model" at ... ;
...
let $doc := m:get-xml-doc("ab1geschichtedes03gind")
let $title := m:get-title($doc)
let $name := m:get-name($doc)
          

As mentioned above, there is nothing inherently wrong with having a single model for a given web application, but there are some use cases for separating database-related code into multiple models. In situations where a web application needs to work with documents in multiple XML markup languages, it might make sense to have a different MVC model (that is, a discrete XQuery module) for each document type. Similarly, if an application includes full-text searching capability, it is logical to group search-related functions into a separate model. This is especially true if your chosen implementation provides its own functions for searching (as opposed to implementing the W3C XQuery Full-Text recommendation), so that in addition to improving code organization per se the search-specific model is isolating implementation-defined functions, to improve code portability and sharing (more on this below).

View

The role of the view is to provide the presentation, whether for human- or machine-readability. For example, a very commonplace pattern (used by all manner of web applications from Amazon to Zappos) is to provide searching, leading to search results, leading to an item-level page showing all relevant information for a particular item. In this approach, the home page containing the search form, the search results page, and the item-level page are separate views, each one implemented as its own XQuery library module (all of which could be grouped in a single views directory). In a web application designed for display in a browser, most views will return an HTML document, whereas a web service might return data as JSON or XML, which is nonetheless a view.

Because most web-based user interfaces are designed with shared components such as headers, sidebars and footers, it makes sense to have an XQuery module that serves as a “layout” (or to have multiple layout modules, if more than one page layout is needed within the same application). In this scenario the view constructs one or more HTML elements constituting the main content of a page, then passes the element(s) (such as a <div>) to a function in the layout module, which returns a complete HTML document with the main content inserted in the appropriate place within the encompassing page layout. The view then returns the complete HTML document.

Figure 1

Figure 1: A view utilizing a layout to return a complete HTML document

Controller

The controller is responsible for receiving user input and responding accordingly. It “controls” what happens in response to input and how the model(s) and views are involved in that process. In a web application context, the controller accesses the input from the HTTP request (URL parameters, form data, request headers, cookies) and then calls the appropriate view, as indicated by those input values. Since the XQuery language doesn’t provide built-in functions for such HTTP-specific matters, the implementation must supply them, but multiple widely adopted implementations do so. (We will look at some practical examples in BaseX and MarkLogic below.)

Some variations on MVC assert that only the controller should call upon the model to retrieve data, after which the controller passes that data to a view, such that a view never interacts directly with the model.

Figure 2

Figure 2: First option: only the controller calls the model

Figure 3

Figure 3: Second option: any given view can call the model

Ancillary modules

In addition to the core MVC components, additional specialized XQuery library modules are often helpful. A module that acts as a “config file” of global variable declarations allows easy management of settings that might change later or might be needed by multiple modules. Conversely, a “utility” module containing general-purpose, shared functions provides convenience and reduces code duplication. In more complex applications, shared functions that pertain to a particular problem domain or MVC component can be grouped and separated into discrete utility modules.

Using available standardizations

One such technique is to utilize what standardizations are available for XQuery web application development. The EXQuery organization (exquery.org) has proposed several specifications for standardizing functionality across XQuery implementations. Some of these proposals are specific to XQuery web application development, and of these the most widely implemented is RESTXQ. Proposed by Adam Retter in 2012, RESTXQ utilizes XQuery 3.0 annotations to associate URLs with XQuery functions (Retter). In other words, RESTXQ standardizes URL rewriting, the process whereby a clean, user-friendly URL accessed by the user (or by a program accessing a web service such as a REST API) is translated into a different URL that is actually used by the web application behind the scenes. More accurately, RESTXQ doesn't standardize URL rewriting so much as obviate the need for it altogether; instead of rewriting URLs from user-friendly to code-friendly, RESTXQ maps URLs to XQuery functions.

For example, returning to the commonplace behavior described above whereby a web application performs a search, leading to search results, leading to an item-level page for each result, let’s say that any given item-level page is indicated by a URL in the format /id/abc123 where abc123 is a unique identifier for a particular item. In an implementation that supports URL rewriting, such as eXist or MarkLogic (Retter, §3.1.2 and §3.2.2), we could take /id/abc123 and translate it to ?id=abc123 before handing it off to the XQuery processor, so that id is available as an HTTP request parameter name.

(: MarkLogic example :)
let $url:= xdmp:get-request-url()
let $regex-id := "/id/(.+)/?"
return
  if (fn:matches($url, $regex-id)) then
    (: extract identifier :)
    let $id := fn:replace($url, fn:concat("^.*", $regex-id, ".*$"), "$1")
    (: replace "/id/whatever" with "/" (implying "/default.xqy") and append "id=whatever" as a URL parameter :)
    let $separator := if (fn:contains($url, "?")) then "&" else "?"
    let $start := fn:replace($url, $regex-id, "/")
    return fn:concat($start, $separator, "id=", $id)
  (: else if ... :)
  (: else if ... :)
  else $url
          

(: BaseX example :)
declare
%rest:path("/id/{$id}")
function c:get-item-view($id as xs:string)
{
  (: the variable $id has the value "abc123" :)
  ...
};
          

The RESTXQ approach is concise and straightforward, and it locates the URL mapping code at the function declaration itself (not in a separate “rewrite engine” module), leading to code that is easier to read and maintain. RESTXQ is supported in BaseX, eXist, and MarkLogic (Walmsley, p. 430), making it effectively a de facto standard. As a result, the code is portable across multiple implementations and shareable among developers and projects.

The use of RESTXQ can easily be incorporated into an MVC approach to code organization by adding RESTXQ annotations to functions in the controller (since the controller receives input from the HTTP request and takes action accordingly). Fleshing out the preceding example a little more:

(: BaseX example :)
module namespace c = "http://balisage.net/ns/Bal2016murr0319/controller";
import module namespace item = "http://balisage.net/ns/Bal2016murr0319/views/item" at "views/item.xqm";

declare
%rest:path("/id/{$id}")
%output:method("html")
%output:version("5.0")
function c:get-item-view($id as xs:string)
as element(html)
{
  (: the variable $id has the value "abc123" :)
  (: call the item-level view :)
  item:get-html($id)
};
          

Isolating implementation-specific functions

The main barrier to code portability across implementations and code sharing across the XQuery web development community is the necessity of implementation-provided functions. One technique to alleviate this problem is to abstract and isolate such functions. This goal can be achieved by creating one or more library modules containing generically named functions that internally utilize the corresponding implementation-specific functions. Then, throughout the codebase, whenever implementation-provided functionality is needed, we simply call the applicable generic function instead of the implementation-specific one. This approach creates a layer of abstraction, separating and hiding implementation-dependent code from the rest of the codebase. For example, we could create an “implementation” module and set up a generically named function for converting HTTP request parameters to an XQuery map, as shown above in a previous code example. Such a module written for BaseX would look like this:

xquery version "3.1";
(: BaseX example -- this module's filename is implementation-basex.xqm :)
module namespace imp = "http://balisage.net/ns/Bal2016murr0319/implementation";
import module namespace request = "http://exquery.org/ns/request";

declare function imp:http-params()
as map(*)
{
  map:merge(
    for $name in request:parameter-names()
    let $value := request:parameter($name)
    return map:entry($name, $value)
  )
};
          

xquery version "1.0-ml";
(: MarkLogic example -- this module's filename is implementation-ml.xqy :)
module namespace imp = "http://balisage.net/ns/Bal2016murr0319/implementation";

declare function imp:http-params()
as map:map
{
  let $map := map:map()
  let $empty :=
    for $name in xdmp:get-request-field-names()
    let $value := xdmp:get-request-field($name)
    return map:put($map, $name, $value)
  return $map
};
          

(: BaseX example :)
module namespace c = "http://balisage.net/ns/Bal2016murr0319/controller";
import module namespace imp = "http://balisage.net/ns/Bal2016murr0319/implementation" at "implementation-basex.xqm";

(: somewhere in some function ... :)
let $params := imp:http-params()