Balisage Paper: XML Applications on the Web

Introduction

As we have claimed before [B16], current XML technologies provide a full stack of modeling languages, implementation languages, and tools for web applications that is stable, platform independent, and based on open standards. A particular strong point of what we call the X stack is that data are encoded with XML end-to-end and that XML technologies can be used where-ever XML data need to be processed.

Following up on that work, in this paper, we take a closer look at implementation strategies: How can we use XML, XQuery, and SCXML to implement the Model component in a web application that is structured in the Model-View-Component architectural style?

After this introduction, in section “Architecture”, we propose a specific architecture for web applications that follows the Model-View-Controller architectural style and puts the heaviest demands on the Model component. Model components, even when running on the web, are basically conventional software components. Hence, they are fertile grounds for exploring how established software engineering modelling and implementation strategies transfer to the realm of XML technologies.

Then, in section “Domain modeling and implementation strategy for Model”, we get more technical in demonstrating how to do functional decomposition of XQuery functions that perform updates. Functional decomposition is necessary for XQuery to be a straight-forward implementation language for the typical object-oriented designs of a Model component. We also consider, which implications transaction management and database locking have for function calls.

Next, in section “The Model component as an event-driven system”, we investigate how to use UML state diagrams in the design of a Web application and how to integrate an SCXML processor that interprets and executes an SCXML encoding of the state diagram into the XML-based implementation of a typical Model component.

Finally, in section “Outlook: multi-client web applications”, we outline work in progress on a RestXQ extension for BaseX that implements the WebSockets protocol [WSM13]. This extension enables us to make a Model observable and, thus, to realize multi-player games that require server push.

We conclude with a number of discussion points and some final remarks.

The practices proposed in this paper are compatible with domain-driven design and model-driven solutions; they pave the way for XML experts to develop XML-based applications on the web.

Throughout the paper, we provide code to illustrate the techniques that we introduce. We have also applied our principles and strategies in a case study, the game Guess the Number (GN). which is documented in a separate document that is available on request. This case study is intentionally kept simple, so that we can focus on principles without being distracted by more complex XML processing. We have student projects for Blackjack and Mancala and the early GameX [SKB14] that follow the same principles as they evolved and that are technically more complex. The case studies demonstrate how end-user developers who are conversant with XML technologies can create their own web applications.

Architecture

Any web application uses by definition a client-server architecture with the following characteristics:

The implementation of the server component typically relies on a backend component such as a database system. Architecturally, the backend component is layered below the web server, and only the web server communicates with the backend component, using some kind of API. This leads to the typical 3-tier architecture of web applications with any number of web browsers that are dynamically instantiated on user request in Tier 1, a web server in Tier 2 and a backend system in Tier 3. The layering of the three tiers is guaranteed through the pattern of communication between the tiers. Only the two pairs of Tier 1 and 2 and Tier 2 and 3 communicate, the former through HTTP and the latter through some kind of API [F02].

Software engineering has settled on the Model View Controller (MVC) architectural style for systems with a user interface [BD09,F02]. As a first cut at an architecture for XML applications on the web, we decide on an MVC variant called Passive View [F02]. In this variant, the heavy-lifting processing is done by components that are run on the server. The classic benefit of this approach is testability. We choose the Passive View variant so that we can first focus our attention on the specific set of technologies, XML, XQuery, RestXQ and SCXML, that we use on the server.

In the Passive View architecture, there is a specific distribution of responsibilities and a specific communication pattern between the three components.

In Passive View, the responsibilities of the View component are elementary. View displays the state of the application as communicated by Controller and offers user interactions on Controller instruction. View notifies Controller of any user interactions and waits for new information from Controller. Thus, View delegates any real processing to the other two Components.

The Model component holds data and provides access to the data in the form of methods that manipulate the data. Model exposes its functionality through an interface or API. In the most simple cases of web applications, we have a passive model: Methods are called by the Controller of Model; state changes in Model always originate from methods that are called by its Controller. Particularly, there is only one Controller per Model.

Controller receives notification from any View about user interactions. It interacts with Model to handle the notification, converting information from Model into information for View (which data to display, which interactions to offer) and returning this information to View.

The hallmark of the Model View Controller architectural style is that it separates the user interface, represented by View, from the data and functionality of the application, represented by Model. A number of View components can work with the same Model component, mediated through Controller components, without Model having to know anything about the Views that are connecting to it, and each View may retain its own methods of presenting data and interacting with users without having to coordinate with any of the other components.

How can we map the three components Model, View and Controller onto the three tiers of a web application? It is tempting to paint with a broad brush and map Views to Tier 1, Controllers to Tier 2 and Models to Tier 3. However, in an XML-based web application, we implement both Controller and Model as XQuery modules that are run by a single backend XQuery processor in Tier 3. Following the MVC architectural style, Controller and Model are layers within Tier 3 that communicate through an API. The web server in Tier 2 acts as a generic component that handles network communication; that is, HTTP requests and responses from and to web clients. The HTTP requests are mapped to XQuery functions in Controller as defined by RestXQ annotations within Controller. Controller also constructs complete HTTP response data that are then just passed back as HTTP responses to the correct web client by the web server. Thus, Tier 2 runs no application-specific software. Both Controller and Model reside in Tier 3, which has to be split into two layers for the two components.

Currently, we use the XML database system BaseX [G17] as XQuery processor in our implementations. Any other RestXQ aware XML database system such as eXist or MarkLogic would work as well. We have chosen BaseX over eXist for its support of the W3C standard XQuery Update Facility, and we prefer BaseX over MarkLogic for no other reason than that it is free.

Figure 1: MVC mapped to 3-tier web application architecure.

Domain modeling and implementation strategy for Model

The Model component is a traditional software component that can be designed and implemented independently from its eventual deployment on the web. From a document engineering perspective, the Model component of a web application is typically seen as a repository of documents that is accessed through query and update functions. However, some types of web applications have state in the Model component beyond the state of a typical backend data layer. Specifically, they require several instances of sub-components that are dynamically instantiated and destroyed under user control. This has not been the case with our early case study GameX [SKB14], in which all players share a single map to play on, but it is the case with Blackjack, where each table of Blackjack requires its own instance of a Blackjack game to be created in the server component, even in the most simple single-player scenario. It is also the case with Mancala and Guess the Number, where several games can be played simultaneously.

More precisely, coming from an object-oriented modeling [BD09] and domain-driven perspective [E04], an application such as a Model component is typically modeled in the object-oriented style by a number of classes, with a class corresponding to a type of domain entity and defining which data an instance of the class (that is, an object), holds and which methods can be performed on these data. This approach is also appropriate for Model components that have state as described above.

When implementing the object-oriented model with an object-oriented programming language, the data of the component are often naturally organized into a number of interrelated objects, with methods that locally operate on the data of an object, using methods on other objects as services. This is the principle of encapsulation, a form of abstraction that together with information hiding and inheritance facilitates software qualities such as ease of maintenance and extensibility.

Our main idea for implementing a Model component with XML technology is to simulate the natural object-oriented implementation of its design in the following manner:

While this may sound like a reasonable strategy from a high enough vantage point, XQuery poses some challenges. XQuery by itself is a functional programming language, which means that its primary units of computation are functions that compute output values from input values without side effects. Functions can be composed and used with recursive schemes, so that XQuery is in fact Turing complete: any function that is computable according to any reasonable definition can also be expressed with XQuery.

XQuery by itself, however, has no means to effect change in some repository of data, so it cannot be used to update the state of objects as we would require. Here, the XQuery Update Facility [R11] comes to the rescue by extending XQuery with insert, delete and replace clauses that manipulate XML structures in data stores.

There are some inconvenient restrictions in the use of update clauses within XQuery expressions: They may only be used in the return clause of an XQuery FLWOR expression and a return clause must not mix an update clause and return of a value, thus severely limiting composability.

How does one, under these conditions, implement, for example, a constructor for a new object that needs

Unfortunately, the natural decomposition of a constructor into steps outlined above does not at all fit the restrictions of XQuery and XQuery Update Facility. We solve this problem by what we call split and delegation of function calls; that is, we replace a call to a functions that needs to update data and return a value with two HTTP requests, one to trigger updates in the object repository and one to trigger retrieval of the return value. The HTTP Client Module of EXPath [G10], that is supported by BaseX and other XML database systems, wraps an XML-encoded HTTP request into an EXPath function call that can be used in XQuery where-ever an XPath expression can be used and that returns the corresponding XML-encoded HTTP response as its value. Hence, HTTP requests are fully compositional and, being defined by an industry standard, are platform independent.

We have described how to represent the state of a Model component with XML documents in an XML database and how to implement the API that a Model component provides with XML and XQuery. Internal calls to XQuery functions can be expressed as HTTP requests that return values as HTTP responses to accomodate restrictions in composability. To elaborate, in the definition of the function, the triggering of a function via an HTTP request can be facilitated with RestXQ annotations. The function needs to return an XML-encoded HTTP response, wrapped into a rest:response element that ensures that the XML-encoded response is turned into an actual HTTP response message that is sent to the client that sent the original request.

To illustrate function call split and delegation, we provide a simple web app that maintains a counter on the server and lets clients advance and query the counter in a single request.

xquery version "3.0"  encoding "UTF-8";

(:~
 : This app demonstrates how to do functional decomposition with
 : XQuery updates, using the split and delegate strategy.
 : Prerequisite: create a database counterStore in BaseX with a ressource,
 : whose top-level element is <counter>0</counter>.
 : Usage: URL .../counterApp/advanceCounter
 : Functionality: Advance the counter in the database and return the
 : updated counter in an XML format.
 : Author: Anne Brüggemann-Klein, TUM, 20170410
 :)

module namespace cntr = "brueggemann/counterApp";

declare namespace http = "http://expath.org/ns/http-client";
declare namespace rest = "http://exquery.org/ns/restxq";

declare variable $cntr:counter := db:open("counterStore")/counter;

declare
%rest:path("/counterApp/advanceCounter")
%rest:GET
function cntr:advanceAndWrapCounter() as element(counterInfo) {
(:~
 : The API of the app.
 :)
  <counterInfo>
    <updatedCounter>{cntr:advanceAndReturnCounter()}</updatedCounter>
  </counterInfo>
};

declare
function cntr:advanceAndReturnCounter() {
(:~
 : This function issues two requests, one to update the counter and one
 : to read the new value, returning the new value in the response
 : the second time.
 : That value is then returned as a return value of the function.
 :)
  (: build the XML representation of the update request :)
  let $requestUpdate :=
    <http:request method="POST"
      username="admin"
      password="admin"
      send-authorization="true"
      href="http://localhost:8984/counterApp/updateCounter">
    <http:body media-type="application/xml">
      <empty/>
    </http:body>
    </http:request>
  (: send the update request triggering a RestXQ annotation
   : and throw away the result
   :)
  let $_responseUpdate := http:send-request($requestUpdate)
  (: build the XML representation of the read request :)
  let $requestRead :=
    <http:request method="GET"
      username="admin"
      password="admin"
      send-authorization="true"
      href="http://localhost:8984/counterApp/readCounter">
    </http:request>
  (: send the read request triggering a RestXQ annotation
   : and save the body of the result
   : note that direct read with cntr:getCounter() leads to database deadlock
   :)
  let $responseRead := (http:send-request($requestRead))[2]
  return $responseRead
};

declare
%rest:path("/counterApp/updateCounter")
%rest:POST
%rest:GET
%updating
function cntr:updateCounter() {
(:~
 : Get the counter value and double-wrap the result
 : into a REST and an HTTP response element.
 :)
  let $oldCounter := cntr:getCounter()
  let $newCounter := xs:string($oldCounter + 1)
  return replace value of node $cntr:counter/text() with $newCounter
};

declare
%rest:path("/counterApp/readCounter")
%rest:POST
%rest:GET
function cntr:readCounter() {
  let $counter := xs:string(cntr:getCounter())
  (: construct a RestXQ response, consisting of an XML representation
   : of an HTTP response and a body
   :)
  let $response := 
    ( <rest:response>
        <http:response status="200" message="">
          <http:header name="Content-Language" value="en"/>
          <http:header name="Content-Type" value="text/plain; charset=utf-8"/>
        </http:response>
      </rest:response>,
      $counter
    )
  return $response
};

declare
%private
function cntr:getCounter() as xs:integer {
  xs:integer($cntr:counter/text())
};

Note that we can restructure the code for cntr:advanceAndReturnCounter so that only a single HTTP request, namely one to update, is needed, into steps that

We have chosen to demonstrate a direct translation of the original functional decomposition into code. In practice, algorithms will be stratified to minimize the number of HTTP requests. A typical pattern is to call an updating function and then to perform a redirection to a query function to get the updated value.

When deciding which function calls to delegate to HTTP requests and which functions to call directly, one has to be aware of database locking mechanims. BaseX treats each XQuery method call as a transaction and aquires a database lock for the complete duration of the execution if it detects or suspects database access anywhere in the execution [E13].

The situation is demonstrated with the counter app. If we had called the method getCounter directly, without wrapping it into an HTTP request, BaseX would have locked the database for reading right at the beginning of execution of the calling method advanceCounter, leading to a deadlock when a request to updateCounter happens, because the updating method can aquire an update lock on the database only when all read locks have been released (two-phase commit protocol, [E13]).

One wishes that XQuery and the XQuery Update Facility would relax the constraints on updating functions that necessitate such awkward and costly procedures as described above. BaseX offers proprietary solution in its configuration parameter MIXUPDATES.

The Model component as an event-driven system

The Model component is basically a set of services that are called by Controller. We can view it as an event-driven system whose top-level activities are triggered by events that happen to be represented by function calls, most commonly originating from user actions that are passed on by Controller. Typically, there are constraints to the legal sequences of events, and the specific activity that is triggered may be dependent only on a specific pattern in the history of previous events. The classical tool to model such abstract “behaviour” of an event-driven system is statecharts. Statecharts have been first introduced by Harel as documented in a book [HP98] he co-authored with Politi. They have later, in the object-oriented variant of state diagrams, become part of UML2; see [SSHK15] for a textbook introduction and [H99] for an extensive discussion of the use of statecharts in software engineering. Statecharts can help to cut the complexity of a model. When modeling a Model component such as a lounge where players for new games are matched, a single main statechart can be used to control the Model itself. In addition, dynamically instantiated statecharts can be used to control the behaviour of a complex object such as a single game during its lifetime.

Most recently, with SCXML [B15], an XML encoding language for statecharts has been standardized, bringing statecharts into the realm of XML technologies. A number of research papers discuss use of SCXML in particular, among them the Bachelor Thesis of Roxendal [R10], invited expert to the W3C committee that defined SCXML.

There are even projects that implement SCXML processors with XQuery, thus bringing not only the description but also the execution of statecharts into the realm of XML technology [TODO SCXML processor for MarkLogic, S15]. Consequently, a statechart that defines the behaviour of a Model component can be encoded in SCXML and can then be run seamlessly by the server-side XQuery processor, for example by BaseX.

The SCXML processor SCXML-XQ [S15] is an XQuery module that provides a method

selectTransitions($configuration as element()*,
  $dataModels as element()*, $event as xs:string) as element()*

that computes the transitions that are triggered by event $event when a statechart is in some state configuration $configuration in the presence of some datamodels $datamodels. A complementary function

computeEntrySet($transitions as element()*) as element()*

takes a configuration and a number of transitions as input and computes the next configuration that is reached when the transitions are executed.

N.B.: When pondering the signatures of these functions, note that a state XML structure is hosted by its statechart structure and that a transition XML structure is hosted by its source state structure.

To demonstrate how SCXML-XQ can be used, we have wrapped it with a rudimentary SCXML interpreter with a REST interface that locates a statechart and a state configuration in a database and replaces the state configuration by the next configuration that the statechart will be in after processing some event.

xquery version "3.0"  encoding "UTF-8";

module namespace scI = "brueggemann/scxmlXqInterface";
import module namespace sc='http://www.w3.org/2005/07/scxml';

declare namespace http = "http://expath.org/ns/http-client";
declare namespace rest = "http://exquery.org/ns/restxq";
declare variable $scI:scxmlRuns := db:open("scxmlRuns")/scxmlRuns;

(:~
 : Get a statechart and a run (configuration) from the database
 : "scxmlRuns" and record
 : the state that is reached for a specific event in the run.
 : Assumptions: No parallel states in the statechart,
 : hence a run consists of
 : a single state, statechart and run are a legitimate
 : combination and the event has a transition in the statechart.
 : No error handling.
 :
 : Precondition: The database scxmlRuns has a ressource with
 : top-level element scxmlRuns,
 : which has encodings of statecharts and state configurations.
 :
 : Usage: URL
 :   .../scxmlXqInterface/nextState/statechart/
 :        {$statechart}/run/{$run}/event/{$event}
 : Functionality: Update $run for $statechart after processing $event.
 : Author: Anne Brüggemann-Klein, TUM, 20170410,
 : following ideas of Andreas Eichner.
 :)
 
declare
%rest:path("/scxmlXqInterface/nextState/statechart/
  {$statechart}/run/{$run}/event/{$event}")
%rest:GET
%updating
function scI:nextState($statechart as xs:string,
  $run as xs:string, $event as xs:string) {
  (: get current state $_state from $statechart and $run :)
  let $_statechart := $scI:scxmlRuns//sc:scxml[@name=$statechart]
  let $_run := $scI:scxmlRuns//scxmlRun[@name=$statechart and @id=$run]
  let $_stateID := $_run/state/text()
  let $_state := $_statechart//sc:state[@id=$_stateID]
  (: get transition $_transition to be executed from $_state :)
  (: result is empty if there is no transition :)
  (: datamodels (second argument for sc:selectTransitions() is empty :)
  let $_transition := sc:selectTransitions($_state, (), $event)
  (: get ID $_nextStateID of nextState from $transition :)
  (: TODO: set $_nextStateID to $_stateID if there is no transition
   : for $event (SCXML requires that there is no state change in that case!)
   :)
  (: assume that sc:computeEntrySet returns a single state :)
  let $_nextStateID := xs:string(sc:computeEntrySet($_transition)/@id)
  return replace value of node $_run/state/text() with $_nextStateID
};

<?xml version="1.0" encoding="UTF-8"?>
<!-- This database collects a number of statecharts,
     identified by their name,
     and runs of statecharts that are represented by their current state.
	 In this version, statecharts have no parallel states,
	 so the run of a statechart
	 is adequately represented by a single state.	 
	 
     Author: Anne Brüggemann-Klein, TUM, 20170410
	 -->
<scxmlRuns>
  <!-- A statechart that accepts the event sequences of language a* b.
       Events that do not match with a transition do not change state. -->	 
  <sc:scxml xmlns:sc="http://www.w3.org/2005/07/scxml"
      name="aStarB" version="1.0" initial="init">
    <sc:state id="init">
      <sc:transition event="a" target="init"/>
      <sc:transition event="b" target="final"/>
    </sc:state>
    <sc:state id="final"/>
  </sc:scxml>
  
  <!-- A run of a statechart identified by name is represented
       by its current state. -->
  <scxmlRun name="aStarB" id="runA">
    <state>init</state>
  </scxmlRun>
  <scxmlRun name="aStarB" id="runB">
    <state>init</state>
  </scxmlRun>
</scxmlRuns>

From this example, it is easily conceivable how a system written in XQuery can use SCXML-XQ to create statecharts dynamically and to carry out independent executions of a state chart by keeping track of dynamic information such as state configurations and possibly datamodels in the database, in combination with object information.

Unfortunately, SCXML-XQ has no documentation. It appears to be able to handle composite states including parallel states and possibly conditional transitions. There is no evidence that it handles final states, assignments to data in a datamodel or method invocations when following a transition or when entering and exiting a state. These features are essential when using statecharts to control even such a simple application as Guess the Number.

One co-author of this paper, Andreas Eichner, has implemented some of the features that are missing from SCXML-XQ [S15] as part of his master thesis [E17]. This implementation has all the features in place that are necessary to control a game of Blackjack.

Outlook: multi-client web applications

In our current architecture, Model is passive and not generally observable; that is, Model changes state only on client request, and the updated state is only of interest to the client that instigated the change. The MVC architectural style in its most general form has an active and “observable” Model component.

An active Model can change its state in response to an outside event such as sensor input or to a request from one of many clients; there might even occur spontaneous change of state within Model. In that case, clients that have not explicitly requested the state change, will need to be informed about the updated state.

Therefore, in the MVC architectural style, Model makes itself “observable”; that is, it allows interested parties to register themselves for update notifications that will be broadcast to all of them when a relevant state change occurs in Model without clients having to explicitly request it on a case-by-case basis. Hence, MVC in its general form comprises the Observer pattern [BD09], which structures a system in such a way that the Observable component (the Model) and the Observer components (the Views or Controller(s) on behalf of the Views, depending on the specific variant of MVC in use) are properly decoupled.

In the web context, this boils down to the requirement that a client and a server are enabled to establish a bi-directional communication channel over the Internet, through which the server can send messages to the client without being prompted by the client (server push). A server who keeps track of all the channels that are currently established with clients is then able to broadcast update notifications by sending some message to each of the clients.

How is this relevant to the class of XML-based web applications that we have considered? The current architecture and implementation strategies cover multi-player games such as Mancala or Blackjack only if all players play through a single browser. The situation changes completely if we want to support a truly distributed game in which, for example, each player at a Blackjack table plays through their own browser but needs to see actions of the other players at the table. Server push capability is needed when the results of one player's action is to be made visible to all players in their own browsers.

How can server push be implemented in the Internet? Communication between components over HTTP is restricted by the request-response rhythm of the protocol. A server component may only speak to a client after being requested to by the client, and then it is only allowed a single response message.

There are ways to emulate the server-push or server-broadcast communication pattern over HTTP by using so-called busy polling or long polling techniques in combination with AJAX-like requests. In fact, our browser game GameX uses busy polling to propagate changes in the map that were caused by one player to all other players. However, these solutions are not stable, and they do not scale. HTML5-based architectures go beyond the request-response cycle of the HTTP protocol towards server push solutions by supporting APIs such as Server-Sent Events and WebSockets [WSM13] in the client. These APIs are matched by server-side implementations, most commonly in Node.js [HW12]. The first message that is sent in the WebSockets protocol is an HTTP request with an upgrade flag that imdicates that the WebSockets protocol is to be used with further messages; these are then in a specific binary WebSockets protocol format.

One co-author of this paper, Michael Conrads, has brought server push into the realm of XML technologies in his master thesis [C17]. As an outlook, we sketch the two approaches that he has investigated; they will be further described in a later paper.

The first approach uses a Node.js server as routing middleware. When a client loads a Web page, a Javascript component establishes a WebSocket connection to the Node.js server. An HTTP request to this server is forwarded to the XML server and the reponse is returned to the client. When the client issues a ressource-modifying request (for example using HTTP methods POST or PUT), the response from the server is returned not only to the requesting client but also to all connected clients via their WebSocket connections. Messages between server and client that go through the WebSocket connection can be encoded in XML. Clients can apply XSLT transformation using the Javascript XSLTProcessor object to massage the XML data into custom formats, most probably into HTML.

A second and more elegant approach ditches the Node.js server and extends BaseX so that it can understand and process a new type of RestXQ annotation, %rest.WEBSOCKET. This annotation is similar to an annotation with an HTTP method such as GET or POST. It signals which format the messages that are exchanged need to have. The annotation still needs to be extended by a naming or grouping mechanism that determines the target clients for broadcasts. The new RestXQ annotation as a server-side interface is in perfect alignment with the REST philosophy [F00].

Both approaches rely on client-side Javascript to send messages over a WebSocket and to handle the incoming messages. Message formats can be in some encoding of XML data. The client-side communication components can be generic, so that they only need to be configured but not programmed for a specific application. The Node.js router in the first approach just forwards all requests; it does not require any custom programming either but only configuration of host and port data.

Conclusion and future work

In this paper, we have laid out a coherent and coordinated set of practices for developing XML-powered web applications. The practices draw on previous work and have been and are being vetted with case studies.

One source of inspiration have been proven principles and practices from software engineering.

All solutions are based on W3C or industry standards and use freely available software components.

One motivation for this line of work has been to support XML experts as end-user programmers, which in sequence has led to the idea of domain modelling, object-oriented design, object-oriented implementation and eventually XQuery tweaks.

It is an open question if it is possible to use XQuery in the spirit it was invented, as a functional programming language, for implementing XML-based web applications. There are very few and not very refined software engineering methodologies for functional programming which could lead to a more natural way of using XQuery.

In teaching a lab course on XML technology, we have made the experience that no-frills web apps, reduced to essentials, which do not require any frameworks, are a useful and appreciated pedagogical approach for teaching computer science students.

Areas for future work are: