Balisage Paper: XDML - an extensible markup language and processor for XDM

Partitioning an XDM value

Consider the situation that an XDM value should convey two code lists, each one represented by a sequence of string items. XDM offers no way to tell where one list ends and the other begins. Similar example: the XDM value is a sequence of XML documents which represent the log data gathered during one hour - how to identify the subsequence corresponding to one day of operation? A quick and simple solution is to insert into the XDM value additional items which delimit subsequences. These items can be regarded as control items, to be distinguished from the original data items. The subsequences are parts of the XDM value which have been turned into new units of information. In order to give names to these parts, we add a "name" attribute to the respective control item. Example:

<xm:part xmlns:xm=”http://www.xdml.org/ns” name=”alpha-codes”/>,
a001,
a005,
a012,

<xm:part xmlns:xm=”http://www.xdml.org/ns” name=”beta-codes”/>,
b002,
b003

And if the uniqueness of part names is not guaranteed, an optional "partID" attribute may accompany the mandatory "name" attribute.

Imposing tree structure

The shown use of control items defines parts of an XDM value in an intuitive way: the contents of a part is simply all items following the part definition and preceding the next part definition, or all following items, if this was the last part definition. But we might also allow "complex parts" - parts containing parts, to be distinguished from simple parts which contain only data items. To encode this structural model, we choose a simple rule: the contents of a complex part ends before an item explicitly "closing" the part, whereas the contents of a simple part is delimited implicitly: it ends before the next control item defining a new part (simple or complex) or closing the surrounding complex part. Note that these parts - simple or complex - are defined in a "streaming" fashion - contents are not children, but a subsequence of items delimited by an item recognized as start point and another item explicitly or implicitly meaning an end point (or the end of the XDM value, as a special case).

In order to keep things simple, we constrain the definition of complex parts: they must not contain data items outside of contained parts. In other words: parts must not be mixed, their content is either a sequence of data items, or a sequence of parts which may be simple or complex. An example using complex parts:

<xm:complexPart name="code-lists"/>,
<xm:part name="alpha-codes"/>,
a001,
a005,
a012,
<xm:part name="beta-codes"/>,
b002,
b003,
<xm:complexPartEnd/>,

<xm:complexPart name="logs"/>,
<xm:part name="log0800" />,
<log>...</log>,
<xm:part name="log0900" />,
<log>...</log>,
<xm:complexPartEnd/> 

Concept: Information units

We have seen how the insertion of control items can partition an XDM value into parts. To denote the concept of such parts we introduce the term information unit. An information unit is encoded by a sequence of XDM items. According to whether whose items represent nested units, two kinds of information units are distinguished. A simple information unit contains only data items, but not any nested information units. A complex information unit, on the other hand, contains other information units, but no data items outside of nested units.

An information unit has the following properties:

Note that this definition renders the sequence of nested information units irrelevant, as these units are associated with names. This corresponds to the modeling practice of XML attributes or JSON members.

Why and how add metadata?

We saw that control items may structure XDM values into information units, which are groups of items or of other information units. These units are entities which do not exist in XDM values without control items. Often they will serve as units of processing, and it is reasonable to expect that different units may be subjected to different processing. Such considerations suggest the usefulness of metadata.

In fact, it is very simple to associate the units with as many metadata as one would like. Every unit is preceded by a control item which amounts to a convenient container where to place those metadata, either as attributes or as child elements. Come to think of it, the control item can be regarded as a full-scale XML document which is still hardly constrained in its contents: only the name of the root element and the use of a "name" and a "partID" attribute are specified, so far. This document is dedicated to defining a unit, and it is ready to be filled with metadata describing the unit.

Returning to the example given above, the units containing a single document of log data may be associated with metadata "startTime" and "devices". To accommodate such data, we can use attributes and child elements of the markup item, like so:

<xm:complexPart name="logs"/>,

<xm:part name="log0800" xmlns:e=”e.com” e:startTime="2010-12-30T08:00:00" xmlns:e="http://example.com">
   <e:devices>…</e:devices>
</xm:part>,
<log>...</log>, 

<xm:part name="log0900" xmlns:e=”e.com” e:startTime="2010-12-30T09:00:00" xmlns:e="http://example.com"> 
   <e:devices>…</e:devices>
</xm:part>,
<log>...</log>,

<xm:complexPartEnd/>

A model of metadata

We have arrived at a very simple method how to impose structure on XDM values, and we have found a slot into which one might throw any amount of metadata pertaining to the emerging units. Now we face two alternatives. We might stop here and regard the semantics of metadata as the realm of proprietary extensions of our simple, general model, in the same way as XML Schema allows annotation attributes. We might, for example, say that any additional attributes and child elements of control items are meta information, to be evaluated in a proprietary way.

But we can also take a different path and attempt to arrive at a generic model of XDM metadata and its processing by a responsive infrastructure. This approach does not remove the option of proprietary extensions, but factors them out and constrains them in a way which allows a generic "XDM parser" to report them in a structured way. The basic principle of such a model is to distinguish metadata meant to control a specific processing from other metadata. The latter might be called "descriptive metadata" and is available for variable uses. The former – "control metadata" – has a defined impact on a defined processing.

Why should one associate data with information which controls their processing? We note an interesting analogy. A key concept of object orientation is to associate data sets with behavior. This is similar to what we try to do. The behavior of objects is implemented by methods; the "behavior" of information units resides in control metadata which define a processing. Control metadata is behavior encoded as data, as opposed to methods which are behavior encoded as code. To get a more practical motivation, imagine writing an XQuery program and regretting the limitations of XQuery. For example, one cannot call XSLT to accomplish some finalization, one cannot trigger actions with side effects (like the execution of the SQL just composed), and one cannot create a map object which the calling application would really like to receive. In this situation there is a way out: let the query code rely on a postprocessing of the query result which is defined by the query and executed by infrastructure. Our model of XDM metadata amounts to a framework for this approach.

Obviously, control metadata and the responsive infrastructure must be modeled as a coherent whole. We assume that control metadata can be further grouped into a set of metadata components, and that a general processing model yet to be defined determines how actual processing depends on those components. But at this point of our argument we want to separate the general idea from our elaboration of it, as we want to protect the value of the idea from the possible weaknesses of our attempts to refine it. For the time being, we remain abstract. We assume a standard infrastructure governed by a set of standard metadata components.

Goals

XDML is a set of rules how XDM values can be designed in order to become more useful entities as compared to ordinary XDM values. The key idea is to insert into the XDM values control information which guides the interpretation and processing of the data. An XDM value thus augmented is called an XDML value. Its usefulness is provided by an XDML processor, which is a generic program evaluating the control information. XDML addresses the following major goals:

Structure model

XDML structures XDM values by grouping the XDM items. The resulting groups are units of usage in a broad sense: conceptual units of information, units of data retrieval and units of data processing. Item groups are called information units. The grouping approach distinguishes:

Metadata model

Information units can be associated with metadata. XDML uses a simple metadata model which

Processing model

XDML values are submitted to an XDML processor which evaluates the control information and is responsible for reporting and processing the data accordingly. The processor is viewed as the sum of two components:

Encoding principles

XDML defines the syntax and semantics of control information embedded in XDM values. We propose four general encoding principles:

A concrete specification of XDML must elaborate these principles into a concrete encoding model. This model must define the names and structure of control items, and it must define the mapping of control items onto content items ("where does the unit begin and end?").

XDML operations

Data processing provided by the XDML processor is modeled as the execution of discrete operations, collectively called XDML operations. XDML operations thus serve as basic unit of data processing: an operation is either executed as a whole or not at all; and any processing can be decomposed into the execution of one or more operations. An operation is supplied with input information, it may produce output information and it may have side-effects. Output information is the return value of the operation. Input information comprises a data context and a request message.

The data context can be regarded as the main input, comparable to the context item of XQuery, the context node of XSLT or the primary input port of XProc. The data context of an XDML operation is (usually) the value of an information unit (as represented by the implementation language of the XDML processor). Therefore one might say that an XDML operation is applied to an information unit, or that an information unit is processed by an XDML operation.

The request message consists of named parameters, comparable to the external variables of XQuery and the global parameters of XSLT. In the case of XProc, the corresponding input sources would be non-primary input ports, options and parameters.

The return value of an operation may be an instance of any type supported by the implementation language of the XDML processor. Note that this value may or may not have a default mapping to an XDM value. In other words: operations may produce a result which is not related to the XDM model, e.g. an object of a custom class.

The XDML provider defines the processing of an information unit by associating it with methods. A method is a processing defined as the sequential execution of one or several operations. It is therefore encoded as one or more request messages and the choice of a so-called execution context. The context determines when to invoke the method and what to do with the return value. Method definition is described in section “Method definitions”. The current section describes XDML operations in general terms, independently of their use in a particular execution context. Main aspects are the data model of input and output, the encoding of input by request messages, the standard library of XDML operations and the extensibility by user-defined operations.

interface OperationRequest {
   QName   operationName();
   String  resultType();

   String  getStringParam(QName name);
   Node    getNodeParam(QName name);
   Node[]  getNodesParam(QName name);

   String  getDynamicStringParam(QName name);
   Node    getDynamicNodeParam(QName name);
   Node[]  getDynamicNodesParam(QName name);

   QName[] getParamNames();
   QName[] getDynamicParamNames();
}

<submitToXSLT serialize=”true”>
   <stylesheet>
      <xsl:transform …>…</xsl:transform>
   </stylesheet>
   <xm:params verbosity=”1”>
      <weatherData><weather>…</weather></weatherData>
   </xm:params>
</submitToXSLT>

<submitToXSLT serialize=”true”>
   <stylesheet>
      <xsl:transform …>…</xsl:transform>
   </stylesheet>
   <xm:params verbosity=”$arg{v}”>
      <weatherData>$arg{weatherData}</weatherData>
   </xm:params>
</submitToXSLT>

<submitToXSLT serialize=”true”>
   <stylesheet>$part{toHTML}</stylesheet>
   …
</submitToXSLT>

interface XDMLRegistry {
   void registerXDMLOperations(XDMLOperations impls);
}

interface XDMLOperations {
   QName[] getOperationNames();
   void    execute(OperationRequest requestMsg,
                   DataUnit         dataContext,
                   DataUnit         returnValue) 
                      throws XDMLException;
}

Methods

In most cases, a desired processing can be provided by a single operation, in other words: the unit of intended processing matches the basic unit of implemented functionality. Sometimes, however, a processing may require two or more operations to be executed. As a generalization, our processing model defines the unit of intended processing as a sequence of one or more operations. This unit we call a method. Assuming sequential execution of the operations, one may wish for flexibility concerning the data context: shall the second operation use, like the first one, the value of the information unit, or shall it use the return value of the preceding operation? This flexibility is easy to implement, and it is easy to encode:

The return value of a method is the return value of its last (or only) operation, unless another operation has been marked with a special attribute ("methodReturnValue") to yield the return value.

Execution context

When defining a method, the control data provide

We distinguish four types of execution context, which, taken together, define the processing model of XDML. Future versions of XDML may add further execution contexts. Each context may be viewed as the intent with which the XDML provider defines the method. He may want to

XDMLProcessor xp = XDMLProcessorFactory.newXDMLProcessor();
XQSequence xdm = ...;
XDML xdml = xp.newXDML(xdm);
xdml.finalize();

…
xdml.finalize(“conferenceProgram”);

…
xdml.execute();

…
xdml.execute(“cleanupScript”);

String[] locations = (String[]) xdml.invoke(“waterReport”, “getLocations”);

<xm:part name="waterReport" type="node">
   <xm:interface>
      <xm:method name="getLocations" returnType="strings">
         <submitToXQuery>
            <query>distinct-values(//location/@name)</query>
         </submitToXQuery>
      </xm:method>
      <xm:method name="getResultTable" returnType="map_string_to_string">
         <submitToXQuery>
            <query>
               declare variable $location external; 
               //location[@name eq $location]//substance/(@name, @quantity)
            </query>
            <xm:params location="$arg{location}"/>
         </submitToXQuery>         
         <createMapFromStrings/>
      </xm:method>
    </xm:interface>
</xm:part>

   informationUnitInterface {
      String[] getLocations();
      Map<String,String> getResultTable(String location)
   }

Map<String,String> map = xdml.getPart("foo").getMapString2String()

Method definitions

The processing of an information unit is organized as the execution of methods. A method consists of one or several operations. The definition of a method consists of the request message(s) launching its operation(s). The definitions are associated with an execution context, where execution contexts and method definitions are related as follows:

The encoding of method definitions reflects these relationships:

<xm:part name="foo" type="bar">
   <xm:finalize>
      <op1>...</op1>
      <op2>...</op2>
   </xm:finalize>
   <xm:execute>
      <op3>...</op3>
      <op4>...</op4>
   </xm:execute>
   <xm:interface>
      <xm:method name="m1" returnType="t1">
         <op5>...</op5>
         <op6>...</op6>
      </xm:method>
      <xm:method name="m2" returnType="t2">
         <op7>...</op7>
         <op8>...</op8>
      </xm:method>
   </xm:interface>
   <xm:translate target="java" type="t3" att1="..." att2="..."/>
</xm:part>

And here comes a realistic example using three execution contexts, “finalize”, "execute" and "enable". It shows an information unit which is finalized into a Perl script to be executed in context "execute" and besides offering a little interface of methods to be invoked explicitly ("writeLog", "save"):

<xm:part name="cleanupScript" type="node" finalType="string">
   <xm:finalize>
      <execAsXSLT serialize="true"/>
   </xm:finalize>
   <xm:execute>
      <execAsPerl>
         <xm:params options="-m cleanup"/>        
      </execAsPerl>
   </xm:execute>
   <xm:interface>
      <xm:method name="writeLog">
         <execAsPerl>
            <xm:params options="-m writeLog -f $arg{fileName}"/>
         </execAsPerl>
      </xm:method>
      <xm:method name="save">
         <execAsPerl>
            <xm:params options="-m save"/>
         </execAsPerl>
      </xm:method>
   </xm:interface>
</xm:part>

XDML user perspective

An XDML value is a set of information units which may be retrieved and – depending on the method definitions – processed in a simplified way. An XDML value is represented by an object whose interfaces provide for retrieval (interface XMDLParser) and processing (XDMLProcessing). The following sections give a brief overview of these and further interfaces which taken together amount to the user perspective of XDML.

XDMLProcessor xp = XDMLProcessorFactory.newXDMLProcessor();
xp.registerXDMLOperations(new WaterOperations());
xp.registerXDMLOperations(new WeatherOperations());

void newXDML(Object dataSource) throws XDMLException;

XQSequence xdm = …;            // procure XDM value
XDML xdml = xp.newXDML(xdm);   // create XDML value

interface XDMLParser {
   InformationUnit next();
   boolean hasNext();
   void rewind();

   InformationUnit getPart(QName partName);
   InformationUnit getPart(QName[] partNames);  // access nested part
   InformationUnit getPartByID(String partID);

   …   
}

class InformationUnit implements DataUnit, MetadataUnit {
   XDML getComplexValue();
   boolean isValueComplex();
   ...

interface DataUnit {
   // *** read value
   Node         getNode();
   Node[]       getNodes();
   int          getInteger();
   int[]        getIntegers();
   String       getString();
   String[]     getStrings();
   Duration     getDuration();
   Duration[]   getDurations();   
   …
   Object       getObject();    // allows for a DataUnit to contain ANY type

   // *** write value
   void         setNode(Node value);
   void         setNodes(Node[] value);
   ...
   void         setObject(Object value, String typeName);
}

interface MetadataUnit {
   MetadataSet getDescriptiveMetadata(String topic);
   MetadataSet getTranslationMetadata(String targetLanguage);
   String[] getDescriptiveTopics();
   String[] getTranslationTargetLanguages();
   ...
}

interface MetadataSet {
   QName[] getPropertyNames();
   String  getStringProperty(QName name);
   Node    getNodeProperty(QName name);
   Node[]  getNodesProperty(QName name);
   …
}

interface MetadataUnit {
   …
   Method getFinalizationMethod();
   Method getExecutionMethod();
   Map<QName, Method> getInterfaceMethods();
}

interface Method {
   int              getOperationCount();
   OperationRequest getOperationRequest(int index);
   Integer          getDataContext(int index);
      // data context is the return value of a preceding operation (>0), or the unit value (0), or null
}

interface XDMLProcessing {
   void finalize();
   void finalize(Arguments args);
   void finalize(QName part);
   void finalize(QName part, Arguments args);

   void execute();
   void execute(Arguments args);
   void execute(QName part);
   void execute(QName part, Arguments args);

   Object invoke(QName part, QName methodName);
   Object invoke(QName part, QName methodName, Arguments args);

   boolean isFinalized();
   boolean isFinalized(QName part);
   boolean isExecuted();
   boolean isExecuted(QName part);
}

Document weatherData = ...;
String location = "NY";

Arguments args = xdml.newArguments();
args.set(new QName("location"), location);
args.set(new QName("weatherData"), weatherData); 

// *** obtain XDML value
XDMLProcessor xp = XDMLProcessorFactory.newXDMLProcessor();
XQSequence xdm = …;   // procure source data (e.g. exec XQuery) 
XDML xdml = xp.newXDML(xdm);

// *** use XDML value
xdml.finalize();
Map<String,String> results = (Map<String,String>) xdml.invoke("report", "getResultTable");
String html = (String) xdml.invoke("report", "getHTML");
String[] cvs = (String[])  xdml.invoke("report", "getCVS");

<xm:part name="toHTML" partID="toHTML" type="node" private="true"/>,
<xsl:transform…>…</xsl:transform>
,

<xm:part name="toCVS" partID="toCVS" type="string" private="true">,
xquery 1.0 …
…
,

<xm:part name="weatherData" partID="we" type="node" finalType="node">
   <xm:finalize>
      <sendSOAP href="…" />
   </xm:finalize>
</xm:part>,
<getWeatherData>…</getWeatherData>
,

<xm:part name="waterData" partID="wa" type="string" finalType="node">
   <xm:finalize>
      <execAsSQL driver="…" host="…" db="…" user="…" password="…" format="xml"/>
   </xm:finalize>
</xm:part>,
SELECT …
,

<xm:part name="report" type="string" finalType="node">
   <xm:finalize requiredParts="we wa">
      <execAsXQuery resultType="node">
         <xm:params>
            <weatherData>$part{we}</weatherData>
            <waterData>$part{wa}</waterData>
         </xm:params>
      </execAsXQuery>
   </xm:finalize>

   <xm:interface>
      <xm:method name="getResultTable" returnType="map_string_to_string">
         <submitToXQuery resultType="strings">
            <query>...</query>
         </submitToXQuery>
         <createMapFromStrings/>
      </xm:method>
      <xm:method name="toHTML" returnType="string">
         <submitToXSLT serialize="true">
            <stylesheet>$part{toHTML}</stylesheet>
         </submitToXSLT>
      </xm:method>
      <xm:method name="toCVS" returnType="strings">
         <submitToXQuery resultType="strings">
            <query>$part{toCVS}</query>
         </submitToXQuery>
      </xm:method>
   </xm:interface>
</xm:part>,
xquery 1.0
declare variable $weatherData as node() external;
declare variable $waterData as node() external;
<waterReport>{
   …
}</waterReport>

Generalization: XDML as an information model

The concept of XDML can be generalized by distinguishing the encoding of XDML values from their information model.

<x:xdm xmlns:x="http://www.xdml.org/ns/xdm">
   <x:item type="document">
      <foo/>
   </x:item>
   <x:item type="element">
      <bar/>
   </x:item>
   <x:item type="attribute" name="a" value="v"/>
   <x:item type="processing-instruction" value="foo a=x b=y"/>
   <x:item type="xs:string">hello</x:item>
   <x:item type="xs:integer">123</x:item>
</x:xdm>