Balisage Paper: The XML info space

External node properties

Every XML node has a set of properties defined by the XDM data model. An element node, for example, has twelve properties, which include the name, relationships to other nodes ([parent], [children], [attributes]), a type, a typed value and a string value. Evaluation of these properties is what drives XPath navigation, guiding it from a set of starting points to a set of destination nodes.

In certain situations, however, the locating of nodes might be greatly simplified if we manage to associate nodes with additional properties which help us to identify the nodes of interest. Such properties could represent metadata, or they could reflect the contained data, for example echoing a value found at a particular data path within the node, or providing a value derived from contained data, like a minimum or maximum. As these additional properties are not XDM properties, their values are not part of the node data. Hence we call them external node properties. Their nature implies an interesting possibility: they can provide information about a node which is accessible without constructing the node. Access to such information might enhance the navigational model of XPath in general, and add support for data-driven navigation in particular. The following sections describe an approach how to achieve this.

An external property has a name and a value. A set of external node properties can be modelled as a map with which the node is associated. The map entries have a key which represents the property name, and a value which represents the property value. Such a map we call a property face of the node, or p-face for short. Striving to keep things simple, we constrain external node properties to have an NCName and a value which is a sequence of atomic items of type xs:untypedAtomic. The value space of p-faces is accordingly constrained.

Property-faced collections

A p-face is not part of the node data – it is merely associated with them. This relationship can be compared to the general relationship between the keys and values of a map: the keys are not part of the values, but only associated with them in the context of the map. Such an association is not global, but scoped to the map.

The association between a node and its p-face is provided by a node collection which contains the node. We introduce the concept of a p-faced collection (short for property-faced collection), which is a conventional node collection plus a mapping associating each node with a p-face. We do not constrain the individual p-faces to use the same property names: the number and names of external properties can vary from node to node.

The following table shows a simple example of a p-faced collection. In this case, all collection members are associated with the same set of external properties. The collection may therefore be represented by a table in which one column identifies the node and the remaining columns represent external properties.

A p-faced collection can be filtered by supplying a set of conditions to be tested against the p-faces of its nodes. Such a set of conditions we call a p-filter. The result of applying a p-filter to a p-faced collection is a sub collection of the original collection, which we call a p-filtered collection.

A subsequent section will define the syntax and semantics of property filters in detail. For the moment, assume that such a filter may be represented by a string, and further assume that a p-faced collection is identified by a URI. Concatenating the URI and the query string, we can then identify sub collections by a single string, for example:

   http://example.com/pcollections/countries?neighbourCountries=be

   fn:nodes($uriQuery as xs:string) as node()*

   declare variable $country external;
   declare variable $countries := "http://example.com/pcollections/countries";

   <neighbours cy="{$country}">{
      fn:nodes(concat($countries, '?', $country))
   }</neighbours>

In summary, p-faced collections promise two distinct benefits. First, they augment the navigational model of XPath with data-based access to information, overcoming the current limitation that document access requires knowledge of the document URI. Second, they offer the possibility to integrate non-XML technologies into the implementation of XML data storage and XPath navigation, which may enhance the scalability of XML based systems significantly. The concept of p-faced collections must now provide

Filter model

A p-filter defines a condition which can be applied to a p-face and yields a Boolean result. As every node in a p-faced collection is associated with a p-face, a p-filter can be used to specify sub collections. This section proposes a generic model of a p-filter, which includes the data model, syntax and semantics.

   <time>2013-01-01T00:00:00</time>
   <time op="&lt;">2013-01-01T00:00:00</time>
   <time op="&lt;" qua="every">2013-01-01T00:00:00</time>
   <keywords op="~">^semantic.*</keywords>
   <keywords op="~">^semantic.*#i</keywords>
   <keywords op="~">semantic%</keywords>
   <codes>
      <p:item>DUS</p:item>
      <p:item>CGN</p:item>
   </codes

   <p:and>
      <time op="&gt;">2012-12-31T22:30:00</time>
      <time op="&lt;">2013-01-01T00:00:00</time>
      <p:or>
         <travelType op="~">^accommodation.*</travelType>
         <supplier>TUI</supplier>
      </p:or>	
      <p:not>
         <eventDate op="&lt;" qua="every">2013-08-01</eventDate>
      </p:not>
   </p:and>

   property-name operator testvalue
   property-name operator (testvalueItem-1, testvalueItem-2, ...)

   condition and condition
   condition or condition                        
   not condition
   not (condition1, condition2, ...)

   time >= 2012… 
   and time <=2013…
   and (travelType ~ ^accommodation.* or supplier=TUI) 
   and not(eventData $> 2013-01-01)

New XPath functions

Given a p-faced collection, p-filters can be used to select a sub collection, retaining only those nodes whose p-face passes the p-filter. This amounts to a logical operation, mapping a collection URI and a p-filter to a sub collection. New XPath functions can make this operation available. Assuming as a tentative function name fn:nodes, three variants may be useful:

   fn:nodes($collectionUri as xs:string, p:filter as element(p:filter)) as node()*
   fn:nodes($collectionUri as xs:string, p:filter as xs:string) as node()*
   fn:nodes($filteredCollectionUri as xs:string) as node()*

   http://www.traveltainment.de/pfc/shoppingCart?booking-date=2013-08-01

The introduction of these functions should be accompanied by an extension of the dynamic context of XPath evaluation: an additional component available property-faced collections provides a mapping of URIs to property-faced collections. A property-faced collection is formally defined as a collection which is associated with a mapping of p-filter instances to sub collections

Exposure by REST-ful web services

A p-faced collection can be exposed as a REST-full web service, using as URI the concatenation of collection URI and (URI-encoded) p-filter described in the previous section. As a web service is expected to return a single document, rather than a sequence of nodes, the selected sub collection may be wrapped by an element and thus integrated into a single document. The proposed convention is to use a <p:collection> element with two attributes providing the collection URI and the (not URI-encoded) p-filter. Example:

   <p:collection uri="http://www.traveltainment.de/pfc/offers" p-filter="destination=PMI"
         xmlns="http://www.traveltainment.de/pfc/offers"
         xmlns:p="http://www.w3.org/2013/xpath-structures"> 
      <offer>...</offer>
      <offer>...</offer>
      <offer>...</offer>
   </p:collection>

Deployed and external collections

An XPath processor may provide access to two different groups of p-faced collections:

A deployed collection is accessed according to configuration data. This might, for example, involve the translation of the p-filter into a SQL query, its execution and the construction of nodes from result set data. See Implementation and configuration for details about the deployment of p-faced collection.

An external collection is accessed by applying the fn:doc function to a URI which is a concatenation of collection URI and (URI-encoded) p-filter in query syntax. The URI is expected to yield a document which represents the selected nodes as child elements of the document element (see Exposure as REST-ful web services)). The fn:nodes function might operate as follows:

Implementation and configuration

A crucial aspect of p-faced collections is that they may be implemented using non-XML technologies, namely relational database tables and NOSQL object collections. This follows from the fact that the node itself is represented by a string and that it is not inspected during the filtering operation. Consequently, the implementation of a p-faced collection is not dealing with any tree structure. The implementation must provide for each node the representation of a logical map in which one entry represents the node itself and the remaining entries represent the names and values of properties. Configuration data include the name of the entry representing the node and specify how the node (if selected) is constructed from the string representing it. The string may be a serialization of the node, or it may consist of locator information, either a regular URI or a locator with implementation-defined semantics.

If the p-faced collection is “homogeneous” in the sense of every node having the same external properties, a relational table is a straightforward choice. If the nodes have different sets of properties, a NOSQL collection may be more appropriate. An example for the latter case are log data, where each log entry has some generic properties including an event type and further properties which are event type-specific. An event of type “file-not-found” may have a property “file-name”, whereas an event “container-delivered” may have properties specifying a container-number and a location.

Configuration data are required in order to enable an XPath processor to access the implementation of a p-faced collection:

Collection features

The model of a p-filter includes details which may not be well supported by a particular technology which on the whole seems well-suited to implement p-faced collections. For example, the used technology may not support …

The concept of collection features is introduced in order to enable a standardized representation of limitations, and to enable a definition of conformance which can be achieved in spite of limited support of the p-filter model. The following table presents a tentative set of collection features.

Introductory example

Scenario: we want to check some country codes, we have access to a dictionary of country codes and names, but this dictionary is represented by a CSV dataset.

Now let us assume there is a standardized XML vocabulary for table data, comprising "table", "row" and "cell" elements. And we further assume the existence of a new fn:doc variant which delivers CSV data as an XML node tree based on the table vocabulary. It enables us to select all invalid country codes by a simple expression:

   doc("foo.xml")//countryCode
      [not(. = doc("countries.csv", "csv")//row/cell[1])]

This example demonstrates a seamless integration of data from XML and non-XML resources: both participate in the info space.

A generic pattern of integration

Let us assume that for a given data format F three sets of rules have been defined:

C1 and C2 imply a constraint concerning round-tripping which starts with a format instance G: the result is a format instance G' which is not necessarily equal G but which is "FORMAT-deep-equal": R1(G) is deep-equal R1(G'). This amounts to defining the information content of a format instance G: it is the node tree provided by R1. Relying on this definition of information content, instances of F can be safely evaluated or modified by evaluating or modifying the corresponding node tree. This is the foundation of an integration of format F into the info space.

A normative definition of rule sets R1, R2 and R3 enables the standardized integration of a non-XML format into the info space. This requires two extensions of the XML standards. The first extension is an XPath function which parses an instance of the format into a node tree, reflecting R1. The second change extends the XML serialization model [5] by the definition of a serialization to format F, reflecting R2 and R3. The interface of serialization is accordingly extended by an additional value of the serialization parameter model.

A concrete proposal for the required XPath function is the introduction of a second variant of fn:doc which has an additional parameter specifying the source data format:

   fn:doc($uri as xs:string, $mediaType as xs:string) as document-node()

   fn:doc("http://example.com/foo.htm", "htm")
   fn:doc("http://example.com/foo.jsn", "jsn")                        
   fn:doc("http://example.com/foo.csv", "csv")
   fn:doc("http://example.com/foo.sql", "sql")
   fn:doc("http://example.com/foo.css", "css")

Vocabulary based integration

Domain-specific languages (like SQL) and non-recursive generic formats (like CSV) may be represented by a particular XML vocabulary. The formal definition of such a vocabulary might comprise an XSD and a (possibly empty) set of additional constraints expressible as XPath assertions. As a simple example, the vocabulary for representing CSV data might be defined as a schema containing element declarations with names like "table", "row" and "cell" as well as type definitions constraining a table element to have only row children and row elements to have only cell children. An additional constraint might prescribe that every row element has the same number of cell children:

   count(distinct-values(//row/count(cell))) eq 1

Wiki dialects are non-XML formats representing a node vocabulary (XHTML). The motivation is of course not integration into the info space, but access to a rendering processor. And here a format was added to an existing node vocabulary, whereas integration adds a node vocabulary to an existing format. Nevertheless, it is interesting to notice the popularity which a non-XML/HTML representation of node trees can attain. Standardization work might take this phenomenon into consideration.

Vocabulary-assisted integration

Vocabulary-based integration is natural for formats which do not themselves generate vocabularies as JSON does. Such a generative format might in principle be mapped to a fixed XML vocabulary, by mapping the names supplied by the format instance to XML data, rather than to XML names. However, if the format has a recursive structure – like JSON – the resulting XML would look unnatural and be inconvenient to evaluate. Such a mapping would have very limited practical value, and it would violate the concept of integration as explained in a generic pattern of integration. The concept is based on regarding non-XML formats as alternative representations of node trees, as if the node trees were the original information and the format were a means to capture this information. If the node tree looks unnatural, the image presupposes the desire to convey information which is unnaturally structured in the first place, which is absurd.

If JSON is redefined to be the representation of node trees, these node trees should probably capture JSON names by node names. Several approaches of mapping JSON to XML have done that as far as possible (e.g. [7], [12], [13], [14], [16]) but they could not achieve two goals:

Native integration

JSON is, like XML, recursively defined and generates vocabularies. The ideal way to integrate it into the info space would be native integration: every JSON instance should be the representation of a node tree which natively expresses the JSON content, so that it can be serialized to JSON based on the node properties only, without requiring the presence of additional nodes guiding the serialization. Native integration of JSON is not possible due to differences of the XML and JSON data models: the XML model is not a superset of the JSON model.

Extension of the XML data model

Rennau [17] proposed an extension of the XML data model which claims to enable a native representation of JSON data. The proposal introduces the same boolean choice of content organization as is used in common programming language like Java: a choice between the sequence based model (which is the only model in current XML) and an alternative model which is truly map-based: the child elements are unordered, and every child element has a string-typed key – which is a new node property, [key]. The choice between the two content models is made by each complex element, and therefore it is modeled as a second new node property, [model]. It has two values, "sequence" and "map", and the relationship between the two new properties is constrained as follows: (a) an element with a non-null [key] must have a parent element with [model] equal "map"; (b) an element with a null [key] must not have a parent element with [model] equal "map". In other words: "map" elements have unordered children which must have keys, and "sequence" elements have ordered children which must not have keys.

Concerning the handling of anonymous items, Rennau proposed the introduction of "unspecific standard names", but this is unsatisfactory, as it renounces the goal of a genuinely native representation: the only natural representation of an anonymous item is an anonymous item. Therefore we should modify the XML data model by allowing anonymous elements. A JSON array would thus be an elegant representation of an element node with [model] equal "sequence" and anonymous child elements.

In summary, the following extensions of the XML data model should be considered, as they promise to overcome the limitations which disqualify the model as foundation of the info space: (a) addition of two new node properties, [model] and [key]; (b) modification of the property [node-name], introducing the possibility of a null value.

Extension of the XML syntax

Introducing new node properties implies the necessity to define the representation of these properties in XML syntax. Rennau proposed the use of "pseudo attributes" which look like attributes but do not represent attribute nodes, as they represent node properties:

However, a refusal to express the new node properties with the same immediateness as the other properties is unsatisfactory. Pseudo-attributes are an adequate representation of a node property when they can be expected to be rare, as for example xml:type attributes. However, knowing in advance that the XML representation of JSON-supplied data will produce XML documents which are studded with pseudo-attributes - rendering them hardly readable - we are motivated to go a step further and truly extend the XML syntax, introducing new syntax constructs. The goal is to ensure readable XML representations also of such node trees as are constructed from JSON instances. The following syntax extensions should therefore be considered:

   <~getWeatherResponse>
         <"temperature min">28</"temperature min">
         <"temperature max">32</"temperature max"> 
   </~getWeatherResponse>
                        
   {
      "temperature min" : 28,
      "temperature max" : 32
   }

   <~getWeatherResponse>
      <temperatureMin>28</temperatureMin>
      <temperatureMax>32</temperatureMax> 
   </~getWeatherResponse>

Extension of the XPath/XQuery language

Integration of non-XML resources into the info space is driven by the desire to make them accessible to navigation and evaluation. The extension of the data model by a new [key] property must therefore be mirrored by an extension of the XPath semantics. Rennau proposed the introduction of a third node test, the "key test" which functions in analogy to the name test: selecting elements with a given key, rather than elements with a given name. Being a node test, the key test is combined with navigational axes in the same way as a name test. The syntax of a key test is a special character (proposed: "#") followed by the key value within quotes. Examples:

   $response/#"temperature min"
   $response//#"2012"                        
   $item/ancestor-or-self::#"AC#A-2917"

   $response//#"weather"/#"temperatures"/*[3]

Whereas the key test provides for navigation, Rennau also proposed an extension of XQuery which enables the construction of nodes using JSON syntax. The extension is purely syntactical: an additional syntax which is a JSON-style shorthand notation for the construction of nodes which, due to their properties, correspond to JSON objects and arrays. For example, the following XQuery constructor:

   <*>
      <"temperature min">{$min}</"temperature min">                        
      <"temperature max">{$max}</"temperature max">
      <"temperatures">{for $v in $values return <*>{$v}</*>}</"temperatures">
   </*>

   {
      "temperature min" : $min,
      "temperature max" : $max,
      "temperatures" : [ $values ]
   }