Balisage Paper: From XML to UDL: a unified document language, supporting multiple markup languages

The node model as a unified document language

The node representation of JSON markup

Extensions of the XML markup language

Extensions of the XPath language

Extensions of the XQuery language

Extensions of the XML node model

The XML node model is extended by two new node properties, [model] and [key]. The result of these changes is a unified node model which can represent XML documents, JSON documents as well as nested combinations of JSON and XML fragments as a tree of nodes which is accessible to XPath navigation and, by implication, XQuery and XSLT processing.

Details

Extensions of the XML markup language

The extensions have two purposes: (a) express the new node properties; (b) support the use of non-XML markup within XML documents.

Extensions of the XML serialization model

The serialization model must be extended in order to support JSON output.

Extensions of the XPath language

The extensions are designed to make the processing of JSON data as powerful and convenient as the processing of XML data. Namely, the [key] property can be checked by a key test, similarly to the checking of the node name by a node test.

Extensions of the XQuery language

As JSON items correspond to element nodes, there is no principal need to introduce new constructor expressions. In element constructors, the pseudo-attributse udl:key and udl:model are used in the same way as they are used in XML markup. In order to reduce verbosity, however, several abbreviated variants of element constructors are introduced.

UDL - pseudo-attributes and pseudo-tags

Pseudo-attributes are syntactical constructs which have the lexical form of attributes but can be distinguished from them by the use of a reserved QName. Pseudo-attributes do not represent an attribute node. Instead, they represent a node property (udl:key, udl:model) or a default value of a property (udl:defaultModel), or they identify the markup language used locally (udl:markup).

Pseudo-tags are syntactical constructs which have the lexical form of element tags but can be distinguished from them by the use of a reserved QName. One pseudo-tag is introduced (udl:markupSection) which delimits a section of non-XML markup.

Four further names from the UDL namespace are used as default element names, given to the nodes constructed from JSON values. It is important to note that these names have no specific semantics and can be used as node name without restrictions like any other QName. The only specific treatment of these names is when serializing to JSON using the json.strict mode. In this case a node name which is different from the default name expected (according to the node properties) is considered information that would be lost during serialization (see section “ Serialization: controlling the loss of information ”).

The following table summarizes the use of QNames from the UDL namespace.

Mixing markup styles

The UDL defines a unified document model which can be represented by different markup languages. This unified content of heterogeneous outward shape invites not only a free choice of the markup language actually used. It also implies that markup languages might be mixed within a document, based on simple rules how to delimit the various chunks of markup. These rules are provided by the udl:markup pseudo-attribute and the udl:markupSection pseudo-tag (see section “ Supporting non-XML markup ”).

Occasionally there may be good reasons to use mixed styles. Consider the case that the document as a whole cannot be represented as JSON (e.g. because of attributes and namespaces), but subtrees represent JSON documents (perhaps imported from pure JSON sources, e.g. logged messages). Without the mixing of markup styles, the resulting serialization would be difficult to read, due to the very verbose XML representation of JSON nodes. It should of course be remembered that this mixing of markup styles has no impact on the information content of the UDL document, which is exclusively defined in terms of nodes and their properties.

The following section describes in detail an additional markup style, which amounts to a "small-scale mixing" of XML and JSON, dubbed telem (text notation for simple elements).

XML syntax variant: telem

XML markup representing JSON data is ugly. Typically it contains many elements which correspond to simple values and are tiresome to read. The distinctive property of the elements is shifted from the eye-catching node name to a pseudo-attribute, and the markup is often inflated by explicit type annotations:

  <udl:value udl:key=”foo”>someContent</udl:value>
  <udl:value udl:key=”bar” xsi:type="xs:integer">99</udl:value>
  <udl:value udl:key=”foobar” xsi:type="xs:boolean">true</udl:value

whereas the JSON representation could not be more succinct:

  "foo" : "someContent",
  ”bar”" : 99,
  "foobar” : true

Fragments containing only JSON nodes can best be represented by switching to JSON. But sometimes such JSON values occur interspersed with conventional XML elements which have specific names, have attributes, etc. In such cases it is attractive to apply the JSON style to the simple values and retain XML style for the fragment as a whole. This option is provided by the telem markup style.

This style is XML markup augmented by a shorthand representation of simple elements meeting several constraints:

The syntax corresponds to the JSON representation of a simple or null value, or of a name/value pair with simple or null value, depending on whether the element has a [key]. If consecutive element children are represented in telem style, these representations are separated by a comma. If the value is not put in quotes, it must be a number or one of the constants true, false or null, which will be interpreted as implicit type information, following the JSON rules. Example: the following fragment

   <e udl:model="map">
      <udl:value udl:key="mode">repeated</udl:value>
      <udl:value udl:key="nrep" xsi:type=”xs:integer”>52076</udl:value>
      <udl:value udl:key="eval" xs:type=”xs:boolean”>true</udl:value>
      <locInfo udl:key="cities">
        <udl:value>Paris</udl:value>
        <udl:value>London</udl:value>
        <udl:value>Manchester</udl:value>
      </locInfo>
   </e>

may be alternatively represented this way:

   <e udl:model="map" udl:markup=”telem”>
      "mode" : “repeated”,
      "nrep" : 52076,
      “eval” : true
      <locInfo udl:key="cities">
        "Paris",
        "London",
        "Mancester"
      </locInfo>
   </e>

Both representations have the same information content.

Deserializing from / serializing to JSON

The exact rules for translating JSON into UDL (deserialization) and for translating UDL into JSON (serialization) are listed in the appendix (section “ Deserialization ” and section “ Serialization ”). In this section, the principles are summarized.

Serialization: controlling the loss of information

Serialization of a document to a markup language should preserve all information so that the serialization is a complete representation from which the document may be reconstructed. Such a lossless serialization of a UDL document is always possible for XML markup; it is only in special cases possible for JSON markup. For example, any attributes or non-default element names are lost when serializing to JSON.

However, it depends on circumstances whether such loss of information renders the serialization result worthless. If, for example, the loss consists of element names only and these names were only introduced in order to facilitate document creation or processing, with an understanding that they will get lost during later processing steps – then a serialization which loses element names might be as valuable as a lossless serialization. Such considerations suggest a refinement of the serialization model: a new serialization parameter might control what losses are acceptable and what losses are not acceptable.

The proposal of a unified document language includes such a new serialization parameter: info-loss. Presently the parameter is only relevant when serializing to JSON. Three values are defined:

In particular, info-loss equal json.projection means:

UDL and XSD

The proposed extensions of the XML node model amount to the introduction of two new node properties. Obviously, they require also an extension of the XSD language. In particular, constraints concerning the [key] property should be supported. However, such changes are out of scope of this paper.

Issue: mapping arbitrary XML to JSON

A serialization of arbitrary XML documents to JSON is usually not possible without a loss of information, as JSON cannot natively express element names, the distinction between attributes and elements, mixed content and the occurrence of siblings with the same name. This does not mean that JSON could not be used to represent the complete information content of an arbitrary XML document. This representation would however not be a serialization of the XML document tree to JSON, but the (serialized) result of a transformation, a different node tree, adhering to a specific format which is designed to capture the content of arbitrary XML documents (e.g. [JsonML] ). The equivalence between the resulting JSON (or the node tree it represents) and the original XML document is not based on the data model, but established on the level of a specific mapping application (as a set of rules).

Issue: mapping JSON to readable XML

A similar problem concerns the translation of JSON documents into readable XML documents: the documents created by parsing JSON as defined by UDL are well-suited for processing (e.g. per XPath, XQuery, XSLT), but when serialized into XML text look hardly readable. The practical need to obtain a well readable XML representation of a given JSON document, however, cannot be denied. (Think, for example, of a web service which may at user option deliver XML or JSON results). Again, it is a transformation from one tree to a different tree what is required, as opposed to serializations into alternative formats.

Issue: Round-tripping XML - JSON - XML

The impossibility of serializing arbitrary XML to JSON of course implies that round-tripping XML-JSON-XML is not generally possible solely based on serialization and parsing.

Conceivable extension of UDL: integration of standardized mappings

Given the scope of UDL’s main goal – a unified document language supporting multiple markup languages – it may be questioned if the UDL concept is complete if not addressing fundamental mapping tasks, too. A conceivable extension of UDL might include two parts: (a) the definition of mappings, which are standardized transformations (XML to JSON, lossless; JSON to readable XML; (b) the integration of these mappings with parsing / serialization into new “first-class” operations, “mparse” (parse & map) and “mserialize” (map & serialize). The appendix contains a first step in this direction [Appendix B], which is, however, limited to the use case of JSON documents in which all names are NCNames. Another limitation is that it does not yet support attributes in the mapping result, which probably cannot be the last say.

The mapping between XML and JSON is a question to which already many answers have been given (e.g. [Lee], [Pemberton], [Couthures], [JsonML], [BaseX]). The diversity is mainly due to differences between the exact goals which the solutions pursue (concerning mapping direction, losslessness, readability, configurability, …). It is unclear if the suggested extension of UDL, which involves standardized mappings between XML and JSON, is a realistic task. But it is not hopeless, neither. First, the unified node model provides a conceptual base which other mapping approaches did not have. Second, the goals can be defined precisely, which greatly removes competition between existent solutions and a new standard. And finally, the use of other, non-standard mappings would always remain possible.

The mapping approach

The mapping approach (e.g. BaseX, Pemberton) is based on an XML representation of JSON data. It uses a simple processing model:

This is a clean solution, provided the XML representation preserves all information contained in the JSON data, and the mapping rule can be applied bidirectionally without loss of information. To define such a mapping is not very difficult, as one can use reserved element names and introduce helper attributes in order to exclude any information loss (see Table V for examples). Outwardly, UDL looks similar to such a mapping-for-the-sake-of-processing; it is tempting to classify it as yet another mapping variant. But that would be a mistake. Mapping approaches treat the problem as an XML application: introduce a specific XML dialect designed to achieve a particular goal. Like any application, these approaches are free to require the use of application-specific element names and the addition of attributes with application-specific semantics, to be evaluated by application code. The type attribute, for example, used in [BaseX] is a helper attribute which clearly duplicates xsi:type for certain values, yet nevertheless had to be introduced as additional attribute, because the value range includes values array and object with ad hoc semantics dictated by the mapping task. Such attributes reveal the fact that the current XML node model does not support a bidirectional mapping into JSON markup. To enable such a mapping, the node tree must contain special items with serialization semantics. This is at odds with the basic principle of serialization being a process solely controlled by serialization parameters, without a need to interfere with the information content of the node tree.

UDL does not map XML nodes to JSON structures. It redefines JSON to be a representation of nodes. As a consequence, it need not "inject" any ad hoc items into the data tree for the sake of controlling a serialization to JSON. None of the reserved element and attribute names in Table II have anything to do with JSON or serialization. Rather, they represent standard names without semantics, reflect node properties or signal the markup language currently used within a well-defined scope. The extended node model is expressive enough to represent JSON structures natively.

The main difference between UDL and mapping approaches concerns the handling of JSON names. Mapping approaches represent JSON names as element names if possible, and if not, resort to one of two possible solutions: either place the JSON name in an additional attribute (e.g. Pemberton), or define a bidirectional name mapping (e.g. BaseX). But there are three differences between the concepts of JSON names and XML names:

Note that the last point (the uniqueness constraint) means that a JSON name resembles an xml:id attribute more than an element name. It can be compared to a locally scoped xml:id attribute (uniqueness among all element children of an element). For these reasons UDL distinguishes the concepts of names and keys. It thus enables native relationships between nodes and XML markup on the one hand and JSON markup on the other hand. As a result it becomes possible to regard JSON markup and XML markup as alternative representations of an information content which is defined in terms of nodes and their properties. Remembering Plato, one kind of “thing” is inferred from - or may cast - two different "shadows".

Should we not keep things simple - do we need to extend the document model and introduce new node properties? Imagine this alternative:

This is an attractive scenario: one can formulate the XPath expressions without a mental translation, just using the names one sees in the source data. The net result is an elegant approach to the processing of JSON data with XML tools.

The approach would be a good – and perhaps a better – alternative to UDL if the processing of JSON documents with XML tools were the only goal. However, UDL's design aims at a unified document model which expresses the entities represented by dominant markup languages in a native way. Only this way can information content and representation (markup) be decoupled and can the latter be switched easily at various scales (whole documents, sections or single elements) and in various contexts (data and program code).

JSONiq

JSONiq [JSONiq], [JSONiq Specification] is an extension of the XQuery language designed to add support for JSON data. Like UDL, JSONiq extends the XDM in order to accommodate JSON structures. However, JSONiq does not change the node sub model of the XDM. Rather, two new item types are introduced, designed to represent JSON data:

It is interesting to note the parallel: both, JSONiq and UDL extend the XDM in order to accommodate JSON data; but the changes UDL proposes are within the node sub model, whereas JSONiq adds a second sub model for structured data, in parallel to the node model. Using the terms proposed in section “Distinction between markup and document language”: JSONiq keeps the XML document language, but extends the XML information language, whereas UDL shifts the changes into the very document language, refraining from changes outside of the node model.

An advantage UDL offers is to represent JSON data as node trees and thus expose them to XPath navigation. JSONiq, on the other hand, might be easier to accept exactly because it does not introduce any change to the document language and therefore restricts itself to the extension of a query language, rather than an extension of XML.

Map items

The W3C XSL Working Group has made a proposal for extending the XDM by a new item type: map items [W3C XSLT 3.0]. They represent generic containers, but nevertheless can represent JSON data. One should note the relationship between JSONiq and the map proposal: both approaches mandate new item types which are not nodes and yet can represent structured data; but one (JSONiq) resorts to JSON-specific items, whereas the other advocates generic containers.

Not being nodes, map items are lightweight containers which can collect items without requiring or imposing a structural relationship. Therefore node relationships between container and members are not possible. This contrasts sharply with the UDL approach which models JSON containers and their members as nodes and their child nodes. Only this way can JSON data be seamlessly integrated into the navigational system based on axes and node tests.

This is not to say that such lightweight containers would not be very useful extensions of XDM. Lightweight containers and nodes cannot replace each other. A node model is required for the navigational power of XPath. Lightweight containers are required to model node relationships independently from their structural relationship. And among other benefits they enable a mapping of keys to node references, rather than the nodes themselves, which is a highly desirable feature.

It is interesting to note a conceptual relationship between UDL and the "map proposal": the content of an element with [model] equal "map" can be described as a map item constrained in the following way: (a) every map value is a child element; (b) every map key is the [key] of the associated value.