Balisage Paper: JXON: an Architecture for Schema and Annotation Driven JSON/XML Bidirectional Transformations

JXON attempts to solve this problem

As an example, the following is an RNG^[5] Compact Notation schema which fully describes both the schema and rules for the bidirectional lossless transformation for the example above. This is all that is needed for JXON to produce both XML to JSON and JSON to XML transformations.

default namespace = ""
#<jxon:pattern name="simple"/>
grammar {
start =  Books

#<jxon:children wrap="array"/>
Books = element BOOKS { Book+ }

#<jxon:json_name omit="true"/>
Book = element BOOK {
      attribute id { xsd:NMTOKEN },
      Title, 
      Price
}
#<jxon:json_name name="title"/>
Title =    element TITLE { text }

#<jxon:json_name name="price"/>
Price = element PRICE { xsd:decimal }
}

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified"  xmlns:jxon="http://www.xmlsh.org/jxon">
  <xs:annotation> <xs:appinfo>
      <jxon:pattern name="simple"/>
    </xs:appinfo>
   </xs:annotation>
  
  <xs:element name="BOOKS">
    <xs:annotation > <xs:appinfo>
        <jxon:children wrap="array"/>
      </xs:appinfo>
     </xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:element maxOccurs="unbounded" ref="BOOK"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  
  <xs:element name="BOOK">
    <xs:annotation> <xs:appinfo>
        <jxon:json_name omit="true"/>
      </xs:appinfo></xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="TITLE"/>
        <xs:element ref="PRICE"/>
      </xs:sequence>
      <xs:attribute name="id" use="required" type="xs:NMTOKEN"/>
    </xs:complexType>
  </xs:element>
  
  <xs:element name="TITLE" type="xs:string">
    <xs:annotation> <xs:appinfo>
        <jxon:json_name name="title"/>
      </xs:appinfo></xs:annotation>
  </xs:element>
  
  <xs:element name="PRICE" type="xs:decimal">
    <xs:annotation> <xs:appinfo>
        <jxon:json_name name="price"/>
      </xs:appinfo>
   </xs:annotation>
  </xs:element>
</xs:schema>

Anonymous values

JSON values are anonymous. They only acquire names by being referenced in an object (map). For example, the following is a valid JSON document.

	"Hello World"	

	<root>Hello World</root>
or
	<root value="Hello World"/>

In the degenerate case this means entire JSON documents can be composed of single (or arrays of) anonymous values which have no direct mapping to the XML data model.

When mapping JSON to XML this additional markup cannot generally be derived from the JSON data itself, but rather needs external rules to decide on the approach. When mapping XML to JSON the presence of an element with text children or attribute value may not necessarily imply creation of an anonymous JSON value, but rather the element and attribute names are commonly used to create named members of JSON objects such as

	{ "root" : "Hello World" }
					or
	{"root" : { "value" : "Hello World" }}

Arrays

JSON has a native representation of arrays which does not exist as native types in XML. XSD Schema has support for values which can be interpreted as tokenized lists, but there is no direct markup for an array of objects. Arrays can be represented in XML in several ways ; repeated elements, repeated elements grouped by a common parent, usefully named attributes, usefully named elements and tokenized strings.

For example the following JSON array

	[ 1 , "String" ]

	<array>1 String</array>
	
	<array><entry>1</entry><entry>String</entry><array>
	
	<array entry1="1" entry2="String"/>
	
	<entry>1</entry><entry>String</entry>

Mapping XML to JSON arrays is possible but since there is no single mapping, and since XML semantics by itself doesnt indicate array representation, one must know via external information to treat specific XML markup as a JSON array. Given any of the above examples, its not obvious that they should be mapped to a JSON array as opposed to some other JSON structure (such as a simple value or object).

Implicit Types (e.g. "Duck Typing")

JSON values make use of implicit types which can be inferred by the serialization format. For example 1 is a Numeric type, while "1" is String type. This is common in programming languages, but not as common in markup languages. In XML, without a schema, the token 1 is xs:anyType and it's up the application to infer a type (possibly using a schema, or up to application logic).

This leads to problems mapping XML to JSON. When a plain character string value from XML is mapped, what JSON type should be used? It's tempting to use the value instance to deduce the type, but this could be very dangerous. An application may be expecting string types and if one instance happens to translate 1 to a numeric type because it looks like one, and another translates "one" to a string this could break the application.

Translating atomic values from JSON to XML is not as problematic as generally typing can simply be ignored because the serialization format for string and numbers is the same in XML, although other atomic types like boolean, dates etc can be more problematic.

Identifiers

JSON allows any string as a identifier (Object member name). XML is much more restrictive for identifiers. Translation of XML identifiers (element and attribute names) to JSON member names causes few problems, but the reverse can lead to invalid XML. For example "a value" is a valid member name in JSON but not a valid attribute or element name in XML

Namespaces

XML supports namespaces, while JSON does not. Mapping QNames in XML with namespaces to member names in JSON can lead to ambiguous, complex or duplicate names. Translating member names in JSON to XML QName may lose namespaces.

Processing Instructions

XML supports Processing Instructions, which are absent from the JSON data model.

Attributes

JSON has no concept or representation for Attributes (as distinct from other values).

When mapping XML to JSON, attributes are often translated to named object members (along with child elements). When mapping JSON to XML it requires external knowledge to determine when to map members to attributes.

Character Set

JSON supports a broader set of text characters (Unicode code points) then XML. For example JSON supports the NUL character ('\0') whereas NUL is invalid in XML even if encoded as an entity (�). This implies that JSON character data cannot be mapped to XML character data losslessly without application specific encodings.

Comments

XML directly supports comments while JSON does not. Comments were originally allowed in JSON but Douglas Crockford removed them ^[6]

Comments could be encoded in JSON using special object member names (such as "_comment") requiring application support to understand these were comments and not data.

A common case is to simply strip comments from XML when mapping to JSON, and to not insert comments when mapping JSON to XML.

Document Node

JSON does not have a Document Node, where XML does. This is generally not problem in practice as the XML Document Node is usually implicitly created in the model, and has no textual representation.

Serialization

The text Serialization format for JSON and XML differ significantly. Often the serialization format is the main focus of JSON vs. XML mappings. I assert that the serialization format is completely irrelevant for purposes of mapping and transformations; the formats are well defined and there exist a large number of implementations which can parse and serialize XML and JSON. If one focuses instead on the abstract data model, the issue of serialization (both parsing and generation) becomes a trivial implementation issue.

Encodings

Related to the Serialization format, encodings for JSON and XML can differ. JSON is defined to support only the UTF-8 encoding while XML can use many different encodings. Like serialization, I assert that encoding is irrelevant. If one works with the JSON and XML abstract data Models instead of their serialization and encoding formats these issues disappear.

Limitations of existing implementations

Existing implementations and designs share many common limitations which make them less then ideal for many use cases. The existing models tend to fall into these use cases.

In cases where round tripping is a goal inevitably either the JSON or the XML end up being very "non friendly". Conversely mappings to "friendly" formats tend to lose too much information to support round tripping (or only in a small subset of documents).

All investigated designs have fixed rules that apply either globally, or if configurable, to an entire document. None of the designs can adjust the mapping rules localized to portions of a document.

The major limitations which JXON attempts to address include the following.

Categories of Transformations

There are several major categories of JSON/XML transformation problems which are really subsets of functionality. Not all use cases require all levels of functionality but by supporting the most stringent requirements implicitly creates support for the rest. In general there is a correlation between information fidelity and the category of transformation. For example a unidirectional XML to JSON transformation may lose significant amounts of information (and yet still be useful). While a full round trippable transformation, by its very nature, must maintain a maximum amount of information in either the instance documents or the transformation logic. Note that these transformation categories are not specific to XML and JSON but apply equally well to any kind of document formats.

Figure 1: Types JSON/XML transformations

Types of JSON/XML transformations

Use Cases for XML/JSON transformations

There are several use cases that require or benefit by one of the categories of XML/JSON transformations. The more useful cases are where the data in a domain not only is desired in a specific format, but also persist or may be manipulated by tools specialized for that format and that the data needs to be exchanged with another domain using a different format. Examples of good use cases for transformations include the following.

Goals

The goals of this design are to address many of the limitations of existing implementations and to provide a simple method for customizing the transformation. At the same time it's recognized that not all limitations can be addressed, nor necessarily need be addressed, and that this design may not meet the objectives of everyone. The use cases See section “Use Cases for XML/JSON transformations” were of particular use in focusing the design goals.

Design Overview

Figure 2: JXON Processor

Figure 1. JXON Processor data flow

Figure 3: XML to JSON

Transforming XML to JSON data flow

Figure 4: JSON to XML

Transforming JSON to XML data flow

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:jxon="http://www.xmlsh.org/jxon">
	<xs:annotation>
		<xs:appinfo>
			<jxon:pattern  name="full"/>
		</xs:appinfo>
	</xs:annotation>
	...

<xs:element name="ITEM">
		<xs:annotation>
			<xs:appinfo>
				<jxon:pattern  name="simple"/>
				<jxon:json_name name="BOOK"/>
			</xs:appinfo>
		</xs:annotation>
...

#<jxon:json_name  name="TITLE"/>
element NAME { xs:string }

<xs:element name="NAME">
		<xs:annotation>
			<xs:appinfo>
				<jxon:json_name name="TITLE"/>
			</xs:appinfo>
		</xs:annotation>
...

Atomic Types

JXON uses the schema type declarations for simple types to attempt to determine the proper JSON type. The XSD schema type hierarchy is searched until a matching known type is found. If a matching type is found then the corresponding JSON type is used, otherwise the String type is used.

Any type derived from xs:double, xs:float or xs:decimal are mapped to a JSON Number.

Any type derived from xs:boolean is mapped to a JSON Boolean literal.

All other types are mapped to a JSON String.

Empty XML elements are mapped a JSON null.

	<value wrap="schema" type="schema" />

	<value type="number" />

Structured Types

JXON maps XSD "list" types (and derived types such as IDTOKENS) to JSON arrays and on reverse maps the associated arrays back to a space separated atomic value.

The value element can be used to force wrapping by a JSON array even if the schema does not identify a value as a list type. For example the following overrides the schema information and tells the JXON processor to wrap the value with a JSON array of strings.

	<value type="string" wrap="array"/>

Naming and namespaces

JXON provides a flexible method for configuring name transformations. JSON and XML names can differ significantly in possible characters and practical use. The default configuration is to ignore namespaces and use the same name for XML elements and attributes as JSON members. Although namespaces are not present in the JSON document they are reconstructed when translating JSON to XML. Thie naming rule can be overridden on a type, element or attribute basis (and child items) by supplying a regular expression which translates the full XML name (in Clark notation^[10]) to the JSON name. This allows the uri and localname portions of a QName to be used in generation of a JSON name. Since the document instance is not avaiable during rule generation, namespace prefixes are not known so are not accounted for in the naming rules. You do not have to supply the reverse regex for reverse mappings, in fact the mappings can be lossy while still being fully reversable because the naming information is stored in the XSLT files. The JSON member names have to be unique within a JSON object, however, in order to be able to match the corresponding XSLT rule (and XML name). It is interesting to note that even if namespace information is not mapped into the JSON output, that namespaces are not lost on reverse mapping. It is only in the case where multiple element or attributes with the same name but in different namespaces are used within a single schema and document instance that namespaces need be considered at all in the nameing mappings to provide for lossless round tripping.

This naming rule can also be used to get around the problem of same-named attributes and elements in XML by assigning them unique names in JSON.

Elements and Attributes

XML Elements and Attributes are both mapped to JSON Object members. JSON Object members must be a direct child of a JSON Object. This implies that all XML Elements and Attributes must be placed as direct children of a JSON Object (must not be parent-less or a direct child of a JSON Array.). The Element and Attribute rules interact with the Wrapping Rules to enforce this constraint as well as provides variants on how elements and attributes are placed within the containing object.

	<children wrap="object" name="_children" />

	<children wrap="none" />

	<attributes wrap="object" name="_attributes" />

	<attributes wrap="none" />

Text elements

Text elements can be treated as simple values or as objects. They can be wrapped along with child elements, or in their own member. The JXON text element determines the behaviour.

The following example from the "simple" pattern wraps all text into an object member named "_text".

	<text wrap="object" name="_text" />

The following example from the "full" pattern treats text the same as child elements (putting each text element as a separate value in the _children array)

	<text wrap="none"/>

JXON Processor

The JXON processor (Figure 2) is run once per schema to produce the pair of transformation XSLT files. The program (called "jsonxslt" Figure 5.) is written partially in Java and partially in XQuery. The Java portion parses command line arguments, reads the Schema (using the Apache implementation of the XML Schema API) and generates a much simpler form of the schema and annotations in XML form. The intermediary schema format corresponds to the same schema as the configuration format as well as the annotation (JXON). This allows the configuration data to be cleanly merged with the annotation and the resulting document and internal representations all having the same namespace and sharing the same meaning for items.

Java is used for this component because parsing, and particularly comprehending XSD schema is a challenging task suited well to reusing an existing API. The simplified schema and the global configuration file is then passed to the XQuery component, once each for conversion to XSLT for the JSON to XML translation and again for producing the XML to JSON. XQuery was chosen for this component because it is a good fit for XQuery. The process is largely that of querying and producing XML (XSLT) output from a complex set of algorithms. The two XQuery programs are tightly coupled (and share common modules). This is necessary because the specifics of the XSLT rules generated for one direction need to match exactly the rules generated for the reverse direction in order to achieve round trip transformations.

The result is 2 XSLT files which have encoded all the rules, augmented by the annotations, into a set of XSLT templates. These templates match every element and attribute of the input (Either XML or JXML), apply the encoded transformation and output the transformed XML. (See section “Example XSLT generation”

Converting between JSON and JXML

As described in the design, the JXON processor consumes and produces JSON in JXML format. This could in theory be any other full fidelity JSON representation in XML.

This JXML format may be useful by itself as it can be converted to other JSON XML formats such as JSONX jsonxor used as an input to other processors that operate purely on the JSON Object Model as apposed to the JSON Serialized Text form.

To convert from JSON to JXML and reverse, two tools are used from xmlsh command library (json2xml, xml2json). These tools are very simple and can be run standalone (outside the xmlsh environment), or could be re-implemented easily. Source code is available for these tools as well as all of xmlsh, also licensed using the Simplified BSD License. The json2xml tool makes use of the JSON API available from json.org, with some minor enhancements, mainly to preserve the order of object members while parsing JSON. The xml2json tool does not use the JSON API because of severe limits to that library, rather the serialization was written from scratch.

Neither of these tools are particularly complicated and could be replaced by user written code. The techniques for parsing and serializing JSON are well known and many implementations are available. You can see from the sample below that the JXML format is extremely verbose, but its advantage is it directly models the JSON data model.

XSLT processing

Once the JXON process is complete the 2 resulting XSLT files can be used at any time. One file is used to transform JSON as represented in JXML to the target XML See Figure 4.

The other file is used to transform XML to JXML See Figure 3.

The conversion between JXON and JSON is performed using the json2xml and xml2json tools See section “Converting between JSON and JXML”

The XSLT files are currently created to run with any XSLT 2.0 non-schema aware processor. The implementation uses Saxon^[12]

Use of XSD Schema

Recognizing that Schema was a core concept in the design, XSD Schema was chosen for the implementation. In concept any schema language should work, but XSD was chosen due to its prevalence and the availability of API's which can query the schema (XML Schema API). XSD Schema is complex. The fact that an API is needed to query it to get meaningful information (instead of say using XQuery directly on the schema), is an indication of how complex it is. Even the API itself is complex. This decision required the use of a Java component instead of relying entirely on XML tools.

Ideally any XML Schema language should be able to be able to be used. Since the source schema is only used early on in the work-flow (to generate a simpler intermediate document), supporting alternative schemas such as RNG natively should not be that difficult and may be introduced in a later version of the implementation. The current support for RNG via translation to XSD is a viable alternative.

Use of Schema Annotations

The choice of using XML Schema (particularly XSD) then led to the decision to put the human augmented information into the XML schema document itself (as annotations). This decision is definitely debatable.

The advantage of annotating the schema itself instead of using an external "rules file" is that information can be associated directly to the specific item (type, element, attribute) without having to target that item externally with perhaps XPath or type references. The transformation logic then resides right with the item definitions in the schema and contains a minimal amount of extra information. On the other hand, this forces editing of the schema in order to modify the transformation logic. Different transformations for the same class of documents then requires separate schemas. This means that different schemas may need to be maintained that differ only in the JXON annotations. While JXON does not require that instance documents use the same schema during transformation as was used during the generation process, this can be a problem as changes to the authoritative schema require editing the schemas used for the JXON configurations.

Another issue of annotating schema directly is that it requires duplication of information if the same properties are to be applied to different items. This is mitigated by the use of "patterns" which group properties and allow you to specify only the pattern name. However if the rules were in an external file they could use a single definition for common properties and use instead multiple targets (e.g XPaths) to apply the properties to multiple items.

Intermediate file (JXON document) generation in Java

Use of the Apache implementation of the XML Schema API requires use of a Java component for part of the generation process. To minimize the reliance on Java coding an intermediate XML file is generated with a simplified version of the schema along with the annotations. This worked out fairly well as it separated the logic for parsing the schema from the rest of the processing. One problem is that the format for the intermediate is a moving target. Exactly what constitutes a "minimal simplified representation of the schema" depends on the needs of downstream processing. Adding new features and making changes to the XSLT generation code often requires going back to the Java code to add or change the content of the intermediate file.

Once the initial stage is complete then the rest of the XSLT generation can be done using pure XML tools. This allowed for easier development and debugging.

Use of an intermediate representation for the schema also benefits from being schema language agnostic. A different input schema language (such as RelaxNG) could be used while generating the same JXON intermediate format and leaving the rest of the processing unchanged.

XQuery for XSLT generation

XQuery is used to generate the XSLT output given the intermediate schema file and a configuration file. Choice of XQuery seems a good fit for the problem. The input and output is entirely XML and the processing logic is query and data oriented logic operations. This fits the XQuery use case quite well. Certainly XSLT or Java could have been used for this component, but the choice of XQuery worked well.

XSLT for transformation

The output of the JXON processor is a pair of XSLT files. These XSLT files are used for the actual transformations. Choice of XSLT seems an obvious match. The transformation is a collection of matching templates along with generation rules. This fits the XSLT model very well.

Naming and namespaces

The mechanism for customizing name and namespace mappings is based on regular expressions. This is quite powerful but may not be powerful enough. The reason regular expressions were chosen is an artifact of the implementation; is that its the most flexible string manipulation feature in XQuery which doesn't require an "eval". This area could use enhancements.

The naming rules only have access to the localname and URI of the QName. This means that prefixes cannot be used as part of the name mappings. When generating XML from JSON prefixes are constructed and namespace declarations are added as needed. The resulting XML will most likely have a different lexical representation then a source XML after round tripping due to the difference in prefixes and locations of namespace declarations. Possibly making use of the prefix bindings in the XSD would be useful although there is no guarantee these same prefixes would be used in instance documents.

Using the default naming rules which ignore the namespace when generating JSON can lead to ambiguity problems if two QNames from different namespaces are allowed as attribute or child element names in a given element. This is resolved by overriding the naming rule for one of the ambiguous names.

Better control over prefix assignment and namespace declaration placement would be a useful enhancement.

Character Set

JXON uses JXML's handling of non-XML representable characters such as \0 by keeping them in JSON escaped format when mapped to XML. While this is losses, it is non-ideal as it takes application logic to parse these values correctly. I know of no better solution to this problem as XML simply cannot encode the full Unicode code-set which is allowed in JSON.