Balisage Paper: XSLT Extensions for JSON Processing

Constructing maps

In XSLT 3.0, maps are constructed using the <xsl:map> instruction. What are the units from which a map is constructed? They are "map entries", also known as key-value pairs, which can be built using the instruction <xsl:map-entry>. A map entry or key-value pair is not in fact recognised as a first-class object in the data model; rather, it is represented simply a singleton map. And in fact, the <xsl:map> instruction simply combines multiple maps (which may or may not be singletons) into a single map.

As a reminder, the <xsl:map> instruction might be used like this:

                     
<xsl:map>
   <xsl:for-each select="employee">
      <xsl:map-entry key="@ssn" select="firstName || ' ' || lastName"/>
   </xsl:for-each>
</xsl:map>

The functionality is very close to that of the map:merge() function available at the XPath level, with one notable exception: there is no control over handling of duplicate keys. The map:merge() function offers a choice of five policies for handling duplicate keys (reject, use-first, use-last, use-any, and combine), whereas the <xsl:map> instruction offers only one: reject.

In fact experience suggests that both these designs are inadequate. In use cases I have encountered, I have wanted to handle duplicates in many different ways, and this can easily be done by allowing the policy to be specified using a callback function. When a duplicate key is encountered, the old value and the new value associated with the key can be combined by calling a user-supplied function that accepts both the old and new values as parameters. The five existing policies of map:merge() can then be expressed using the functions:

Other policies I have found useful are:

This callback capability can readily be added to both the map:merge() function and the <xsl:map> instruction. (Extending XSLT is always easier of course, in consequence of the use of XML syntax: adding attributes to an existing instruction is always straightforward.)

There's a bit of a problem with this model, however. If <xsl:map-entry> creates a key-value pair, then to transform maps into other maps it would also be useful to have further functionality for operating on key-value pairs; and it would be useful to have this in XPath. Specifically:

By providing complementary facilities for composing and decomposing maps, operations that transform maps immediately become simpler. For example filtering of maps can be done using

                     
<xsl:map>
   <xsl:for-each select="map:entries($input-map)">
      <xsl:if test="map:single-key(.) => starts-with('2022-')">
         <xsl:sequence select="."/>
      </xsl:if>
   </xsl:for-each>
</xsl:map>

Constructing arrays

There's no xsl:array instruction in XSLT 3.0, which is a major gap. Anyone trying to produce JSON output from a stylesheet will have come across this. The reason for this is the unfortunate timing of the specs: although XPath 3.1 was published before XSLT 3.0 (March 2017 versus June 2017), the XSL Working Group was reluctant to make the language dependent on anything that wasn't in XPath 3.0 — which means that support for arrays in XSLT 3.0 is rather limited.

There are two constructs in XPath for constructing arrays, and neither is completely general:

The only way to construct arrays where both the size of the arrays and the size of each member vary dynamically is to use functions such as array:join() or array:fold-left(), which can be rather clumsy.

Saxon has attempted to fill the XSLT gap with an extension instruction, <saxon:array>. Like <xsl:map, which takes a sequence of "map entries" as its operand, <saxon:array> takes a sequence of "array entries". Array entries can be constructed using an instruction <saxon:array-member> which is nicely symmetric with <xsl:map-entry>. For example the following code fragment:

                     
<saxon:array>
   <xsl:for-each select="1 to 5">
       <saxon:array-member select="0 to ."/>
   </xsl:for-each>
</saxon:array>

constructs the array [(0, 1), (0, 1, 2), (0, 1, 2, 3), (0, 1, 2, 3, 4), (0, 1, 2, 3, 4, 5)].

But what exactly does the <saxon:array-member> instruction return?

In the current Saxon implementation, <saxon:array-entry> returns an "extension object" - a Saxon extension to the XDM data model, as sanctioned by provisions in the XSLT specification (see §24.1.3). This isn't a very satisfactory solution, but it works.

The underlying requirement is to deliver a sequence wrapped up as an item (it has to be an item, because as we've seen, it's intrinsic to the XSLT language that instructions return items, and the results of separate instructions are concatenated to form a single sequence). More recently [see https://dev.saxonica.com/blog/mike/2021/06/arrays.html] I've coined the term "parcel" to describe this idea, which has applications that go beyond array construction. There are a number of ways parcels could be implemented:

The choice is in many ways arbitrary. In this paper I have adopted the second option; but experience of writing transformations this way suggests that the fourth option gives better type safety and therefore easier debugging.

However we choose to represent a parcel, there are a number of simple operations that can be defined:

We'll see that these operations are useful not only when constructing arrays, but also when processing them; one benefit is that all the XSLT and XPath machinery for processing sequences (and in particular, <xsl:apply-templates>) immediately becomes available for processing arrays.

Array and Map Construction: Summary

At the XPath level, we're proposing four functions/operations for maps, and a corresponding set of four operations for arrays:

In both cases the first two operations have exact analogues in XSLT instructions, but the second pair are XPath only, invokable from XSLT.

At one point I was recommending specific XSLT-level instructions for decomposing a map or array, for example <xsl:for-each-entry> or <xsl:for-each-member>. In fact, with the right decomposition functions available in XPath, such instructions aren't needed: we can instead use <xsl:for-each select="map:entries($M)"> or <xsl:for-each select="array:parcels($A)">. This immediately makes other decomposing operations in XSLT available for maps and arrays, for example <xsl:iterate> and <xsl:for-each-group>.

Because composition and decomposition operations use the same primitives, transformation of maps and arrays immediately becomes a great deal simpler: the general pattern is: decompose; then use all the machinery for operating an sequences to effect a transformation; then recompose.

Type-based pattern matching

We introduce pattern syntax to match items by their type.

In all cases this can be followed by predicates, for example match="array(xs:integer)[array:length(.)=2]".

However we choose to represent parcels, it's useful to have a construct match="parcel(type)" to match them, because we will be using <xsl:apply-templates select="array:parcels($array)" to decompose an array, and each parcel then needs to be matched by a template rule. (Here type is a sequence type that has to match the content of the parcel.)

Context for match patterns

To compensate for the absence of a parent axis to match items by context, we allow tunnel parameters to be referenced in match patterns. Specifically, a tunnel parameter declared in an <xsl:param> element within the body of a template rule may be referenced within the match pattern.

In addition, a match pattern in a template rule can use the new function via() to refer to items on the route by which the item was selected: specifically, via() returns a sequence of items representing the stack of apply-templates calls by which the template rule was reached, including both explicit template rules and implicit built-in rules. This is in top-down order, so fn:via()[1] represents the context item of the immediate caller.

For example

                     
<xsl:template match="record(address)[via()[1] instance of type(person)]">

matches an address only if the invoking <xsl:apply-templates> instruction had a context item of type type(person).

Built-in template rules

The current built-in template rules do not handle maps and arrays particularly well. In particular, if the processing mode has <xsl:mode on-no-match="shallow-copy"/>, and the target item is a map or array, then it is deep-copied. (This is because shallow-copy is defined in terms of the <xsl:copy> instruction, which performs a deep copy if applied to a map or array.)

To tackle this we'll define a new option, provisionally written on-no-match="shallow-copy-all", which differs from the existing shallow-copy in the way maps and arrays are handled. For arrays, it does

<xsl:apply-templates select="array:parcels(.)" mode="#current"/>

that is, it wraps each of the array members into a parcel and processes each one separately.

For maps, it does:

<xsl:apply-templates select="map:entries()" mode="#current"/>

that is, it decomposes the map into a sequence of entries (represented as singleton maps) and processes each one independently.

Because the action of the default template rule here is to construct a new array or map from the values returned by the selected template rules, there is a requirement that these template rules return values of an appropriate type for inclusion in an array or map. In particular, when constructing an array the template rules must return parcels, and when constructing a map they must return maps (which are merged).

Similarly, we define on-no-match="shallow-skip-all" which is similar to shallow-skip, but for arrays and maps it applies templates to the components of the array or map rather than omitting them entirely.

Here is a simple example that transforms the input map {'red':12, 'green':13, 'yellow':14, 'blue':15} to a new map {"green":13, "orange":20, "lilac":21, "blue":25}:

                     
<xsl:mode on-no-match="shallow-copy-all"/> 
 
  <xsl:template name="xsl:initial-template">
    <xsl:apply-templates select="map{'red':12, 'green':13, 'yellow':14, 'blue':15}"/>
  </xsl:template>
  
  <xsl:template match="record(red)"/>

  <xsl:template match="record(yellow)">
    <xsl:map-entry key="'orange'" select="20"/>
    <xsl:map-entry key="'lilac'" select="21"/>
  </xsl:template>
  
  <xsl:template match="record(blue)">
    <xsl:map-entry key="'blue'" select="25"/>
  </xsl:template>

The initial call on <xsl:apply-templates selects a map which is not matched by any template rule, so the built-in template kicks in. This constructs a new map and builds its content by applying templates to each entry (key-value pair) in the input map. The first entry (red) is matched by a rule that returns empty content, so this key is effectively dropped. The second entry (green) is not matched, so it is copied unchanged to the result. The third (yellow) is matched, and results in the addition of two new entries to the result. The fourth (blue) results in an entry being added to the result with the same key, but a different value.

First use case: bulk update

This JSON example was taken from json-schema.org^[2]:

                     
[
    {
        "id": 2,
        "name": "An ice sculpture",
        "price": 12.50,
        "tags": ["cold", "ice"],
        "dimensions": {
            "length": 7.0,
            "width": 12.0,
            "height": 9.5
        },
        "warehouseLocation": {
            "latitude": -78.75,
            "longitude": 20.4
        }
    },
    {
        "id": 3,
        "name": "A blue mouse",
        "price": 25.50,
        "dimensions": {
            "length": 3.1,
            "width": 1.0,
            "height": 1.0
        },
        "warehouseLocation": {
            "latitude": 54.4,
            "longitude": -32.7
        }
    }
]

The requirement here is: for all products having the tag "ice", increase the price by 10%, leaving all other data unchanged.

The solution to this in my 2016 paper was very cumbersome; it required a helper stylesheet with a range of supporting library functions.

With <saxon:deep-update>, this can be solved simply as

                     
<saxon:deep-update
  root="json-doc('input.json')"
  select="?*[?tags = 'ice']?price"
  action=". * 1.1"/>

I've already mentioned that I'm not entirely comfortable with this extension, partly because formalising the semantics is challenging, and partly because it involves some messy contraints on the form of the expression in the select attribute (only downwards selections allowed). However, it's reasonably usable.

How would it look as a rule-based transformation?

First, we define the default template rule to do a shallow-copy, JSON style:

                     
<xsl:mode on-no-match="shallow-copy-all"/>

The map entries we need to change are handled by the rule:

                     
<xsl:template match="record(price)[via()?tags = 'ice']">
   <xsl:map-entry key="'price'" select="?price * 1.1"/>
</xsl:template>

and that's it. Vastly simpler than the pure XSLT 3.0 solution - and no more complex than the call on <saxon:deep-update>.

The way this works is that data is processed by built-in template rules until we hit something that matches the explicit rule. The built-in template rule for the top-level map splits it into separate entries, one per key, and matches each one independently as a singleton map. The only one of these maps that matches is the one that has the key price, and is therefore matched by the pattern record(price).

The predicate on this pattern uses the via() function to match the relevant price entries according to their context. The via() function returns all context items on the stack, and the predicate matches so long as at least one of them is a map with atomized value of the tags field containing the string "ice". (We're relying here on the way that atomization works on arrays.) So long as all <apply-templates> calls are making simple downward selections, the via() function has much the same effect as using the ancestor axis when processing an XML node tree.

Second Use Case: Hierarchic Inversion

The second use case from my 2016 paper transform a JSON document that lists courses with the students enrolled on each, to create an inverted JSON document that lists students with the courses the are enrolled on.

Here is the input dataset:

                     
[{
    "faculty": "humanities",
    "courses": [
        {
            "course": "English",
            "students": [
                {
                    "first": "Mary",
                    "last": "Smith",
                    "email": "mary_smith@gmail.com"
                },
                {
                    "first": "Ann",
                    "last": "Jones",
                    "email": "ann_jones@gmail.com"
                }
            ]
        },
        {
            "course": "History",
            "students": [
                {
                    "first": "Ann",
                    "last": "Jones",
                    "email": "ann_jones@gmail.com"
                },
                {
                    "first": "John",
                    "last": "Taylor",
                    "email": "john_taylor@gmail.com"
                }
            ]
        }
    ]
},
{
    "faculty": "science",
    "courses": [
        {
            "course": "Physics",
            "students": [
                {
                    "first": "Anil",
                    "last": "Singh",
                    "email": "anil_singh@gmail.com"
                },
                {
                    "first": "Amisha",
                    "last": "Patel",
                    "email": "amisha_patel@gmail.com"
                }
            ]
        },
        {
            "course": "Chemistry",
            "students": [
                {
                    "first": "John",
                    "last": "Taylor",
                    "email": "john_taylor@gmail.com"
                },
                {
                    "first": "Anil",
                    "last": "Singh",
                    "email": "anil_singh@gmail.com"
                }
            ]
        }
    ]
}]

The goal is to produce a list of students, sorted by last name then first name, each containing a list of courses taken by that student, like this:

                     
[    {
        "email": "ann_jones@gmail.com",
        "courses": [
            "English",
            "History"
        ]
    },
    {
        "email": "amisha_patel@gmail.com",
        "courses": ["Physics"]
    },
    {
        "email": "anil_singh@gmail.com",
        "courses": [
            "Physics",
            "Chemistry"
        ]
    },
    {
        "email": "mary_smith@gmail.com",
        "courses": ["English"]
    },
    {
        "email": "john_taylor@gmail.com",
        "courses": [
            "History",
            "Chemistry"
        ]
    }
]

The approach I used in the 2016 paper (constrained by the inability to access ancestors in the JSON tree) was a two pass approach: first flatten the data into a normalised table (represented as a sequence of maps) containing (course, student) tuples; then group these tuples by student name.

The basic approach with new language features remains the same, but I think we can improve a little on the detail.

First, we can collect together the "flattened" sequence of (course, student) tuples like this:

                     
<xsl:mode name="flatten" on-no-match="shallow-skip-all"/>

<xsl:variable name="flattened" as="record(course, first, last, email)*">
  <xsl:apply-templates select="json-doc('input.xml')" mode="flatten"/>
</xsl:variable>

<xsl:template match="record(first, last, email, *)" mode="flatten">
  <xsl:sequence select=". => map:put("course", via()?course)"/>
</xsl:template>

and then we build a new hierarchy over the flat sequence of records:

                     
<xsl:template name="xsl:initial-template">
   <xsl:array>
      <xsl:for-each-group select="$flattened" group-by="?email">
         <xsl:array-member select="map{'email': ?email, 
                                       'courses': array{current-group()?course}}"/>
      </xsl:for-each-group>
   </xsl:array>
</xsl:template>

Again, vastly simpler than the pure XSLT 3.0 solution in the 2016 paper.