Balisage Paper: Carrot

Global variables

A global variable definition is very similar to a variable declaration in XQuery, except that you don't need the "declare variable" verbiage. Whereas in XQuery you would write:

declare variable $foo := "a string value";

In Carrot you would instead write:

$foo := "a string value";

Functions

A function definition is just like a function declaration in XQuery except that you don't need the "declare function" verbiage and, instead of curly braces, you use the same binding operator (:=) as a variable definition. For example, whereas in XQuery, you would declare functions like this:

declare function my:foo() { "return value" };
declare function my:bar($str as xs:string) as xs:string { upper-case($str) };

In Carrot, you would instead write:

my:foo() := "return value";
my:bar($str as xs:string) as xs:string := upper-case($str);

Why not just use the regular XQuery syntax? Two reasons: conciseness (lower signal-to-noise ratio) and consistency (with the other two types of definitions).

Rules

The third type of definition is a rule. This corresponds to a template rule in XSLT. For example, this rule matches any element node (*):

^foo(*) := "return value";

Unlike a function definition, the "argument" of a rule definition ("*" in the above case) is not an (optional) formal parameter list; instead it is a required pattern (as XSLT defines a pattern). Thus, it's illegal to have an empty set of parentheses in a rule definition:

^foo() := "return value"; (: NOT LEGAL :)

Note the asymmetry with ruleset invocations, where it is legal to call ^foo(), which is short for ^foo(node()).

Of course, rules can also have parameters (just as template rules can have parameters in XSLT). The syntax for declaring these is very similar to an XQuery function parameter list, except that it comes after the pattern and is separated from the pattern by a semicolon:

^foo(* ; $str as xs:string) := concat($str, .);

Carrot also supports tunnel parameters, as in XSLT. To indicate a tunnel parameter, you add the keyword "tunnel" before the parameter:

^foo(* ; tunnel $str as xs:string) := concat($str, .);

Unlike XQuery functions, parameters in a rule are identified by name, not position. Thus the syntax for passing them looks very similar to how they are declared, and the order of parameters is insignificant. The following expression applies the "foo" ruleset to the context node, passing the tunnel parameter $str with the value "Hello":

^foo(. ; tunnel $str := "Hello")

What about conflict resolution among multiple matching rules? Carrot behaves the same as XSLT: rules with higher import precedence win, followed by rules with higher priority. Default priority is based on the syntax of the pattern, just as in XSLT. You can also specify the priority explicitly (right before the binding operator :=), as in the first rule of this example, which explicitly sets the priority to 1:

^author-listing( author[1]      ) 1 :=           ^();
^author-listing( author         )   := ", "    , ^();
^author-listing( author[last()] )   := " and " , ^();

Ruleset invocations

Ruleset invocations (i.e., "apply-templates" in XSLT) are largely Carrot's raison d'etre. They are not possible in XQuery; thus, the extension is required. Not only that, but XSLT can't invoke rules (apply templates) in an expression either. In Carrot, all definitions are bound to an expression, so the only way to "do" anything is to write an expression. (Unlike XSLT, Carrot does not make a distinction between "instructions" and "expressions"; everything is an expression.)

Shallow copy constructors

Shallow copy constructors are possible in XSLT but not XQuery. The difference between a copy constructor and using an XQuery element constructor is that, in the latter case, the namespace context comes from the query rather than the source document. XQuery allows you to perform deep element copies from the source document, but not shallow copies. Without this ability, modified identity transforms are impractical in XQuery. The semantics of Carrot's copy constructor are essentially the same as XSLT's <xsl:copy> instruction. For example, when the context node is not an element node, it behaves the same as if a deep copy were being performed.

Text node literals

Carrot also adds text node literals, using the back-tick (`) for the delimiter. This extension may at first seem to be of minimal value, since XQuery already allows you to construct text nodes using text{…}, and strings using quotes (or apostrophes). However, in practice, text node literals will often be the preferred syntax, as the following examples should make clear. Consider the following template rules in XSLT:

<xsl:template mode="file-name" match="doc">doc</xsl:template>
<xsl:template mode="file-ext" match="doc">.xml</xsl:template>
 
<xsl:template match="/doc">
  <result>
    <xsl:apply-templates mode="file-name" select="."/>
    <xsl:apply-templates mode="file-ext" select="."/>
  </result>
</xsl:template>

In Carrot, you might naturally rewrite the above as follows:

^file-name(doc) := "doc";
^file-ext (doc) := ".xml";
^(/doc)         := <result>{ ^file-name(.), ^file-ext(.) }</result>

The problem is that this will produce an undesired result:

<result>doc .xml</result>

The extra space results because of the way in which sequences of atomic values are combined to make a text node in XQuery. Contiguous sequences of text nodes, on the other hand, are merged together without any intervening spaces, so you could fix things by using explicit text node constructors:

^file-name(doc) := text{"doc"};
^file-ext (doc) := text{".xml"};

The problem here is that it may be an edge case with a large syntactic cost if you want to cover your bases (six extra characters for every text node). If in 90% of cases, using a string will result in the exact same behavior as if you had used a text node, you will be strongly tempted as a user to use quotes instead of text{…} everywhere. However, you will get bugs in the remaining 10% of your code because of the way sequences of strings are concatenated to make a text node in XQuery.

Whereas it's more verbose in XQuery to construct a text node (using text{…}) than it is to return a string (using quotes), it's more verbose in XSLT to return a string (using <xsl:sequence>) than it is to return a text node (using a literal text node in the stylesheet). Text node literals in Carrot address this imbalance by making it equally convenient to create text nodes and strings. Thus, we naturally rewrite our Carrot definitions to get the desired result, without having to think about whether this is an edge case or not:

^file-name(doc) := `doc`;
^file-ext (doc) := `.xml`;

The existence of text node literals makes it easy to follow a simple rule: use text node literals when you are constructing part of a result document; use string literals when you know you want to return a string.

Expression semantics

Expressions in Carrot, unless otherwise noted here, are assumed to have the same semantics as in XQuery. Carrot operates on exactly the same data model as XQuery 1.0 and XPath 2.0.

One exception is that namespace attribute declarations on element constructors in Carrot do not affect the default element namespace for XPath expressions. Carrot is more like XSLT in this regard, in that it makes a distinction between the default namespace for input documents and the default namespace for output documents ("xpath-default-namespace" in XSLT), thereby correcting what is arguably a design bug in XQuery.

What about xsl:for-each, xsl:for-each-group, etc.?

Given that XQuery expressions do not include everything that it's possible to do in an XSLT template rule, that begs the question: What do all the XSLT instructions get mapped to in Carrot? In many cases, Carrot simply does not have an analogue. In some cases, that's because XQuery already provides a different way to achieve the same use case. For example, <xsl:for-each> does not have a direct analogue in Carrot. For iteration over a sequence, you can use "for" expressions, or even just "/" when applicable. The following Carrot (and XQuery) expression constructs a new <bar> element for each <foo> element, rendering <xsl:for-each> unnecessary for this case: foo/<bar/>. Similarly, Carrot does not support <xsl:sort>. For sorting sequences in Carrot, you would instead use "order by", as in XQuery. Local variables are defined using "let" expressions. Etc.

The biggest area not currently addressed by Carrot—and which remains an open question—is how to perform grouping. There are a few answers to this question, not all mutually exclusive:

At this stage, the operative answers to this question are #2 and #3.

Designing support for multiple output documents (corresponding to <xsl:result-document> in XSLT) and how it interacts with document{} node constructors is on my TODO list. (If you have ideas, I'd be happy to hear them.)

Simple mapping operator

I think XPath needs a "simple mapping operator" that behaves similarly to "/" except without its restrictions and special behavior with regard to node sequences. This is one possible extension that could be added to Carrot, without having to wait for XSLT/XQuery 3.0 (if it's even being considered for inclusion).

Mode merging

Another more recent idea (which would be straightforward to implement in Carrot) would be "mode merging."

In XSLT, a single template rule can declare itself to be a part of more than one mode. However, a single call to apply-templates cannot invoke rules in more than one mode. The ability to merge modes would provide a static mode extension mechanism, the chief benefit of course being that you wouldn't have to go add a new mode to each template rule's list of modes (and in the case when it's in the default mode, go add mode="#default new-mode" to each rule).

In XSLT:

<xsl:apply-templates mode="foo bar"/>

In Carrot:

^foo|bar()

This would be especially handy in multi-stage transformations where each stage of processing makes an incremental change to its input, but some stages need to handle things slightly differently, for example, to avoid transforming an already-converted element more than once. Mode merging would allow you to invoke statically determined subsets and supersets of rules.

Underlying language development

Finally, Carrot is a project that can grow with the languages it is based on. As various features are added in XSLT/XQuery 3.0, such as JSON support or the ability to apply templates to sequences of atomic values, Carrot will (happily) be updated accordingly.