Balisage Paper: Preprocessing XQuery using Custom Module URI Resolvers

Implementation of modifying module URI resolver

I implemented such a modifying module URI resolver with Saxon. Saxon expects a module URI resolver to provide a resolve() method that takes a module namespace URI, a base URI, and an array of location URIs and return an array of StreamSource objects. Each StreamSource hands an input stream or a stream reader to the processor for it to parse.

There a few fine points and choices to consider in the implementation.

I implemented my resolver by subclassing StandardModuleURIResolver. The resolve method looks something like this:

// Resolve the module URI
public StreamSource[] resolve(
  String moduleURI,
  String baseURI,
  String[] locations
) throws XPathException {
  StreamSource[] result = super.resolve(moduleURI, baseURI, locations);
  for (int i = 0; i < locations.length; i++) {
    int index = locations[i].indexOf('?');
    if (index >= 0) {
      String parameterString = locations[i].substring(index + 1);
      String[] parameters = parameterString.split("&");
      HashMap<String,String> mapping = new HashMap<>();
      boolean hasParameters = false;
      for (int j = 0; j < parameters.length; j++) {
        int pindex = parameters[j].indexOf('=');
        if (pindex >= 0) {
          try {
            String pname = URLDecoder.decode(
              parameters[j].substring(0, pindex), "UTF-8"
            );
            String pvalue = URLDecoder.decode(
              parameters[j].substring(pindex + 1), "UTF-8"
            );
            mapping.put(pname, pvalue);
          } catch (Exception e) {
            // Convert URI exceptions into XPathException
            throw new XPathException(e);
          }
          hasParameters = true;
        }
      }
      // Optimization: just use base stream if no patterns
      if (hasParameters) {
        InputStream istream = result[i].getInputStream();
        if (istream == null) {
          // Try without query stream: for file: URIs this helps
          String[] bareLocations = new String[]{locations[i].substring(0, index)};
          StreamSource[] bareResult = super.resolve(moduleURI, baseURI, bareLocations);
          istream = bareResult[0].getInputStream();
        }
        if (istream != null) {
          InputStream stream = new TemplateFilterInputStream(istream, mapping);
          String systemId = result[i].getSystemId();
          String publicId = result[i].getPublicId();
          result[i] = new StreamSource(stream);
          // Carry over public and system IDs
          result[i].setPublicId(publicId);
          result[i].setSystemId(systemId);
        } // null stream; continue
      }
    }
  }
  return result;
}
        

The standard resolver is used when there are no query parameters on the location URI. Otherwise the parameters are parsed to create a mapping of patterns to replacements which are applied by a special stream class. The bulk of the work is done within the new TemplateFilterInputStream class. Here is what that looks like:

public class TemplateFilterInputStream extends InputStream {
  public TemplateFilterInputStream(
    InputStream stream,
    HashMap<String,String> mapping
  ) {
    // Construct a BufferedReader so we can read line by line
    _input = new BufferedReader(new InputStreamReader(stream, _encoding));
    _mapping = mapping;
  }

  // Read and process the next line
  private boolean nextLine() throws IOException {
    String line;
    while ((line = _input.readLine()) != null) {
      // Apply replacements
      line = filterLine(line);
      if (line != null) {
        // Keep line-numbering consistent with source
        line = line + '\n';
        _buffer = new ByteArrayInputStream(line.getBytes(_encoding));
        return true;
      }
    }
    return false;
  }

  // Replace patterns in the line
  private String filterLine(String line) {
    for (Map.Entry<String,String> e : _mapping.entrySet()) {
      line = line.replace(e.getKey(), e.getValue());
    }
    return line;
  }

  // Read next character
  @Override
  public int read() throws IOException {
    // Refill buffer as needed
    if (_buffer == null) { if (!nextLine()) return -1; return read(); }
    int ch = _buffer.read();
    if (ch == -1) { if (!nextLine()) return -1; return read(); }
    return ch;
  }

  // Make sure reader is closed
  @Override
  public void close() throws IOException { _input.close(); }

  private HashMap<String,String> _mapping;
  private BufferedReader _input;
  private ByteArrayInputStream _buffer;
  private Charset _encoding = StandardCharsets.UTF_8;
}
        

We read from a BufferedReader line by line, applying the replacements in the filterLine() method.

Given the new resolver class, making use of it is easy: add the relevant classes to the class path and configure Saxon to use it. Configuration is also easy: either reference the class name in the configuration file as the value of the moduleUriResolver attribute for the xquery element, or set it on the command line with the -mr option.

Similar kinds of APIs are available with some other processors.

Other Modifying Module URI Resolvers

Simple string substitution is easy to implement, but there are other possibilities, with different strengths and weaknesses. I have experimented with implementations of many of these variants.

Templating

External variables in modules allow us to parameterize the behaviour of an XQuery module. But XQuery provides no mechanism for overriding the default value of an external variable. XQuery processors provide such mechanisms. In Saxon, an external variable can be set from the command line. With the modifying module URI resolver, we can set it from the import line and create a variant of the module with the appropriate value, or different values in different contexts.

import module namespace ex="http://example.com/test/external#3"
       at "test/external.txqy?«max»=3";
        

module namespace this='http://example.com/test/external#«max»;
declare variable $this:max as xs:integer := «max»;

declare function this:hello($n as xs:integer) as xs:string {
  if ($n > $this:max)
  then error(QName('http://example.com/errors','BAD-ARGS'), 'n is too big')
  else (),
  'Hello '||$n||' times!'
};
        

A common purpose for macro preprocessing in languages that have the capability is to enable or disable debugging code. The idea is to remove the code entirely, not just make it conditional on some debugging variable. In this example new character sequences (<~ and ~>) delimit the conditional code. Applying a mapping to spaces enables the code; applying a mapping to comment delimiter sequences comments it out.

import module namespace ex="http://example.com/test/conditional"
       at "test/conditional.xqy?%3C~=%28:&~%3E=:%29";
       (: <~ == comment start; ~> == comment end :)
ex:hello(4)
        

import module namespace ex="http://example.com/test/conditional"
       at "test/conditional.xqy?%3C~=%20&~%3E=%20";
       (: <~ == space; ~> == space :)
ex:hello(4)
        

module namespace this='http://example.com/test/conditional';
declare variable $this:max external as xs:integer := 3;

declare function this:hello($n as xs:integer) as xs:string {
  <~
  if ($n > $this:max)
  then error(QName('http://example.com/errors','BAD-ARGS'), 'n is too big')
  else (),
  ~>
  'Hello '||$n||' times!'
};
        

A less dangerous alternative is to use specially marked comments instead of new syntax. In this example I use << and >> as markers for these special comments. Mapping the full special comment sequences to spaces enables the debugging code which is otherwise inside a normal XQuery comment and is therefore ignored.

module namespace this='http://example.com/test/conditional';
declare variable $this:max external as xs:integer := 3;

declare function this:hello($n as xs:integer) as xs:string {
  (:<<
  if ($n > $this:max)
  then error(QName('http://example.com/errors','BAD-ARGS'), 'n is too big')
  else (),
  >>:)
  'Hello '||$n||' times!'
};
        

import module namespace ex="http://example.com/test/conditional"
       at "test/conditional.xqy?(:%3C%3C=%20&%3E%3E:)=%20";
       (:  (:<< == space; >>:) == space :)
ex:hello(4)
        

Aesthetic modifications

I came across someone suggesting this trick with the C++ macro preprocessor: defining certain Unicode box characters in such as way that code nesting could be represented as boxes. The trick doesn't actually work: box characters are not valid in identifiers, which the C++ macro preprocessor requires. However, we can play this trick with the modifying module URI resolver, because the pattern is not required to be an identifier.

import module namespace ex="http://example.com/test/documentation"
       at "test/documentation.txqy?╔=(:&╝=:)";
        

╔══════════════════════════════════════════════════════════════════╗
║ Experiment with resolver-based modifications.                    ║
║ @since March 2025                                                ║
║ @custom:Status Bleeding edge                                     ║
╚══════════════════════════════════════════════════════════════════╝
module namespace this="http://example.com/test/documentation";

╔══════════════════════════════════════════════════════════════════╗
║ Compute the dot product of two vectors.                          ║
║                                                                  ║
║ @param $p1: one vector                                           ║
║ @param $p2: another vector                                       ║
║ @return p1·p2                                                    ║
╚══════════════════════════════════════════════════════════════════╝
declare function this:dot($p1 as xs:double*, $p2 as xs:double*) as xs:double
{
  fn:sum(
    this:map2(
      function ($a as xs:double, $b as xs:double) as xs:double {$a * $b},
      $p1, $p2
    )
  )
};
        

In C++ and languages with similar macro preprocessors, macros can take parameters of sorts that define functional forms. With simple string substitution we can't really accomplish that, but if the module URI resolver performs regular expression matching and substitutions, we can, albeit with a certain amount of struggle. Escaping all the regular expression metacharacters becomes a real chore and the result is an unreadable and somewhat brittle mess. As nasty as this regular expression is, even unescaped, it can fail to properly match when a complex expression is used in the module text.

import module namespace ex="http://mathling.com/test/regex"
       at "test/rex.txqy?ceil_div%5C(([%5E,%5C)%5Cs]%2B),%5Cs?([%5E%5Cs%5C)]%2B)%5C)=($1%20%2B%20$2%20-%201)%20idiv%20$2";
       (: ceil_div\(([^,\)\s]+),\s?([^\s\)]+)\)=($1 + $2 - 1) idiv $2 :)

ex:test(10, 2)
        

module namespace this='http://mathling.com/test/regex';

declare function this:test(
  $a as xs:integer,
  $b as xs:integer
) as xs:integer
{
  ceil_div($a,$b)
};
        

Support for function-like macros more directly would require more work: both the patterns and the module text would need to be parsed to properly identify the parameter bindings, which would then need to be presubstituted.

Just defining a new function seems a whole lot easier.

Syntax Augmentation

Modifying the language syntax is a little dangerous. Let us live more dangerously still, and create custom operator symbols. Here I am using the symbol □ (U+25A1) to stand in for a function call to construct a square, ○ (U+25CB) to stand in for a function to construct a circle, and ↗ (U+2197) to stand in for a mapped function that performs a translation affine transformation.

□ == geom:box(0,0,10,10)     %E2%96%A1=geom:box(0,0,10,10)
○ == geom:circle((0,0),10)   %E2%97%8B=geom:circle((0,0),10)
↗ == =>geom:translate        %E2%86%97=%3D%3Egeom:translate"
        

module namespace this="http://example.com/test/geometric";
import module namespace geom="http://example.com/geometric"
       at "../geo/euclidean.xqy";

declare function this:shapes(
  $location as xs:double*,
  $scaling as xs:double
) as map(xs:string,item()*)*
{
  (□,○)=>geom:scale($scaling)↗($location)
};
        

Still, part of the point might be to experiment with new syntax. If we can try out new operator symbols before cracking open our favourite XQuery processor and doing a lot of complex implementation work, so much the better.

A more compelling example of using alternative symbols is for localization of an API. (See Swidan23 for a fascinating discussion of programming language localization in general.) Suppose we have some standard library, but want to make it more accessible to non-English speakers. Users of the library can operate more comfortably when the names are mapped. Such a use case is best served using configured mappings that apply universally. The imported module is the same in either case, so maintenance is not complicated by having multiple localized versions.

:cross => :перекрестный
:rotate => :вращать
...
        

let $vec := (1.0, 2.3, 4.0)=>v:cross((2.0, 1.0, -4.0))
let $angle := math:pi() div 3
return $vec=>v:rotate($angle)
        

let $век := (1.0, 2.3, 4.0)=>в:перекрестный((2.0, 1.0, -4.0))
let $угол := math:pi() div 3
return $век=>в:вращать($угол)
        

Implementation Accommodation

When standards evolve, sometimes there are lags in which parts of that standard an implementation supports. Sometimes implementations get ahead of things, implementing something that isn't final and gets changed later. Perhaps a particular processor doesn't support a particular function or operator. Perhaps the specification isn't even close to final. The module rewrite can fix that. For example, the mapping %C3%B7=%20div%20 replaces the proposed ÷ operator symbol (XQuery 4.0) with the older div. The name get-years-from-yearMonthDuration (early XQuery draft) can be replaced by years-from-yearMonthDuration (final Recommendation). The processor specific xdmp:function-name can be replaced by fn:function-name. And so forth.

Testing

One class of use case for this pattern/replacement paradigm is to perform quick tests of changes before rewriting a raft of code. The intention is not to have a more permanent kind of change as with the templated vector example, but to just try things out in a lightweight fashion. In Figure 16 the rewrite rules modify the code for a data model change.

import module namespace ex="http://example.com/test/model"
       at "test/model.txqy?/root=/top/root";

ex:things(document { <top><root><thing>Thing</thing></root></top> } )
        

module namespace this="http://example.com/test/model";

declare function this:things($doc as document-node()) as xs:string
{
  string-join($doc/root/thing, " ")
};
        

In a similar vein, we can test out a replacement for some old function before we edit a bunch of code. Or we could have multiple alternative implementations, suitable for different environments, and select which one to use. In Figure 17 the function name ex:hello is replaced by the version we mean to try out.

import module namespace exc="http://example.com/test/api/caller"
       at "test/apicaller.txqy?:hello=:simple-hello";

exc:test()
        

module namespace this="http://example.com/test/api/caller";
import module namespace ex="http://example.com/test/api"
       at "test/api.txqy";

declare function this:test() as xs:string
{
  ex:hello(3)
};
        

module namespace this="http://example.com/test/api";

declare function this:simple-hello($n as xs:integer) as xs:string
{
  "Hello "||$n||" times!"
};

declare function this:multiple-hello($n as xs:integer) as xs:string
{
  string-join(
    for $i in 1 to $n return "Hello", " "
  )
};
        

In Figure 18 we are again replacing one function with another. Here we are looking at a particular annotation on the function (ann:deprecated) and using it as a trigger to change the name of the function to something else, in order to force an error. With simple string substitution this kind of replacement depends a great deal on certain coding conventions.

import module namespace ex="http://example.com/test/deprecation"
       at "test/deprecation.txqy?%25ann:deprecated%20function%20this:=function%20this:deprecated-";

ex:new-hello(10),
ex:old-hello(10)
        

Readability and sharing

Code will work incorrectly or even fail to parse unless you use the right module resolver. The more you monkey with syntax, the more completely you depend on it or on the "compiled" version of things. Other tools won't understand special symbols or unsubstituted values. For example, part of my toolchain relies on XQDoc to process structured comments. It does not use a resolver at all, and fails to parse modules with extended syntax.

Even if a debugging tool can use the modifying resolver, without being able to see the unescaped mappings or the expanded text, debugging can become quite difficult. If the modifications change line locations, error messages will point to the wrong parts of the code.

Idiosyncratic syntax extensions may not be comprehensible to other people. For example, the special operators in Figure 14 will not make sense without additional information.

Security issues

Code injection is a live concern. Here a substitution is used to reference a perfectly ordinary XQuery module and use it to exfiltrate data:

import ex="http://example.com/example" at "anodyne.xqy?};=,unparsed-text('/secret.txt')};";
        

The billion laughs attack can be used to perform a denial of service attack on the processor, as long as expansions happen in the right order.

Mappings in location URI:

«lol9»=«lol8»«lol8»«lol8»«lol8»«lol8»«lol8»«lol8»«lol8»«lol8»«lol8»
«lol8»=«lol7»«lol7»«lol7»«lol7»«lol7»«lol7»«lol7»«lol7»«lol7»«lol7»
«lol7»=«lol6»«lol6»«lol6»«lol6»«lol6»«lol6»«lol6»«lol6»«lol6»«lol6»
«lol6»=«lol5»«lol5»«lol5»«lol5»«lol5»«lol5»«lol5»«lol5»«lol5»«lol5»
«lol5»=«lol4»«lol4»«lol4»«lol4»«lol4»«lol4»«lol4»«lol4»«lol4»«lol4»
«lol4»=«lol3»«lol3»«lol3»«lol3»«lol3»«lol3»«lol3»«lol3»«lol3»«lol3»
«lol3»=«lol2»«lol2»«lol2»«lol2»«lol2»«lol2»«lol2»«lol2»«lol2»«lol2»
«lol2»=«lol1»«lol1»«lol1»«lol1»«lol1»«lol1»«lol1»«lol1»«lol1»«lol1»
       

Module content:

element content { «lol9» }
       

Interference with module caching

Module identity is by namespace, so unless the namespace of the imported module is also modified, all modifications coming from the query parameters of the location will be baked in from first use. Consider the module from Figure 9 but without the numeric addendum to the namespace URI, so that it is just "http://example.com/test/external".

Suppose this module gets imported with different mappings for «max», say, once directly as 3 and once indirectly through some other imported module as 5. The first copy seen will be cached by Saxon and used for both variants. So the effect will be either as if the max variable was set to 3 in both cases, or as if it were set to 5 in both cases. If we need to have both variants in the same program, we need to also adjust the module namespace URI as we did in Figure 9

Misnesting

Since the substitutions are just applying to text, they do not obey any rules of program structure. They can include separators such as commas, which can make for some strange looking source code.

The mapping includes a stray comma:

import ex="http://example.com/example"
       at "ex.txqy?FORMAT='[Y0001]-[M01]-[D01] [H01]:[m01][Z01]', ";
      

The module source looks weird:

format-dateTime(current-dateTime(), FORMAT "en","AD","US")
      

But the expansion is valid:

format-date-time(current-dateTime(), '[Y0001]-[M01]-[D01] [H01]:[m01][Z01]', "en","AD","US")
      

Extra separators can make the code incorrect. Here the replacement introduces an extra comma:

PAIR=$data("a"),$data("b")
      

max(PAIR)
      

max($data("a"),$data("b"))
      

It is all rather unfortunate.

Operator precedence

Textual substitutions into expressions, if not handled with care, can run afoul of operator precedence rules. In XQuery the union operator | has lower precedence than the child path operator /. If we substitute in an expression using the union operator in a context with the path operator, results may not be what we expect.

Suppose we expand the following mappings in order:

TAG=((P)/c/@tag)
(P)=a|b
      

The source text looks fine:

$root/TAG(p)!string(.)
      

The expansion will give us:

$root/((P)/c/@tag)!string(.) => $root/(a|b/c/@tag)!string(.)
      

It is unlikely this is what was intended. More likely we meant:

$root/((a|b)/c/@tag)!string(.)
        

This kind of example is more plausible with function-style macros. The mapping would be:

TAG(P)=(P/c/@tag)
      

The source would then provide the parameter:

$root/TAG(a|b)!string(.)
      

The source still looks OK, but the expansion is as before, with an unintended result.

Name clashes

Ill-advised match patterns can create name collisions in the XQuery text. This may cause syntax errors or incorrect code.

Here someone has attempted to create a shorter name for a function. They even attempted to limit the applicability with parentheses.

empty()=ex:set-matrix-all-to-zeroes()
      

It will end in tears, however, because of the clash with a standard built-in XQuery function in an expression like this:

if ($seq=>empty()) then ...
      

Similarly, binding of variables in the replacement text can create unintentional variable shadowing.

For example, this function code snippet looks like it is summing the numbers from 3 to 12, but with the mapping below it is actually summing the numbers from 1 to 10.

let $start := 3
let $end := 12
return «init»sum($start to $end)
      

Mapping causing the trouble:

«init»=let $start := 1 let $end := 10 return
      

Duplication of work

In C++ with function-style macros there is the danger of unexpected duplication of side-effects. In a functional language like XQuery we don't have to worry about that, although duplication of effort could still be an issue.

Suppose we have a macro defined as follows:

square(x)=x*x
      

square(ex:complex-calculation($data))
      

The substitution will result in the program text of:

ex:complex-calculation($data)*ex:complex-calculation($data)
      

While this will produce the correct result, if ex:complex-calculation takes a long time to compute, we may end up executing it twice, unless the processor optimizes the second call away.