Balisage Paper: Type Introspection in XQuery

Type Tests for Type Values

For each kind of component value there is a sequence type test that matches values of that kind. Since types might have names, sequence type tests for type values can have a QName and will match if either the type has that name or it has a base type that matches.

Obtaining Type Values

Linkage functions get component values from various kinds of items. They are defined over all kinds of items, returning the empty sequence if necessary. These functions all take any item as a parameter and have a zero-argument form that applies to the context item. This is a small usability feature, but makes such expressions as /root/child/sc:simple-type() possible.

In this paper, the schema component linkage and accessor functions are all defined as being in the namespace bound to the prefix sc which the examples assume to be predefined. The specific namespace chosen is not important. The experimental implementation used http://marklogic.com/xdmp/schema-components.

XQuery introduces an interesting wrinkle that is not found in most other programming language type systems: some data may be untyped. The special types xs:untyped and xs:untypedAtomic represent such untyped complex and simple content.

For atomic values that are instances of a particular simple type, the sc:simple-type linkage function returns that simple type as a component value. If the atomic type is untyped, then a simple type definition value for xs:untypedAtomic is returned. The accessor sc:simple-type operates on attribute nodes in a similar fashion. The operation on element nodes is similar, except that if the element is an instance of a complex type, sc:simple-type returns the empty sequence. For other kinds of items, the empty sequence is returned.

The linkage function sc:complex-type works analogously like sc:simple-type except atomic values and attributes can never be instances of a complex type, so sc:complex-type is only useful for element nodes. If an element node is an instance of a specific complex type, the value will be that complex type as a component value. If the element node is untyped, xs:untyped will be returned as a complex type component value. When an element is an instance of a simple type, sc:complex-type returns xs:anyType as a complex type component value. For other items, the empty sequence is returned.

The sc:type accessor combines certain aspects of the sc:simple-type and sc:complex-type accessors. It returns a type definition of the appropriate kind. For untyped element nodes, xs:anyType is returned as a type definition value.

Functions and Operators on Type Values

Obtaining types as values is well and good, but what can we do with them other than perform sequence type tests on them? At a minimum, equality operators should work on these values: equals, =, and !=. It is also useful to be able to test whether one type is a subtype of another.

declare function my:spaceship-operator ($t1 as type(), $t2 as type()) 
{
  if ($t1 subtype of $t2) then -1
  else if ($t1 = $t2) then 0
  else if ($t1 subtype of $t2) then 1
  else xs:double("NaN")
};
       

XQuery F&O30 defines a large number of functions. However, only a few apply to all items. The sequence functions, such as fn:subsequence can work on sequences of component values with no problem. The general functions fn:string, fn:boolean and fn:not don't really apply and will raise errors if applied to component values.

If we have equality on type component values, then not only can we define fn:deep-equal over them, we can define fn:deep-equal with them. Figure 9 shows a version of deep-equal that is stronger than that defined by XQuery: it requires type equality as well as structural equality.

declare function my:deep-equal ($p1 as item()*, $p2 as item()*, $coll as xs:string) 
as xs:boolean
{
   let $c1 := count($p1)
   let $c2 := count($p2)
   return
   if ($c1 != $c2) then false()
   else if ($c1 != 1) then 
     every $i1 at $i in $p1 satisfies my:deep-equal($i1, $p2[$i], $coll)
   else if (exists(sc:simple-type($p1)) and exists(sc:simple-type($p2))) then
     sc:simple-type($p1) eq sc:simple-type($p2) and
     if (($p1 instance of xs:string) and ($p2 instance of xs:string))
     then compare($p1, $p2, $coll) eq 0
     else
     try {
       let $v := $p1 eq $p2
       return ($p1 eq $p1) or (($p1 ne $p1) and ($p2 ne $p2)) (: NaN :)
     } catch  * {
       false()
     }
   else typeswitch ($p1)
   case document-node() return 
      (typeswitch ($p2) 
       case document-node() return true() default return false()) and
      my:deep-equal($p1/(*|text()), $p2/(*|text()), $coll)
   case element() return
      (typeswitch ($p2) 
       case element() return true() default return false()) and
      node-name($p1) eq node-name($p2) and
      exists(sc:type($p1)) and exists(sc:type($p2)) and 
      sc:type($p1) eq sc:type($p2) and
      my:deep-equal( for $a1 in $p1/@* order by name($a1), 
                     for $a2 in $p1/@* order by name($a2), $coll ) and
      if (exists(sc:simple-type($p1)))
      then my:deep-equal(data($p1), data($p2), $coll)
      else my:deep-equal($p1/(*|text()), $p2/(*|text()), $coll)
   case attribute() return
      (typeswitch ($p2) 
       case attribute() return true() default return false()) and
      node-name($p1) eq node-name($p2) and 
      my:deep-equal(data($p1),data($p2),$coll)
   case processing-instruction() return
      (typeswitch ($p2) 
       case processing-instruction() return true() default return false()) and
      node-name($p1) eq node-name($p2) and 
      compare(string($p1), string($p2), $coll)
   case namespace-node() return
      (typeswitch ($p2) 
       case namespace-node() return true() default return false()) and
      my:deep-equal(node-name($p1), node-name($p2), $coll) and
      compare( string($p1), string($p2), $coll ) eq 0
   case text() return
      (typeswitch ($p2) case text() return true() default return false()) and
      compare( string($p1), string($p2), $coll ) eq 0
   case comment() return 
      (typeswitch ($p2) case comment() return true() default return false()) and 
      compare(string($p1), string($p2), $coll)
   case function() return error("err:FOTY0015")
   case component() return $p1 eq $p2
   default return false()
};
      

Once types are available as values, they can be returned in place of type names to perform introspection on functions, for example.

declare function local:check($old as item(), $new as item())
{
   let $f := $old/sc:type()
   return sc:type-apply($f,$new)
};

local:check(doc("config.xml")/config/language, "en"),
local:check(doc("config.xml")/config/encoding, "utf-8")
        

declare function local:check($old as item(), $new as item())
{
   let $f := $old/sc:type()
   return $f($new)
};

local:check(doc("config.xml")/config/language, "en"),
local:check(doc("config.xml")/config/encoding, "utf-8")
        

declare my:run-plugin($config as element(), $plugin as function(*), $args as item()*)
{   
   let $expected := sc:type-named(data($config))
   if (sc:return-type($plugin) = $expected)
     $plugin($args)
   else 
     $expected($plugin($args))
};
      

<xs:simpleType name="address">
  <xs:annotation>
    <xs:documentation>
      A numeric internet address.
    </xs:documentation>
  </xs:annotation>
  <xs:restriction base="xs:token">
    <xs:pattern value="[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+"/>
  </xs:restriction>
</xs:simpleType>

<xs:element name="address" type="address">
  <xs:annotation>
    <xs:documentation>
      The server socket bind numeric internet address.
    </xs:documentation>
    <xs:appinfo>
      <admin:default>"0.0.0.0"</admin:default>
      <admin:cold-restart/>
      <admin:help xmlns="http://www.w3.org/1999/xhtml">
      <span class="help-text"><span style="color:#800000">address</span> 
      specifies the IP address for the App Server.</span>
      </admin:help>
    </xs:appinfo>
  </xs:annotation>
</xs:element>

        

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
           xmlns:record="http://example.com/meta/records"
           xmlns:base="http://example.com/meta"
           targetNamespace="http://example.com/meta/records"
           elementFormDefault="qualified">

<xs:import namespace="http://example.com/meta"
           schemaLocation="meta.xsd"/>

<xs:simpleType name="studio">
  <xs:annotation>
    <xs:appinfo>
      <base:input-handler>record:normalize-studio</base:input-handler>
      <base:formatter>record:format-studio</base:formatter>
    </xs:appinfo>
  </xs:annotation>
  <xs:restriction base="xs:string"/>
</xs:simpleType>

<xs:complexType name="record-metadata">
  <xs:complexContent>
    <xs:extension base="base:metadata">
      <xs:sequence>
        <xs:element ref="studio" type="studio"/>
      </xs:sequence>
    <xs:extension>
  </xs:complexContent>
</xs:complexType>
    

declare function base:format-metadata ($metadata as element())
{
  for $meta in $metadata/*
  let $methods := $meta/sc:element-decl()!sc:annotations()/xs:appinfo
  let $formatter := function-lookup($methods/base:formatter, 1)
  return $formatter($meta)
};
    

Facets

Accessing facet information is important for working with simple types. Figure 18 summarizes the additional type tests and accessors for working with facets. The sc:facets accessor returns sequences of facet component values when applied to simple type components; it returns the empty sequence otherwise. The accessors already defined apply to facet component values as well. The sc:name accessor returns the name of the element used to define the facet (e.g. xs:enumeration for the enumeration facet), and the standard accessor fn:data returns the value of the facet.

Let's return to the date example from the introduction (Figure 2). With access to types and their facets, as well as function lookup and function application, we can create a generic framework for processing dates without having to hardcode any type or function names. A fragment of this framework is shown in Figure 19.

...
declare function my:process-date($date)
{
  for $date-type in fn:collection("templates")/date/sc:simple-type() 
  let $date-pattern := sc:facets($date-type)[. instance of facet(xs:pattern)]
  let $process-func := 
     fn:function-lookup(fn:data(sc:annotations($date-type)//my:date-func), 1)
  where fn:exists($date-pattern) and
        fn:exists($process-func) and
        fn:matches($date, fn:data($date-pattern))
  return $process-func($date-type($date))
};

      

Element and Attribute Declarations

As the example in Figure 16 shows, it is frequently more natural to obtain the annotations from element and attribute declarations rather than from the types associated with those declarations, Annotations often have more to do with the use of a type in a particular context than with the type itself. Adding accessors to obtain element and attribute declarations from element and attribute nodes, and defining the component accessors over these new component types allows for these use cases. Figure 20 summarizes the additional type tests and accessors for working with element and attribute declarations. The sc:element-decl and sc:attribute-decl accessors return component values when applied to element and attribute nodes, respectively; otherwise they return empty sequence. In addition, the accessors already defined apply to these new declaration component values as well. The sc:type, sc:complex-type, and sc:simple-type accessors return the type associated with the declaration.

Fitting element declaration values into the type hierarchy leads us to go still further: element declarations are but one kind of term in a content model, along with wildcards and model groups. Given that we already have element declaration values, it makes sense to expose the content model of a type in general. Given that we already have attribute declaration values, it makes sense to expose the attributes declared on a complex type.

The XML Schema component model creates some complexity in this area and we need to decide how to expose the information in the a usable way. That model makes a distinction between particles and terms, where the particle carries occurrence information. It also makes an analogous distinction between attribute uses and attribute declarations, where attribute uses carry value constraints. Attribute uses and particles also can be used to carry local context-specific annotations.

In addition to the new accessors listed in Figure 20, the sc:attribute-decl accessor applies to attribute use component values.

Figure 22: Declaration and Particle Values

More Accessors, Functions, and Operators

This paper has focused mainly on exposing types as values in XQuery with a few basic accessors, with some forays into some related schema components. Some meta-programming techniques rely on being able to examine the full richness of the schema component model — terms, particles, attribute uses, schema component affiliations, abstractness, and so on. The program to provide such full access is similar to what was outlined here: define type tests for the component, define linkage functions or accessors to get to that component, define accessors to get at the component properties, and define whatever other useful functions and operators seem appropriate.

More extensive additions to the syntax of XQuery to allow variables to be used as the operand of instance of and cast as expressions would improve usability further:

declare function my:filter-by-type($t as type(), $nodes as node()*)
{
  for $n in $nodes where $n instance of $t return $n
};
       

Relationship to Schema Component Paths

Once we embark on the program of providing full access to all the schema components and their properties, the use of schema component paths arises. Schema component paths define an XPath-like syntax for describing and navigating W3C XML Schema XSD11.1 component models. Abbreviated syntax is defined for some path steps, and certain axes that skip over intermediate components can provide effective ways of navigating through content models, or selecting multiple components with a since path.

Schema component paths navigate through the schema component graph of an assembled schema. As with XPath, each step in the path consists of an axis combined with some kind of test and perhaps with a predicate. In schema component paths, the test is a name test (possibly a wildcarded name test, and possibly a test for an anonymous component), and the only predicate defined is a positional predicate (selecting the Nth of like components). There are no accessors for non-component properties: the axes act as accessor for component properties, just like in XPath. About two dozen axes are defined, for each of the component-to-component relationships in the graph. For example, the annotation:: axis corresponds to the sc:annotations accessor.

/schemaElement::p:outer/type::0/schemaAttribute::p:inner
/p:outer/~0/@p:inner
/type::p:second/model::sequence/schemaElement::p:duplicate[2]/type::*
/~p:second/model::sequence/p:duplicate[2]/~*
      

Figure 23 shows some schema component paths rooted at the component for the schema as a whole. The first path starts at the root of the assembled schema (/). It then traverses the schemaElement axis (schemaElement::) with a name test (p:outer), selecting a global element declaration with the name outer. The path continues through the type axis (type::) with a name test (0) that in this case matches a type definition with no name (0 being the indicator for this case). The path to this point will select the locally declared anonymous type of the element declaration 'outer'. Finally the path concludes by traversing the schemaAttribute axis (schemaAttribute::) with a name test (p:inner), selecting an attribute declaration whose name is 'inner' within the anonymous type definition. As a whole then, this path selects a particular attribute declaration of a particular element declaration. The second path means the same thing, but uses abbreviated syntax.

The third path again starts at the root of the assembled schema (/). This time it traverses through the type axis (type::) with a name test (p:second), thus selecting a global type definition with the name 'second'. The path then traverses the model axis (model::) with a name test (sequence), thus selecting the model group in the content model of the type, but only if it is a sequence. The path continues along the schemaElement axis (schemaElement::) with a name test (p:duplicate) and a positional constraint ([2]), thus selecting an element declaration within the sequence that has a name of 'duplicate', but referring to the second such element within the content model. Finally, the path traverses the type axis (type::) with a wildcard (*), selecting the type of the given element declaration, whatever it might be. The fourth path means the same thing, but uses abbreviated syntax.

How does this compare to the accessors we have defined above?

The accessors operate not on components but on XQuery values. For example, sc:element-decl applies to nodes. Some accessors, such as sc:type, apply to component values also, and so behave somewhat like the schema component axes. To get the full range of axes available in schema component paths, it would be necessary to define an accessor function corresponding to each axis, and allow it to apply to component values. To get the effect of the name tests, XPath predicates must be applied to the results of the accessor. Some accessors return simple properties of the components, such as its name. Schema component paths do not provide for access to the non-component properties. Finally, schema component paths use the slash (/) as a syntactic separator between steps. Since schema components are not nodes, XQuery forbids using them inside a path (although they can be the last step of a path). A FLWOR expression, nested function call, or the XQuery 3.0 simple mapping operator (!) must be used instead.

Since schema component accessors do not apply to schemas in isolation, it is not possible to construct XQuery expressions that are exactly equivalent to the schema component paths given previously. However, we can give some expressions that are roughly equivalent, to give some sense of how they compare.

$context/sc:schema()!
  sc:element-decl()[sc:name(.)=xs:QName('p:outer')]!
    sc:type()[empty(sc:name(.))]!
      sc:attribute-decl()[sc:name(.)=xs:QName('p:inner')]

$context/sc:schema()!
  sc:type()[sc:name(.)=xs:QName('p:second')]!
    sc:particles()[
      typeswitch (sc:term(.)) 
      case model-group(xs:sequence) return true()
      default return false()]!
        sc:particles()[
          typeswitch (sc:term(.)) 
          case particle(p:duplicate) return true() 
          default return false()][2]!
            sc:type()
       

Figure 24 shows XQuery expressions using the accessors that are roughly equivalent to the schema component paths shown in Figure 23. These expressions have to be grounded on some XQuery item ($context), and it was necessary to invent a new accessor to link to the root of the assembled schema governing the type information for that item (sc:schema).

Schema component paths clearly express traversal through a schema component graph more compactly than the accessor and linkage functions do, in that they combine the accessor and the type test in one path step. In addition, schema component paths generally elide the distinction between terms and particles, which the chained accessors do not.

On the other hand, schema component paths do not provide a complete answer, however. They do not provide access to non-component information, nor do they bridge the boundary between schema components and nodes or other item types.

We could imagine allowing schema component paths to mix directly with normal XPath paths to provide that linkage. Then a path such as /book/chapter/title/type::*/annotation::*/xs:appinfo/my:special-stuff/@my:type would switch back and forth between XPaths and schema component paths. Under this scheme the type axis in the fourth step of the path would start a relative schema component path with some component as the default. In this case, the context is the element declaration component for the title element.

Mixing schema component paths with normal XPaths directly in this way is apt to be very confusing, however. The rules governing the slash are very different. In addition, abbreviated schema component paths in this mixed context confuses interpretation tremendously. Is the @my:type at the end of the path a normal attribute node on the my:special-stuff element or is it an attribute declaration? Careful specification of the switching rules may eliminate ambiguities, but it won't eliminate confusion for humans reading and writing such paths. The similarity of schema component paths to XPaths that helps make them more intuitive in isolation causes difficulty when used in the same context as XPath. This confusion also makes implementation more difficult: higher levels of analysis would be required to determine what kinds of operations are allowable, or to optimize node paths properly.

Keeping schema component paths and XPaths clearly separated provides the power of both, without the confusion. Wrapping schema component paths within some kind of scoping expression and providing the rich set of accessor functions seems the best way to achieve this.

/book/chapter/title/sc:element-decl() ! 
  scp {type::*/annotation::*}/
     xs:appinfo/my:special-stuff/@my:type
      

In summary, both schema component paths and the XQuery linkage and accessor functions both have their place. Schema component paths are suited to navigating through schema component graphs or pointing to specific schema components for schema analysis or reporting. The accessor functions are more useful to exposing the schema information relevant to a particular XQuery item in the context of some operations over data.