Balisage Paper: XQuery topic tools - concept, user interface, development framework

Tool invocation model

The invocation of a topic tool specifies a named operation and (optionally) the values of named parameters. The syntax used is the same for any topic tool. Technically speaking, a topic tool has a single parameter named “request”, which is a structured string identifying the operation and supplying parameter values. Operation name and parameters are separated by a question mark, and parameters are separated by a comma. This is the general pattern:

   operationName ? param1=value1, param2=value2

   docs    ? doc=/logs/log01.xml /logs/log02.xml /logs/log03.xml                
   doctype ? docs=^/projects/xsd/xsd-fpml%*                
   names   ? docs=^/projects/xsd/niem-2.1%*
   names   ? docs=^/projects/xsd/niem-2.1%*, scope=elem, names=*augment*

Which operations are available? Given an operation, which parameters are possible and which are mandatory, what is the minimum/maximum number of value items, and which character must be used as item separator? What are the parameter types, and what the default values, if any? The answers are stored in the tool scheme, which models any aspect of the tool interface. Every topic tool supports a _help operation providing usage assistence, which is based on the actual tool scheme. The _help operation of every topic tool supports the same set of parameters. The use of _help is facilitated by a syntactic shortcut: the operation name may be omitted (if parameters are specified) or replaced by a question mark (if no parameters are specified). Invocations #1 and #3, and invocations #2 and #4 are equivalent:

   _help
   _help?ops=paths, default=true, type=true   
   ?
   ?ops=paths, default=true, type=true

Called without parameters, _help displays a simple summary of the topic tool, consisting of operation names and the parameter names and cardinalities:

REQUEST:
?
   
==>
TOOL: xclear

OPERATIONS   PARAMS
====================================================
_help        default, mode, ops?, type
dcat         dfd+
docs         dcat*, doc*, docs*
doctypes     attNames, dcat*, doc*, docs*, elemNames
names        dcat*, doc*, docs*, names?, scope
paths        dcat*, doc*, docs*, paths?
values       dcat*, doc*, docs*, nvalues?
====================================================

To see the types and default values, set the parameters type and default to "true". You can also restrict the report to particular operations by supplying a name filter as value of parameter ops (more about name filters later).

REQUEST:
?type=true, default=true, ops=_h* do*

==>
TOOL: xclear

OPERATIONS   PARAMS
============================================================================
_help        default=false....... : xs:boolean
             mode=overview....... : xs:string
             ops................. : nameFilter?
             type=false.......... : xs:boolean

docs         dcat................ : docCAT* (sep=;)
             doc................. : docURI* (sep=WS)
             docs................ : docDFD* (sep=;)
               At least 1 of these parameters must be set: dcat, doc, docs

doctypes     attNames=false...... : xs:boolean
             dcat................ : docCAT* (sep=;)
             doc................. : docURI* (sep=WS)
             docs................ : docDFD* (sep=;)
             elemNames=false..... : xs:boolean
               At least 1 of these parameters must be set: dcat, doc, docs
============================================================================

Operation and parameter names can be abbreviated by leaving out trailing characters, as long as the used prefix unambiguously identifies the operation or parameter, respectively. Operation and parameter names are case sensitive from the developer perspective, but case insensitive from the user perspective; entered names are normalized to the case defined by the tool scheme.

Another syntactic shortcut applies to parameters of Boolean type. They can be specified by only supplying the name (equivalent to param=true), or the name preceded by a tilde (equivalent to param=false). Example:

?ops=paths, ~default, type

An XQuery based topic tool receives the invocation string as value of the single external variable, named request. Call syntax of course depends on the actual XQuery processor. Using BaseX, for example, calling a topic tool looks similar to this:

   basex -b "request=names ? docs=^/projects/xsd/niem-2.1/*" /projects/xclear/xclear.xq

Tool scheme

A topic tool is defined by a tool scheme which models every aspect of the tool interface. The tool scheme supports both, the tool user and the tool developer in their work. The tool scheme is automatically constructed by collecting annotations contained by the tool sources, augmenting them and integrating them into a single XML document. The scheme specifies:

Every parameter has a type and a cardinality. The type can be a built-in schema type, a ttools-defined type (a generic type defined by the topic tools framework described in this paper, available in any framework-based topic tool) or a tool-specific type. All built-in schema types are supported, excepting xs:ID and xs:NOTATION.

An example of a framework-defined type is docDFD, which represents a filter extracting XML documents from directories. ("DFD" denotes "directory filter descriptor".) An example of a tool-specific type is travelScenario. This type is defined by a topic tool for evaluating touristic data; parameter values are structured strings describing a travel in terms of travel dates and traveller birthdates.

A facet constrains the parameter value beyond the constraints implied by the parameter type. A facet has a facet type and a string value. Each facet type defines a particular type of constraint applying to parameter item values. The following table lists the framework-defined facets, as of writing. Besides these generic facets, topic tools may also define tool-specific facets (see section “Advanced usage - extending the framework”).

The _help operation provides usage assistence, which is based on the tool scheme. An XML representation of the complete tool scheme can be obtained by calling _help with parameter mode set to scheme, for example:

   basex -b "request=? mode=scheme" /projects/xclear/xclear.xq

Object parameter types

The topic tools framework defines several parameter types providing high value to the tool user, as well as to the tool developer. The parameter values are not treated as strings, but as shortcut representations of structured information. The structures obtained from the parameter values are recognized by powerful functions, and the behaviour of these functions conveys to the parameter value a special meaning. For example, when the string

   get* fetch* ~*IFF#c*

is used as argument for a parameter of type xs:string, it is this string what arrives in the tool application. When the string is used as argument for a parameter of type nameFilter, however, the application does not receive a string, but a complex XML element, which contains elements and attributes representing positive and negative string matching conditions, regular expressions and regex options. The element is the XML representation of a filter, selecting names starting with “get” or “fetch” (case-insensitively), but excluding names ending with “IFF” (case-sensitively). The filtering behaviour is provided by functions that "understand" the element structure and to which the filter element is passed as a parameter. These functions and the XML element are of course designed jointly, the element for defining behaviour, and the functions for implementing behaviour. Element and functions should therefore be viewed as a conceptual unit encapsulating data and functionality, irrespective of whether the functions are formally bound to the element, as would only be possible when using an object-oriented language. Parameter types matching this pattern - they are parsed into specific elements which can be handled by specific functions - are therefore collectively called object parameter types. Conceptually, the user supplies objects, rather than strings. The user benefits from the high expressiveness of such parameters – the possibility to convey much information in a very concise way. The developer benefits from the rich functionality immediately available to him. An additional advantage for the user is the stylistic consistency which object parameter types enforce. Dealing with different operations and even with different topic tools, the style of specifying, say, name-based filtering is always the same.

   10 % 0 :  *id *timestamp* % 100 : status

   2.0/*.xsd

   ^/projects/niem-2.1 ^/projects/niem-2.0 /project/niem-submissions % *.xsd

   ^/projects/niem-2.1 ^/projects/niem-2.0 /project/niem-submissions % *.xsd % edxl*//*

   2014-04-02 2014-12-06 14 % 32 31 5 2

Storing and reusing invocation

Complex functionality tends to involve a considerable number of invocation parameters, and object parameter types invite rather complex parameter values. A call with four or more parameters is a bit of a strain, and the more so if they involve object parameters. On the other hand, routine use of topic tools can be fairly repetitive, using often very similar calls which are only distinguished by a parameter or two, whereas several other parameters are set to the same values every time. In such situations in can be helpful to store invocations and reuse them either verbatim or partially overwriting or extending them on a call by call basis. Every topic tool supports the storeq feature: if called with two question marks (instead of a single question mark), the tool returns a simple XML representation of the invocation. After storing the invocation in a file, it can be referenced in future tool invocations, omitting the operation name and using the file name instead, preceded by the character @. An example follows.

Saving an invocation:

   basex –b 'request=priceItems??otds=/projects/otds/otds.xml, cond=occupancy personImpact 
   dayImpact conditionalTags, scope=all, class=*, absolute' otdsv.xq > piStd.tti

Reusing it unchanged:

   basex –b 'request=@piStd.tti' otdsv.xq

Reusing and partially overwriting and extending it:

   basex –b 'request=@piStd.tti?scope=absolute, summary' otdsv.xq

Summary

To summarize, the user of topic tools benefits from several features:

The next section explains how topic tools are developed. It introduces a very lightweight framework which enables the developer to create topic tools quickly, to extend them with minimum effort, and to achieve high quality.

Overview

The development support offered by ttools has four main aspects:

The main module of a topic tool is generated. Code generation is based on annotations scattered over XQuery library modules. Annotations are not formal annotations as defined by XQuery 3.0, but XML snippets placed in XQuery comments conforming to a syntax convention. Annotations typically define tool operations in terms of an operation name and the names, types and other properties of operation parameters. Annotations may also define new parameter types and new kinds of parameter facets.

Code generation is effected by an application build. The build is triggered by a command line call of ttools, which is a topic tool for building topic tools. The build call provides a tool name and a tool directory. During an application build, ttools investigates all library modules found in the application directory, and all annotations which they contain are used. The independent annotations scattered over the modules are integrated into a single model of tool functionality, called the tool scheme. A tool scheme defines tool functionality in terms of named operations and operation-specific sets of parameters.

The tool scheme is translated into an XQuery main module implementing a command line interface which gives access to all operations. On invocation, the main module parses, evaluates and validates user input, creating a comprehensive error report, if errors occur. Otherwise the main module integrates user input into an XML request message and invokes the operation provider function, the function which implements the operation. The provider function is hand-written by the tool developer. It has a standardized signature, receiving a single parameter which is the request message assembled by the main module, and producing an annotation-defined result type. The request message is an XML element intended to be used exclusively via a set of access functions designed for the purpose. These access functions can be regarded as the message interface, functionality bound to its data content. The request message thus resembles a program object, which hides user input data behind an interface of accessors to input information.

For example, if a parameter is typed as a document URI, the user data is a URI, but the accessor returns a document node. While the user supplies a multiple-valued parameter as a concatenated string, the accessor delivers the parameter as a sequence of typed items. Annotation-defined default values are delivered in a way indistinguishable from explicitly supplied parameters. The difference between input data and input information is especially relevant in the case of object parameters (e.g. name filters), where input data are strings from which “objects” are constructed: XML elements designed to provide functionality when passed to the appropriate functions (e.g. tt:matchesNameFilter). It is important to realize how the use of the request object and its accessor interface in general, and the use of object parameters in particular, may simplify the work of application writing significantly.

ttools based development profits from utility functions in a dual way: (a) using “object functions”, as just described, which are functions designed for use with XML elements constructed by ttools infrastructure; (b) general purpose functions, providing useful functionality not related to ttools specific structures. Examples of general purpose functions are a function for parsing CSV formatted text and a function for creating tabular text.

Every ttools based topic tool supports a range of built-in operations, implemented by modules which are automatically added to the tool directory during tool instantiation. A prominent role plays the generic help operation, which translates the tool scheme into usage information. Other built-in operations provide generic functionality which may add to the usefulness of the topic tool. As an example, built-in operation doctypes reports the document types (root element names and namespaces) encountered in a set of XML resources, which may be specified by any combination of document URIs, directory filter descriptors (see section “Directory Filter Descriptors (docDFD, txtDFD)”) and document catalogs. Using further parameters, additional information about the documents can be requested.

Workflow

The ttools based development of a topic tool can be schematically described as the following workflow.

Writing XQuery code

Coding is significantly simplified by the packaging of all input information into a single container (the request message) and its exposal via accessor functions (e.g. tt:getParam). Especially helpful are three facts:

An interesting aspect concerns the information flow between XQuery functions. When the operation providing function delegates some tasks to other functions, these may have a function signature including a request message parameter. The request message passed on may be constructed from scratch, as the message interface includes functions enabling such a construction. But it may also be the original request, or a modified copy of it - the request interface includes functions for creating copies with parameters removed, overwritten or added. When the request passed on is not constructed from scratch, but received from the outside, whether or not subsequently modified before being passed on, it may carry information which the called function evaluates, but which the calling function need not be aware of. As an example, consider a function which operates on the results of some kind of search. Such a function may pass the original request on to the search function, trusting the request (which it received from the user and which has passed validation) to contain all information relevant to the search function. Now the search function may evolve, incrementally extend its parameter usage and evaluate additional input parameters, without the calling function being aware of it – search control parameters do not concern it and are not inspected. As a result, the search function may evolve freely and independently of other functions operating on the search results, as the request message hides the changes of search related input. If today the search function begins to evaluate some additional input, the code of search result processing functions need not be changed. The only adaptation necessary concerns the annotations: in order to benefit from the extended search functionality, a search result processing operation should add the new, search related parameters to its own operation signature. But even this adaptation can as well be postponed, until the need is felt.

Such an interaction of functions may be called message-based – as opposed to parameter list based. ttools provides a sophisticated set of message related functionality, including functions for transforming messages by extension, value overwriting and reduction. The primary use of this functionality is the dealing with the original request message, constructed by infrastructure from user input. However, beyond that, the availability of message related functionality encourages the practise of message-based interactions among functions. A recommended design pattern of topic tools can be summarized as follows: (a) identify pieces of core functionality, which have a potentially growing set of input information; (b) provide such functionality with a message-based signature, rather than a parameter list based signature. This approach can considerably help in the process of evolving complex applications in a safe way.

Writing annotations

Writing code and editing the annotations are two activities performed in tandem. While writing the code of an operation, the developer switches over to the annotations whenever he decides to modify operation input, typically by adding a further control parameter. Then he resumes code writing, referring to the added parameters using request message accessors.

The annotation of an operation defines the operation in terms of its parameters. For every parameter, the following information is specified (the last two items optionally):

The editing of annotations takes effect immediately after re-building the topic tool. This is achieved by a command line call (invoking the build operation of ttools) and usually lasts less than a second.

Advanced usage - extending the framework

Developing topic tools using the ttools framework gives access to the automated construction of object parameters and automated facet validation. ttools ships with a range of built-in object parameter types and facet kinds.

A topic tool may extend the framework by defining tool specific parameter types and facet kinds. This is very easily done. In order to define a tool specific object parameter type, one has to write a parsing function which constructs from an input string an element of the appropriate structure. The signature must take a single input parameter of type xs:string, and the return value must be a single item of arbitrary type (item()). Similarly, in order to define an additional facet kind, one must write a facet validation function which will validate a parameter value item against the new facet kind. The signature must take two input parameters, the first of type item() receiving the value item to be checked, the second of type xs:string receiving the facet value; the return value must be element(z:errors)?. A library module containing type definitions and/or facet validation functions must contain an annotation of type extensions, which binds the parsing functions to parameter types and the facet validation functions to facet kinds. The implementation of framework extensions may for example look as follows:

(:~ 
 : ttoolsExtensions.mod.xq - application specific extensions of the parameter parser and facet validator.
 :)

(:~@extensions
<extensions>
    <type name="travelScenario" parser='parseTravelScenario'/>
    <facet name="blacklisted" validator='checkBlacklisted'/>    
</extensions>    
:)
module namespace m="http://www.ttools.org/otdsv/xquery-functions";
...
declare namespace z="http://www.ttools.org/structure";
...
declare function m:parseTravelScenario($text as xs:string)    
        as item() {...}

declare function m:checkBlacklisted($item as item(), $facetValue as xs:string)    
        as element(z:errors)? {...}

After dropping this library module into the tool directory, the annotations of operations may define parameters of type travelScenario and constrain parameters to conform to a blacklisted facet. The definition of additional object parameter types is typically accompanied by functions taking elements of the respective structure as input. These functions constitute the "interface" of the new object type and largely determine the value which the object type has for the tool application.

It should be remembered that object construction and facet validation are performed automatically, so that the application never needs to call the functions which it provides for the definition of additional types and the evaluation of additional facet kinds.

Summary

The development of a topic tool is simplified by code generation. What remains to be done is the writing of functions which are exposed as operations. Code writing is accompanied by annotation editing, which provides the code with an environment of validated information. Frequent extension of this information – for example caused by stepwise refinement of functionality – is accomplished in a painless way. Whenever the need arises, just add a parameter to the operation annotation: from now on the parameter value is available to the operation code.

Code writing is greatly simplified by having all external input encapsulated in a single container – the request message – which is accessed via functions, completely isolating application code from the original forms of external input. Application code is thus embedded in an infrastructure which takes on the responsibility of input validation and input augmentation. In the case of so-called object parameter types, the embedding infrastructure transforms user input into entities corresponding to program objects.

The message-related functionality provided by ttools also encourages the use of messages in function signatures in general, whether or not the function is exposed as a topic tool operation. Messages enable the "tunneling" of information, so that a calling function may be unawares of the information requirements of the called function. The use of message-controlled functions thus favours a robust integration of complex functionalities - like search and processings of search result - which is an important concern in general, and when developing topic tools in particular.

Tool instantiation

The very first step of tool development is tool instantiation. We call the ttools operation new, specifying a tool directory whose name is equal to the intended tool name:

basex -b "request=new?dir=/projects/xclear" /projects/ttools/ttools.xq

==>
===============================================================

Topic tool created:  xclear
Tool directory:      /projects/xclear

The tool can already be called. Example:
   basex -b "request=?" /projects/xclear/xclear.xq

Use operation 'add' for adding module prototypes. Example:
   basex -b "request=add?dir=/projects/xclear, mod=fooMod, ops=fooOp barOp foobarOp" /projects/ttools/ttools.xq
   
===============================================================

As a result, the tool directory xclear has been created and filled with various XQuery library modules, as well as a generated main module xclear.xq, which can be called.

01.07.2014  22:12             4.473 xclear.xq
07.04.2014  23:47               599 _constants.mod.xq
27.04.2014  12:50             8.656 _csvParser.mod.xq
28.06.2014  13:11            11.855 _dcat.mod.xq
28.06.2014  12:39             5.172 _docs.mod.xq
01.07.2014  22:12             1.155 _extensions.mod.xq
29.06.2014  08:52             8.333 _help.mod.xq
15.04.2014  00:09            19.666 _nameFilter.mod.xq
15.04.2014  00:13            14.944 _nameFilter_parser.mod.xq
07.04.2014  22:29            20.893 _namespaceTools.mod.xq
07.04.2014  22:29             1.744 _reportAssistent.mod.xq
29.06.2014  09:38            55.935 _request.mod.xq
28.06.2014  13:18            10.144 _request_getters.mod.xq
12.04.2014  08:29             2.908 _request_parser.mod.xq
28.06.2014  12:02             3.260 _request_setters.mod.xq
27.04.2014  14:37             3.606 _stringTools.mod.xq

Exploring the initial tool version

A topic tool has a single parameter, request. If supplied with the parameter value “?”, a topic tool is expected to give an overview of its operations. We give it a try:

basex -b "request=?" /projects/xclear/xclear.xq            

==>
TOOL: xclear

OPERATIONS   PARAMS
====================================================
_help        default, mode, ops?, type
dcat         dfd+
docs         dcat*, doc*, docs*
doctypes     attNames, dcat*, doc*, docs*, elemNames
====================================================

Our freshly created tool has already four operations! Operation _help is what is called when the request value starts with a question mark, rather than an operation name. The other three operations are built-in operations augmenting the functionality of any topic tool we build using ttools. For example, operation doctypes provides basic information about a set of documents. In order to take a closer look at its parameters, we refine our _help request, specifying the Boolean parameters type and default, in order to learn the parameter types and see any default values. We narrow down the response to operation doctypes by setting the optional parameter ops to the value doct*:

basex -b "request=?default, type, ops=doct*" /projects/xclear/xclear.xq

==>
TOOL: xclear

OPERATIONS   PARAMS
==========================================================================
doctypes     attNames=false...... : xs:boolean
             dcat................ : docCAT* (sep=WS)
             doc................. : docURI* (sep=WS)
             docs................ : docDFD* (sep=;)
             elemNames=false..... : xs:boolean
               At least 1 of these parameters must be set: dcat, doc, docs
==========================================================================

The parameter types docURI, docDFD and docCAT are very important, because they are the parameter types used for supplying the tool with XML documents. If an operation demands input which may consist of a set of documents, rather than necessarily a single document, it is a recommended pattern to use a triple of parameters of type docURI, docDFD and docCAT, respectively, and (if appropriate) mark them as a group of which at least one parameter must be set. This gives the user great freedom how to supply input documents, as he can use any combination of three approaches:

A dcat can be created calling operation dcat and supplying it with a document filter descriptor. If stored in a file, the dcat can later be supplied to any parameter of type docCAT. To get a feeling for our new tool, we invoke operation doctypes, supplying it with a directory filter descriptor. Suppose we are interested in the XML resources directly or indirectly contained by a directory /projects/xsd/niem-2.1. This is how we request a doctypes report:

basex -b "request=doctypes?docs=^/projects/xsd/niem-2.1/*" /projects/xclear/xclear.xq

==>
  <z:doctype name="schema@http://www.w3.org/2001/XMLSchema" count="123">
    <doc href="file:///C:/projects/xsd/niem-2.1/ansi-nist/2.0/ansi-nist.xsd"/>
    <doc href="file:///C:/projects/xsd/niem-2.1/ansi_d20/2.0/ansi_d20.xsd"/>
    <doc href="file:///C:/projects/xsd/niem-2.1/apco/2.1/apco.xsd"/>
    ...
  </z:doctype>

We just used a directory filter descriptor as input to an operation. One may also first use it to create a dcat, and later use this dcat in order to supply an operation with all documents which the dcat references. To create a dcat, we call the built-in dcat operation, supplying the directory filter descriptor as value of parameter dfd:

basex -b "request=dcat?dfd=^/projects/xsd/niem-2.1/*" /projects/xclear/xclear.xq

==>
<dcat dirs="^/projects/xsd/niem-2.1" files="" subDirs="" countFiles="123" t="2014-04-18T12:57:44+02:00" 
xml:base="file:///C:/projects/xclear/">
  <doc href="file:/projects/xsd/niem-2.1/ansi-nist/2.0/ansi-nist.xsd"/>
  <doc href="file:/projects/xsd/niem-2.1/ansi_d20/2.0/ansi_d20.xsd"/>
  <doc href="file:/projects/xsd/niem-2.1/apco/2.1/apco.xsd"/>
   ...
</dcat>

If we store the output in a file (say, /projects/dcats/dcat-niem-2.1.xml), we can then supply the file name to any parameter of type docCAT, effectively passing to the tool all documents referenced by the catalog. For example, the following call of doctypes yields the same result as the first one:

basex -b "request=doctypes?dcat=^/projects/dcats/dcat-niem-2.1.xml" /projects/xclear/xclear.xq

By now we have some feeling how to use a topic tool, namely how to specify aggregate input in alternative ways. We are ready to fill our tool with new functionality.

Adding prototypes

Our tool shall be helpful when we want to explore an XML document, or a set of documents. The first version shall have three operations: names, paths and values. We decide to place their implementation in a single module, called items. A convenient way to create the module is to use the add operation of ttools. We specify the topic tool (parameter dir), the module name (parameter mod) and the operations (parameter ops):

basex -b "request=add?dir=/projects/xclear, mod=items, ops=names paths values" /projects/ttools/ttools.xq

==>
===============================================================
XQuery module created: items.mod.xq
Operations:            names paths values

Directory:             /projects/xclear
Topic tool:            xclear
           
The new operations are available. Example:

   basex -b "request=names?doc=doc1.xml doc2.xml doc3.xml" /projects/xclear/xclear.xq
   
To implement them, edit these functions: names paths values    
===============================================================

We check that new operations have appeared in the tool interface:

basex -b "request=?" /projects/xclear/xclear.xq

==>
TOOL: xclear

OPERATIONS   PARAMS
====================================================
_help        default, mode, ops?, type
dcat         dfd+
docs         dcat*, doc*, docs*
doctypes     attNames, dcat*, doc*, docs*, elemNames
names        dcat*, doc*, docs*
paths        dcat*, doc*, docs*
values       dcat*, doc*, docs*
====================================================

Indeed, the tool now has three new operations - names, paths and values. Their parameter sets are just a first proposal which we will extend or change. Now we inspect the generated module items.mod.xq. A complete listing can be found in the appendix Appendix A. Here we look at selected parts. The text begins with this:

(:
 : -------------------------------------------------------------------------
 :
 : items.mod.xq - Document me!
 :
 : -------------------------------------------------------------------------
 :)
 
(:~@interface
<interface>
   <operations>
      <operation name="names" type="node()" func="names">     
         <param name="doc" type="docURI*" sep="WS" pgroup="input"/>
         <param name="docs" type="docDFD*" sep=";" pgroup="input"/>           
         <param name="dcat" type="docCAT*" sep="WS" pgroup="input"/>
         <pgroup name="input" minOccurs="1"/>         
      </operation> 
      <operation name="paths" type="node()" func="paths">     
         <param name="doc" type="docURI*" sep="WS" pgroup="input"/>
         <param name="docs" type="docDFD*" sep=";" pgroup="input"/>           
         <param name="dcat" type="docCAT*" sep="WS" pgroup="input"/>
         <pgroup name="input" minOccurs="1"/>         
      </operation> 
      <operation name="values" type="node()" func="values">     
         <param name="doc" type="docURI*" sep="WS" pgroup="input"/>
         <param name="docs" type="docDFD*" sep=";" pgroup="input"/>           
         <param name="dcat" type="docCAT*" sep="WS" pgroup="input"/>
         <pgroup name="input" minOccurs="1"/>         
      </operation>
    </operations>
</interface>    
:)  

module namespace f="http://www.ttools.org/xclear/xquery-functions";
...

The build operation of ttools searches all library modules in the tool directory for text enclosed between (:~@interface and the closing :). This text is interpreted as XML annotation declaring operations which the containing library module contributes to the topic tool. Each operation is defined in terms of a name, a function implementing it (@func, defaulting to the operation name), a return type (@type, defaulting to node()), parameters (<param>) and optional parameter groups (<pgroup>). Parameter groups can be used for expressing cardinality constraints referring to a group of parameters, rather than to a single parameter - for example "at least one (at most one / exactly one) of these parameters ( ... ) must be specified".

Each parameter is defined in terms of a name (@name), a type (@type), (optionally) facets (@fct_xyz) and (optionally) membership in a parameter group (@pgroup). Facets constrain valid parameter values beyond the constraints implied by the type. The type consists of an item type, optionally followed by a cardinality constraint. The parameters as they were generated are of course only meant as a starting point, to be extended and/or changed. We note that the generated module conforms to the recommended practise to support three alternative ways of specifying input documents. It uses three parameters of types docURI, docDFD and docCAT and joins them into a parameter group, constraining the user to provide at least one of these parameters.

Now we inspect the generated functions implementing the operations. For example:

(:~
 : Document me!
 :
 : @param request the operation request
 : @return a report describing ...
 :) 
declare function f:names($request as element())
        as element() {
    let $docs := tt:getParams($request, 'doc docs dcat')    
    return
        <z:names countDocs="{count($docs)}">{
           ()
        }</z:names>
};

So the initial version just creates an element echoing the operation name and fills it with a @countDocs attribute displaying the number of input documents. Let’s try, using a directory /projects/xsd/niem-2.1 as input:

basex -b "request=names?docs=^/projects/xsd/niem-2.1/*" /projects/xclear/xclear.xq

==>
<z:names xmlns:z="http://www.ttools.org/xclear/structure" countDocs="123"/>

First operation, first implementation

Now we know that we are “connected” – the tool can be supplied with input documents - and we are ready to add code. Here comes a first version of the names operation:

declare function f:names($request as element())
        as element() {
    let $docs := tt:getParams($request, 'doc docs dcat') 
    let $atts := 
        for $n in distinct-values($docs//@*/local-name(.))
        order by lower-case($n) return <z:a n="{$n}"/>
    let $elems := 
        for $n in distinct-values($docs//*/local-name(.))
        order by lower-case($n) return <z:e n="{$n}"/>
    return
        <z:names countDocs="{count($docs)}">{
           <z:atts count="{count($atts)}">{$atts}</z:atts>,
           <z:elems count="{count($elems)}">{$elems}</z:elems>           
        }</z:names>
};

We repeat our call of the names operation:

basex -b "request=names?docs=^/projects/xsd/niem-2.1/*" /projects/xclear/xclear.xq

==>
<z:names xmlns:z="http://www.ttools.org/xclear/structure" countDocs="123">
  <z:atts count="29">
    <z:a n="abstract"/>
    <z:a n="attributeFormDefault"/>
    <z:a n="base"/>
    ...
  </z:atts>
  <z:elems count="40">
    <z:e n="annotation"/>
    <z:e n="any"/>
    <z:e n="anyAttribute"/>
    ...
  </z:elems>
</z:names>

First refinement - adding a parameter

Now let us think about refinement. We want to give the user the possibility to determine the scope – only elements, only attributes, or both. To achieve this, we add to the annotation of operation names a new parameter:

<param name="scope" type="xs:string" fct_values="att, elem, all" default="all"/>

Note the default value and the values facet, constraining valid parameter values to the strings “att”, “elem” and “all”. The function is easy to adapt: (a) add a line for reading the parameter value and (b) wrap the output of attribute and element names in if expressions dependent on the parameter value. Here comes the new version:

declare function f:names($request as element())
        as element() {
    let $docs := tt:getParams($request, 'doc docs dcat')
    let $scope as xs:string := tt:getParams($request, 'scope')      (: READ PARAM :)
    let $atts :=    
        for $n in distinct-values($docs//@*/local-name(.))
        order by lower-case($n) return <z:a n="{$n}"/>
    let $elems := 
        for $n in distinct-values($docs//*/local-name(.))
        order by lower-case($n) return <z:e n="{$n}"/>
    return
        <z:names countDocs="{count($docs)}">{
            if ($scope eq 'elem') then () else                      (: DEPENDENT ON PARAM :)
                <z:atts count="{count($atts)}">{$atts}</z:atts>,
            if ($scope eq 'att') then () else                       (: DEPENDENT ON PARAM :)
                <z:elems count="{count($elems)}">{$elems}</z:elems>           
        }</z:names>
};

Note the type of the parameter value in this line:

let $scope as xs:string := tt:getParams($request, 'scope')

We can confidently use the type without “?”, as the annotation implies a cardinality of exactly one item, and the framework will never call the function unless the user input is valid. We re-build and call _help in order to verify that our annotations have been compiled into code:

basex -b "request=build?dir=/projects/xclear" /projects/ttools/ttools.xq
basex -b "request=?type, default, ops=names" /projects/xclear/xclear.xq

==>
TOOL: xclear

OPERATIONS   PARAMS
============================================================================
names        dcat................ : docCAT* (sep=WS)
             doc................. : docURI* (sep=WS)
             docs................ : docDFD* (sep=;)
             scope=all........... : xs:string; facets: values=att, elem, all
               At least 1 of these parameters must be set: dcat, doc, docs
============================================================================

We check that validation proceeds as it should, supplying an invalid scope value:

basex -b "request=names?docs=^/projects/xsd/niem-2.1/*,scope=attributes" /projects/xclear/xclear.xq

==>
Invalid call
============

INVALID_PARAMETER_FACET       Parameter 'scope': item value (attributes) not among the valid item 
                                 values (att, elem, all).

-------------------------------------

Calling the tool correctly, we get the correct result:

basex -b "request=names?docs=^/projects/xsd/niem-2.1,scope=att" /projects/xclear/xclear.xq

==>
<z:names xmlns:z="http://www.ttools.org/xclear/structure" countDocs="123">
  <z:atts count="29">
    <z:a n="abstract"/>
    <z:a n="attributeFormDefault"/>
    <z:a n="base"/>
    ...
    <z:a n="use"/>
    <z:a n="value"/>
    <z:a n="version"/>
  </z:atts>
</z:names>

Adding a name filter

We add one more refinement: the user may optionally specify a name filter, so that only items with a matching name are returned. It does not take more than a minute to achieve this. First we add an optional parameter names to the annotation, using the object parameter type nameFilter:

   <param name="names" type="nameFilter?"/>

The adaptation of the function is minimalistic. We read the parameter value:

   let $filter as element(nameFilter)? := tt:getParams($request, 'names')

and add a couple of predicates to the path expressions returning attributes and elements, respectively:

    let $atts :=    
        for $n in distinct-values($docs//@*
            [not($filter) or tt:matchesNameFilter(local-name(.), $filter)]/local-name(.))    (: FILTER ! :)
        order by lower-case($n) return <z:a n="{$n}"/>    
    let $elems := 
        for $n in distinct-values($docs//*
            [not($filter) or tt:matchesNameFilter(local-name(.), $filter)]/local-name(.))    (: FILTER ! :)
        order by lower-case($n) return <z:a n="{$n}"/>

Done! After a re-build, we can retrieve name-filtered item names:

basex -b "request=build?dir=/projects/xclear" /projects/ttools/ttools.xq
basex -b "request=names?docs=^/projects/xsd/niem-2.1/*,scope=att, names=*min*" /projects/xclear/xclear.xq

==>
<z:names xmlns:z="http://www.ttools.org/xclear/structure" countDocs="123">
  <z:atts count="1">
    <z:a n="minOccurs"/>
  </z:atts>
</z:names>

Second operation, first implementation

Now we tackle the second operation, returning data paths. First we write a little function which returns for any XML attribute or element its data path:

declare function f:path($n as node())
        as xs:string {
    string-join(
        $n/ancestor-or-self::node()/concat(self::attribute()/'@', local-name(.)),    
        '/')        
};

With this function in place, the operation is implemented very quickly:

declare function f:paths($request as element())
        as element() {
    let $docs := tt:getParams($request, 'doc docs dcat')   
    let $paths := 
        for $p in distinct-values($docs//(*, @*)/f:path(.))
        order by lower-case($p)
        return <z:path p="{$p}"/>
    return
        <z:paths countDocs="{count($docs)}" countPaths="{count($paths)}">{
           $paths
        }</z:paths>
};

After a re-build, we can retrieve a path list:

basex -b "request=build?dir=/projects/xclear" /projects/ttools/ttools.xq
basex -b "request=paths?docs=^/projects/xsd/niem-2.1/*" /projects/xclear/xclear.xq

==>
<z:paths xmlns:z="http://www.ttools.org/xclear/structure" countDocs="123" countPaths="653">
  <z:path p="/schema"/>
  <z:path p="/schema/@attributeFormDefault"/>
  <z:path p="/schema/@elementFormDefault"/>
   ...
</z:paths>

Adding a path filter

Now we refine the operation by adding a path filter parameter. This enables the user to search for paths with a particular pattern, e.g. ending with a particular element or attribute, or all children or descendants of elements with a particular name. This is easily done. We extend the operation annotation by an optional filter parameter, using the object parameter type pathFilter:

   <param name="paths" type="pathFilter?"/>

And we change the function code as follows. We read the parameter value:

   let $filter as element(pathFilter)? := tt:getParams($request, 'paths')

and we introduce a predicate which filters the paths, rejecting any candidates not matching the path filter, if a path filter has been supplied:

   distinct-values($docs//(*, @*)/f:path(.))[not($filter) or tt:matchesPathFilter(., $filter)]

After a re-build, we can, for example, learn all data paths pointing to elements and attributes under xs:appInfo elements:

basex -b "request=build?dir=/projects/xclear" /projects/ttools/ttools.xq
basex -b "request=paths?docs=^/projects/xsd/niem-2.1/*, paths=appinfo//*" /projects/xclear/xclear.xq > paths.xml

==>
<z:paths xmlns:z="http://www.ttools.org/xclear/structure" countDocs="123" countPaths="27">
  <z:path p="/schema/annotation/appinfo/@source"/>
  <z:path p="/schema/annotation/appinfo/ConformantIndicator"/>
  <z:path p="/schema/annotation/appinfo/resource"/>
  ... 
</z:paths>

Third operation, first implementation

This operation reports data values found in attributes and elements. For each attribute and element name, a sample of values is displayed. In the first version, the sample size is set to the value 10. The complete code looks like this:

declare function f:values($request as element())
        as element() {
    let $docs := tt:getParams($request, 'doc docs dcat')   
    
    let $limit := 10
    let $atts := $docs//@*
    let $elems := $docs//*[not(*)][text()]
    let $attNames := distinct-values($atts/local-name(.))    
    let $elemNames := distinct-values($elems/local-name(.))
    let $attValues :=
        for $name in $attNames
        let $values := distinct-values($atts[local-name(.) eq $name])
        let $show := for $v in $values[position() le $limit] 
                     order by lower-case($v) return <z:v v="{$v}"/>
        order by lower-case($name) return
            <z:a n="{$name}" count="{concat(count($show), '/', count($values))}">{$show}</z:a>            
    let $elemValues :=
        for $name in $elemNames
        let $values := distinct-values($elems[local-name(.) eq $name])
        let $show := for $v in $values[position() le $limit] 
                     order by lower-case($v) return <z:v v="{$v}"/>
        order by lower-case($name) return
            <z:e n="{$name}" count="{concat(count($show), '/', count($values))}">{$show}</z:e>            
    return
        <z:values countDocs="{count($docs)}">{
            <z:atts countAtts="{count($atts)}">{$attValues}</z:atts>,
            <z:elems countAtts="{count($elems)}">{$elemValues}</z:elems>            
        }</z:values>
};

After a re-build we can request reports which yield for each attribute and element name a sample of values found in items with that name:

basex -b "request=build?name=xclear, dir=/projects/xclear" /projects/ttools/ttools.xq
basex -b "request=values?docs=^/projects/xsd/niem-2.1" /projects/xclear/xclear.xq

==>
<z:values xmlns:z="http://www.ttools.org/xclear/structure" countDocs="123">
  <z:atts countAtts="102132">
    <z:a n="abstract" count="1/1">
      <z:v v="true"/>
    </z:a>
    <z:a n="attributeFormDefault" count="2/2">
      <z:v v="qualified"/>
      <z:v v="unqualified"/>      
    </z:a>
    <z:a n="base" count="10/816">
      <z:v v="ansi-nist:ALSCodeSimpleType"/>
      <z:v v="ansi-nist:BTYCodeSimpleType"/>
      <z:v v="ansi-nist:COLCodeSimpleType"/>
      <z:v v="ansi-nist:CSICodeSimpleType"/>
      <z:v v="ansi-nist:CSNCodeSimpleType"/>
      <z:v v="ansi-nist:CSPCodeSimpleType"/>
      <z:v v="ansi-nist:NISTImageType"/>
      <z:v v="s:AugmentationType"/>
      <z:v v="s:ComplexObjectType"/>
      <z:v v="xsd:token"/>
    </z:a>    
    ...
</z:values>

Adding a name filter map

Now we want to make the size of value samples dependent on the item name, so that the user may instruct the tool, for example, to return no values at all for some names, and different numbers for different name patterns. To achieve this, we extend the operation annotation, adding a parameter of type nameFilterMap(xs:integer):

   <param name="nvalues" type="nameFilterMap(xs:integer)?"/>

Then we modify the function code as follows. We read the parameter value:

   let $fmap as element(nameFilterMap)? := tt:getParams($request, 'nvalues')

and set the number of terms dependently on the item name:

   let $limit := tt:nameFilterMapValue($name, $fmap, $defaultLimit)

That was it. The following call, for example:

basex -b "request=values?docs=^/projects/xsd/niem-2.1/*, nvalues=0 % 5:docum* block % 10:target* % 20:use" 
/projects/xclear/xclear.xq

sets the value sample sizes to the following values:

The result looks accordingly:

<z:values xmlns:z="http://www.ttools.org/xclear/structure" countDocs="123">
  <z:atts countAtts="102132">
    <z:a n="abstract" count="0/1"/>
    <z:a n="attributeFormDefault" count="0/2"/>
    <z:a n="base" count="0/816"/>
    <z:a n="block" count="2/2">
      <z:v v="extension"/>
      <z:v v="restriction"/>
    </z:a>
    <z:a n="default" count="0/12"/>
    ...
</z:values>

See appendix for the full source code of this operation, as well as module items.mod.xq as a whole, both its initial state as generated by ttools (Appendix A), and the final state after our coding work (Appendix B).