Balisage Paper: Scripting Documents with XQuery: Virtual Documents in TNTBase

The Core

The core of TNTBase consists of the xSVN module, which integrates Berkeley DB XML [Ber09b] into a Subversion server [SVN08]. DB XML stores HEAD revisions of XML files; non-XML content like PDF, images or LaTeX source files, differences between revisions, directory entry lists and other repository information are retained in a usual SVN back-end storage (Berkeley DB [Ber09a] in our case). It is worth mentioning here that TNTBase also supports branching as SVN does (see also Section section “Multiple Versions of Documents” for a use-case of virtual documents with on this). Keeping XML documents in DB XML allows us to access those files not only via any SVN client, but also through the DB XML API that supports efficient querying of XML content via XQuery [BCF+07] and modifying that content via XQuery Update [CDF+08]. As with many XML-native databases, DB XML (and hence TNTBase) supports indexing, which improves performance of certain queries. TNTBase also adopted transactional support (atomicity, consistency, isolation, durability) from DB XML.

The TNTBase system is realized as a web-application that provides two different interfaces to communicate with: an xSVN interface and a RESTful interface (for details refer to [Z+10]) for XML-related tasks. The xSVN interface behaves like the normal SVN interface — the mod_dav_svn Apache module serves requests from remote SVN clients — with one exception: If one of the committed XML files is ill-formed, then xSVN will abort the whole transaction. The RESTful [JSR09] interface provides XML fragment access to the versioned collection of documents:

For more information about the TNTBase core refer to [ZK09]. Since then we have significantly increased stability and performance that can be proved that TNTBase is being used for the LATIN Project [LAT], for General Computer Science lectures repository (that counts more than 2000 XML documents with over 2500 revisions) and Translation SUMO to OMDoc Project [Mis10]. Moreover, we are mirroring some of TNTBase repositories to normal SVN repositories by replication functionality adopted from Subversion. This possibility once more justifies the decision of combining the two systems. If something went wrong with a TNTBase repository and the data got corrupted (actually, it never happened to us), then we can easily restore them from a replicated SVN repository with all history preservation.

Application Layer of TNTBase

TNTBase: TNTBase

In our experiments it turned out that many tasks specific to particular XML formats can be done by TNTBase. This was a reason to derive a separate layer on top of the TNTBase core and augment this layer with format-specific functionalities (see Figure TNTBase). Although the detailed information can be found in [ZKR10], let us briefly describe the major features:

VDoc Specifications and Skeletons

VDoc Specification (VDoc Spec) is the most important part of any VDoc. Basically, it is a document template with XQuery inclusions and some other auxiliary elements helping TNTBase to figure out how to execute a particular XQuery and how to populate query results. It must be XML. In Listing Example of a VDoc Spec one can see a simple example of a VDoc Spec that is supposed to define a VDoc that would contain thematic lecture exercises together with their authors. For the complete RelaxNG [Rel] schema refer to [ZK].

<tnt:virtualdocument xmlns:tnt=”http://tntbase.mathweb.org/ns”>
  <tnt:skeleton xml:id=”exercises”>
    <omdoc xmlns:dc=”http://purl.org/dc/elements/1.1/”>
      <dc:title>Exercises for Computer Science lectures</dc:title>
      <dc:creator>Michael Kohlhase</dc:creator>
      <omdoc>
        <dc:title>Acknowledgements</dc:title>
        <omtext>
          The following individuals have contributed material to this document:
          <tnt:xqinclude query=”tnt:collection(’/exercises//*.omdoc’)//dc:creator”>
            <tnt:return><tnt:result/></tnt:return>
          </tnt:xqinclude>
        </omtext>
      </omdoc>
      <omdoc>
        <dc:title>Exercises</dc:title>
        <tnt:xqinclude>
          <tnt:query name=”exercises.xq”/>
          <tnt:return><tnt:result/></tnt:return>
        </tnt:xqinclude>
      </omdoc>
    </omdoc>
  </tnt:skeleton>

  <tnt:query name=”exercises.xq”>
    for $t in $topics return
      tnt:collection(concat(’/exercises/’, $t, ’/*.omdoc’))//exercise[position() le $max]
  </tnt:query>

  <tnt:params>
    <tnt:param name=”max”>
      <tnt:value>10</tnt:value>
    </tnt:param>
    <tnt:param name = ”topics”>
      <tnt:value>search</tnt:value>
      <tnt:value>graphs</tnt:value>
    </tnt:param>
  </tnt:params>
</tnt:virtualdocument>

A VDoc Spec consists of a VDoc Skeleton (VDoc Skel), number of named queries that are referenced from VDoc Skel and arbitrary parameters that are used in XQueries. Let us consider these elements in order:

<tnt:xqinclude query=”tnt:collection(’/exercises//*.omdoc’)//dc:creator/text()”>
  <tnt:return><omtext><tnt:result/></omtext></tnt:return>
</tnt:xqinclude>

tnt:collection(’/exercises//*.omdoc’)//dc:creator/text()

<omtext>Paul</omtext>
<omtext>John</omtext>

<tnt:virtualdocument xmlns:tnt=”http://tntbase.mathweb.org/ns”>
  <tnt:skeleton href=”/basic-spec.xml”/>
  <tnt:query name=”exercises.xq”>
    for $t in $topics return
      tnt:collection(concat(’/exercises/’, $t, ’/*.omdoc’))//exercise/statement
  </tnt:query>
</tnt:virtualdocument>

VDocs as TNTBase FS Entities and Their Materializing

In order to make VDocs as a part of a TNTBase file system and expose them to users, one has to utilize the RESTful API of TNTBase [Z+10]. When creating a VDoc, a user has to provide a path and a name of a VDoc, a VDoc Spec and its revision which a VDoc will be linked to and, optionally, a set of parameters, analogously to those that a VDoc Spec has (thus, one can override VDoc Spec parameters or define new ones). Note that it is possible to associate a VDoc with a VDoc Spec of a particular revision, not only with the HEAD revision. Parameters associated with VDoc file system entities make VDoc Spec even more reusable. When retrieving content of a VDoc (i.e. the expanded version of a VDoc Spec), a user might also provide parameters that will override those defined in a VDoc Spec and in a VDoc itself. It might be very useful for dynamic alternation of a VDoc or during debugging. In Figure VDoc Workflow the content of a VDoc is presented in the second picture at the bottom. We see how data are aggregated and mashed up with the static parts of a VDoc Spec.

Currently every time VDoc content is requested, TNTBase executes every query included into a VDoc Spec and aggregates the results. For certain VDoc Specs it may be time-consuming. Furthermore, generated on the fly VDoc content is not accessible via xSVN working copy. Therefore a user may want to fix the content and make it versioned. In our example when a user is satisfied with exercises list he got through a VDoc, he may desire to make it persistent by putting it into a repository file under a certain path. To satisfy these demands TNTBase RESTful interface provides a VDoc feature that turns the content of a VDoc into a regular file in a repository. We call such a process as VDoc materializing. If there is already a document under a provided path, the materializing process results in a new revision of that file. Thus we make VDocs content accessible via SVN client and track the revision history.

Querying VDocs

The contents of a VDoc can be addressed in a query via the TNTBase XQuery extension function tnt:vdoc($path as xs:string), where $path is a path of a VDoc in a TNTBase repository. Thus one may combine querying of usual repository files together with multiple VDocs. It is also possible to retrieve just the expanded version of a VDoc Spec (not the content of a VDoc which might be different due to additionally defined parameters). The XQuery extension function looks similar - tnt:vd-spec($path as xs:string), but instead of a VDoc path, we provide a path to a VDoc Spec. As an example, assume that we want to get a number of authors that contributed to exercises in a VDoc. The query will be simple:

count(tnt:vdoc(’/path/to/vd’)//dc:author)

VDoc Editing

One of the strongest features of VDocs is that they can be edited and committed to TNTBase via the RESTful interface. Changed parts of a VDoc that came from files in a repository will be transparently propagated back to the sources with repository history preservation, i.e. a new revision will appear in TNTBase. In Figure VDoc Workflow on the bottom right picture we can see the final phase of a VDoc workflow: editing and submitting it back -- the modified parts (marked with red) are populated back to their “home”. All changes are performed in a single xSVN transaction and only those files will be part of it that were implicitly affected by VDoc editing.

In a VDoc we distinguish static parts (i.e. those that come from the VDoc Spec) from generated ones (i.e. those parts that are results of a particular VDoc Spec query). Currently, static parts are not editable in a VDoc – TNTBase will abort a commit, if they have been changed. However, static parts can be modified by changing VDoc Spec file directly in a repository. From the generated part, only the XML elements that come from the repository are editable; TNTBase annotates them with tnt:doc and tnt:xpath attributes that cache information about the element source needed for the commit. In our exercise example an editable part might look like:

<exercise topic=”algebra” tnt:xpath=”/omdoc[1]/theory[3]/exercise[2]”
                           tnt:doc=”/exercises/algebra/isomorphic-sets.omdoc”>
          ...                
</exercise>
          ...
<exercise topic=”algebra” tnt:xpath=”/omdoc[1]/theory[2]/exercise[1]”
                           tnt:doc=”/exercises/algebra/relations.omdoc”>                   
          ...
</exercise>

A user may add attributes, text, comments, new elements or delete the old ones, but he is not allowed to modify tnt:doc and tnt:xpath attributes, otherwise TNTBase will abort committing.

So far we have considered only editable parts of the generated part. There could be, however, non-editable, generated nodes (e.g. dates or average of some values), which we call constructed. In some cases, we can even partially modify these and propagate changes back to a repository: If XQueries wrap some of DB XML elements into some other elements, making these constructed. For example, the query associated with some tnt:xqinclude element could be:

<author>{tnt:collection(’/exercises//*.omdoc’)//dc:creator}</author>

So the results might be:

<author>
 <dc:creator>Paul</dc:creator>
 <dc:creator>John</dc:creator>
</author>

In such cases we can make these elements editable via the XQuery wrapper function tnt:make-editable supplied by TNTBase for this purpose. This function adds tnt:doc and tnt:xpath attributes to the wrapped elements if possible (i.e. if the input sequence of elements are nodes in DB XML) to mark them as editable. So in our example the query should really be:

<author>{tnt:make-editable(tnt:collection(’/exercises//*.omdoc’)//dc:creator)}</author>

This leads to the following result which parts are editable:

<author>
 <dc:creator tnt:doc=’…’ tnt:xpath=’…’>Paul</dc:creator>
 <dc:creator tnt:doc=’…’ tnt:xpath=’…’>John</dc:creator>
</author>

The editing approach has a number of natural limitations:

Last but not least, TNTBase follows the “update-modify-commit-merge” cycle from the underlying version control system in this process, so if another user modified repository contents while we were editing a VDoc, submitting of the latter will fail, since our VDoc is “out-of-date” and we have to get the VDoc content again and merge our changes. This mechanism guarantees that we will not overwrite somebody else’s changes.

VDoc Schema Validation

The obvious well-formedness constraints for a VDoc Spec can easily be expressed in a RelaxNG schema, which we supply as [ZK]. But schema-validity of the VDoc Spec does not ensure validity of the resulting VDoc because it does not take the constraints of the target format into account. Fortunately, we can easily integrate the VDoc Spec schema with the target schema, if the latter meets (or is extended to meet) some modularity requirements. As an example we provide such a combination for OMDoc language in Listing A RelaxNG Schema for Virtual OMDoc Specifications.

include ”tnt-vd-spec.rnc” {
        skeleton.model  = grammar {include ”omdoc/omdoc.rnc”
                                                         {ss |= parent xq?}}
        return.internal.el = grammar {include ”omdoc/omdoc.rnc”
                                                         {start= omdoc.class|CMP|FMP
                                                          ss |= parent result.el}}}

Here we made use of the fact that the OMDoc schema has a hook (the ss schema macro) that allows replacement of elements in the declarations: all element declarations are of the form

CMP = (ss | element CMP {CMP.attribs & CMP.model})

In this situation, we only had to replace the content and attribute model of the tnt:skeleton element^[3] with the OMDoc document model, which is upgraded to allow an tnt:xqinclude element in place of all “ss-replaceable” elements. The content model of the tnt:return element is instantiated to the model of OMDoc “ss-replaceable” fragments, updated to allow tnt:result element in place of “ss-replaceable” elements. When VDoc format-aware schema is ready, we can associate it with a VDoc Spec via tntbase:validate xSVN property (details in [ZKR10]), and thus ensure that TNTBase allows committing only those VDoc Specs which would always produce content that is valid against our format schema.

Implementation Details

VDocs are realized^[4] in XQuery with help of XQuery external functions written in Java. External function are used, for instance, for dynamic query execution from another XQuery (for expansion of a VDoc Spec), getting the revision information from a repository (to control that no other revisions have been committed while editing a VDoc) or committing changes under certain path (to propagate changes to original files one edited VDoc has been submitted back). In order to support VDoc editing workflow, there is a simple XQuery implementation of XML differencing, that

Automated Exam Generation

This is a dogfood use case from our academic practice, and is (partially) used in day-to-day operation: The second author teaches a first-year, two-semester Introduction to Computer Science a Jacobs university and – over the last six years – has accumulated a collection of about 1000 homework, quiz, and exam problems encoded into the XML-based OMDoc format [Koh]. For the courses we need to prepare regular four exams, four “grand tutorial test exams” and two make-up exams per year. While the homework problems are typically new (and add to the corpus of well-tested problems), we assemble the exams from it semi-automatically with a VDoc Spec that generates random exam sheet based on the input list of topics we intend to cover throughout an exam.

There are two kind of proper exams: midterms and finals. Midterms usually are meant to be for 1 hour, although sometimes it takes 75 minutes or so, whereas finals are designed for 2 hours. Thus we also provide an exam duration as an input parameter for our exam VDoc. Changing only this parameter together with the topic list allows us to get different exam sheets that do not exceed the certain time and cover desired topics. All necessary information is encoded into the problems as RDFa metadata annotations. Our XQuery for a VDoc Spec takes care about adjusting the timing closely to the provided limit. When VDoc content is generated, it can be rendered by utilizing XSLTs and developed in our group JOMDoc library [JOM10] for rendering MathML[W3C07]. Everything is embedded into TNTBase, and once an exam VDoc is installed it is a matter of one click in the TNTBase web interface to get the unique human-readable exam sheet for the students.

VDoc Editing facilities also find an application in our use case. Before giving generated exam to students we test it on our teaching assistants that may express some of the comments or suggestions how to improve particular problems. Then we edit the contents of an exam VDoc and commit it back – all modifications are automatically patched into original XML sources: easily and painlessly. If one does not like a particular problem to be included into exam, we can adjust a VDoc parameter that excludes them from the exam.

The biggest advantage of current exam generation approach is that we write a VDoc Spec once and reuse it next semester by simply adjusting few parameters to a VDoc. When one is satisfied with the exam presented, it can be materialized and saved in a repository as a normal file that can be referenced in future to keep track how students performed on different assignment and figure out what their weaknesses are.

Although a presented approach already meets our requirements, there are some issues that could be improved. For instance, we might want to take total exam difficulty into account to generate exams that do not exceed a certain duration and that have difficulty in a certain range (again difficulty information is embedded into problems XML). That will lead to a more complicated queries for a VDoc Spec, but is still feasible. Apart from generating exams, this use cases might be used by students that are willing to sharpen their knowledge: they could generate practice sheets starting from easy tasks and end up with the complex ones. Some parameters in e.g. cookies may keep track of what exercises already appeared in the practice sheet, and a VDoc will never show them again. It could easily be done by providing dynamic parameters to a VDoc retrieval method as was described in the previous section.

Multiple Versions of Documents

In most scenarios with long-lived documents, we encounter the problem of document versions. Let us consider the case of W3C specifications like XQuery 1.0/1.1, XPath 1/2, XML 1.0/1.1., or even MathML1.0/1.0.1/2.0/2.0(2e)/3.0. They are encoded in XML format XMLSpec [XML09], so TNTBase and VDocs apply. Usually some parts of specification remained the same, while other parts change between the versions, and it is an important task to track the differences. For this use case we are experimenting with XML 1.0 and 1.1 specifications to supply the user with a view that will show only the relevant changes in the formal parts of specification branches. It is rather simple to provide an Diff VDoc via an XQuery that summarize changes in formal parts (the rules of the XML grammar are marked up by special elements in XMLSpec), ignores document order (grammars are sets, not lists of rules), and presents them as a document XMLSpec documents upgraded with difference alternatives. Note that our XQuery-based XML-diff comes in handy here. This VDoc gives a user better understanding in which direction the development is going and what changes are intended ones and which are made by mistake. Our Diff VDoc is also editable that allows a user to fix obvious bugs right on spot, without navigating to the source files. Once Diff VDoc is settled in TNTBase it can be reused to filter only relevant differences as well as transparently editing them, all in one place. Currently W3C stores specifications in a CVS repository, but does not make use of its differencing facilities for version tracking as diff is text-based and outputs even less and least relevant differences.

Note that the Diff VDoc encapsulates a particular notion of relevance in the filtering part, which may need to be explained in a document preamble. Thus the representational form of a VDoc which mixes document parts and queries is beneficial. Moreover, there can be multiple Diff VDocs for tracking (and editing) various aspects of the differences in the specifications. Such Diff VDocs may even take over the role of conflict editors we currently have in version control aware IDEs.

Managing Document Collections

The exam generation use case described above can be seen as a special case of managing (here extracting custom documents from) a collection of primary (content) documents and creating secondary documents from them that aggregate parts of the content. These secondary documents can either be used for communication to the outside (payload documents) or for management of the document collections. In this terminology, the exams above can be seen as the payload documents derived from the content documents in the problem collection. That is where VDocs may naturally come into play as we have seen above.

A very simple application of VDocs in payload documents are queries for a table of contents (collecting all sectioning elements in a narrative document), the references (collecting all citations, sorting them, and completing them with information from a bibliographic database), or an index. In DocBook [WM08] these aggregated document parts generated by XSLT stylesheets in the presentation phase, which may incur performance bottlenecks in practice, since this is not supported by indexing and caching. Moreover, VDocs make separate conceptually the issue of auto-aggregation and presentation, which allows to support workflows like previews/editing of aggregated document parts and materialization (e.g. of branches and tags) for archiving.

Another simple application of VDocs in technical payload documents is in XML-based literate programming [Knu92], where program text is intermingled with its documentation and explanation in a single document. Here a VDoc can be used to extract the program text (with comments that cross-link to) from the literate source. As a concrete XML-based example take the XMLSpec-based source of the MathML3 Recommendation [ABC+09] from which we generate the MathML3 RelaxNG Schema. A VDoc would have considerably simplified this process.

We have already seen Diff VDocs as examples of management VDocs for version management in the last section. But VDocs can also support proofreading, a very important task in the document life cycle. Often one wants to proofread special aspects of a document, e.g. whether certain technical terms are used consistently. For this we can quickly specify these terms as parameter to an XQuery that assembles all paragraphs that contain them. Then we can proofread (and edit) the text passages, commit them back to the collection, and move on to other proofreading tasks.

Refactoring Ontologies

Finally, VDocs can be used for refactoring OWL [SWM04] Ontologies that are written in XML Syntax, e.g. OWL 2 XML [MPPS09]. VDocs become very handy when making changes to a small subsets of multiple large ontologies. In the first phase we can preview ontologies changes in a VDoc using XQuery transform functions. In the second phase, when we are satisfied with results we can materialize a VDoc thus obtaining a refactored ontology as a usual document in a repository. Refactorings that can be done using VDocs include renaming entities, factoring out or merging modules, rewriting axioms, lowering expressivity or stripping axiom annotations. For more detailed information concerning ontology refactoring using VDocs refer to [LZ10].