Balisage Paper: Test Assertions on steroids for XML artifacts

Test Assertion Model

Test assertions (TA) is a familiar concept for QA engineers and test developers. A test assertion is a testable or measurable statement for evaluating the adherence of part of an implementation to a normative statement in a specification. Test assertions provide a link between the narrative of a specification (i.e. rules, schema, requirements, system definition) and the test suites that assess conformance of implementations. Test assertions have been mostly used in the domain of software engineering, and less often in more specialized domains such as XML artifacts, where ad-hoc solutions - and also very specialized tools - have flourished instead. Test assertions are usually declarative (logical) statements that are written as a blueprint for test cases, the latter being the actual executable tests.

A major benefit in writing test assertions, is that they represent a "conformance contract" understandable by all parties involved - domain experts, test writers, end-users. An additional interest in the XML space - where all material under test is in XML format, is that test assertions can be directly scripted using such dialects as XPath or XQuery, thus becoming themselves executable test cases. A general-purpose model for test assertions has recently been developed in the OASIS Test Assertions Guidelines (TAG) OASIS committee [8]. In this model, a test assertion (TA) is a well-structured object defined as follows:

The authors have profiled and extended this model so that test assertions become directly executable over XML artifacts, thus becoming "test cases" grouped in test suites.

The profiling consists of the following:

The Test Target and its Context

A test assertion will always focus on a “primary” target instance, but may need to access contextual material in order to test this target. This "side" material is identified in "variables" added to the TA. A simple example of this is schema-validation of a document. In the target scripting below, the test assertion will refer to the contextual document (a schema) while targeting a purchase order line item:

<testAssertion id="1234" lg=”xpath20” >
	  <var name=”poschema” type="string">http://www.mysupplychain_xyz.com/2009/04/12/po.xsd</var>
	  <target>//xyz:purchaseOrder/xyz:lineItem</target>
	  <predicate>$target instance of schema-element($poschema, xyz:lineItem) </predicate>
	  ...
	  </testAssertion>

The predicate validating a lineItem element will refer to this contextual document using the conventional variable notation ($). The predicate expressions will be pre-processed into executable XPath. The above test assertion applies to every line item of any purchase order. Variable expressions (<var>) may refer to any contextual material - either inside the same document or external. An XPath variable notations($) may then be used either to parameterize the location of a document, or to refer to the current value of the target expression:

<testAssertion id="2345" lg=”xpath20”>
	  <var name=”herbooks” >document($allbooks)/book[@author = $target/name]</var>
	  <var name=”herpublishers” >document($allpublishers)
	  //directory/publisher[fn:index-of( fn:distinct-values(
	  ‘for $bk in $herbooks return $bk/@publisher’), @name) gt 0 ]</var>

	  <target>//whoswho/arts[@section='literature']/biographies/author</target>
	  ...
	  </testAssertion>

In the above, "$allbooks" and "$allpublishers" are references to a documents that have been defined outside the test assertion. The variable "$herbooks" denotes the subset of books from this author. The variable "$herpublishers" is the subset of publishers this author has been dealing with. The target expression is matched against a third document, the main input. A predicate for the above target may express a condition over the author (the target), her related list of books ($herbooks) and of publishers ($herpublishers).

Reporting Test Outcomes

The additional Reporting element added to the test assertion structure may override the default outcome which is:

Indeed, other possible outcomes are: These outcomes are not only intended for a final test report. They can also be tested and influence the test suite execution if the test assertion that produces them is referred to in predicates and prerequisites of subsequent test assertions.

Inheritance and Composition

An important benefit of clearly identifying target categories, is the ability to leverage inheritance and composition relationships between targets. Targets often belong to a class system, in the object-oriented sense (a target may be part of an other target, may be a subclass of another target, etc.). This leads to an enhanced test execution model that is able to leverage such relationships. In particular:

The authors are in favor of supporting two modes of representation for such a model: (1) "inline" relationship information can be embedded in each test assertion as needed. (2) a different mark-up separate from test assertions will hold target model information. While (2) is a more rational and scalable approach (avoid redundant information from on test assertion to the other, etc.), (1) is a convenient approach well suited for the test assertion development phase. Examples of inline model information are shown below. The target element uses a qualification notation to indicate the super-class (message) of SOAPmessage target class.

	<target type="message:SOAPmessage" > ... </target>
      

The composition link between the main target of a test assertion - here a WSDL binding - and the related embedding target of a prerequisite reference - here the WSDL file itself - is indicated using an XPath expression relative to the selected target node ($target) as "argument" of the test assertion reference (tag:BP2703):

	<testAssertion id="BP2403">
	<target type="binding" idscheme="..." >//wsdl:definitions/wsdl:binding</target>
	<prerequisite> tag:BP2703($target/..) = 'pass' </prerequisite>
	...
	</testAssertion>
      

Another major attribute of a Target class, is the ID scheme itself an XPath expression that will return a unique ID string for each target instance, to appear in the test report.

Chained Test Assertions

A powerful aspect of the otherwise simple TAG model, is that a test assertion (TA) may refer to other test assertions. It may do so in two ways:

There are two main reasons for chaining test assertions as described in (a) above:

(a1) When a set of tests should logically be done in a particular order, meaning that every single test should only be executed if the target instance passed the previous tests. For example, the following sequence of tests is expected to be done in this order, regarding a Web service definition:

These four test assertions are chained via their prerequisite elements. This chaining means that if one fails, the subsequent tests will not be performed: whatever their outcome is, it might not make any sense or would at best produce an unnecessary distraction in the test report.

(a2) In order to "reuse" (both at scripting time and at run-time) a complex expression outcome that has already been handled by another TA. For example in the Web services basic profile, a wsdl:binding must either be an rpc-literal binding or a document-literal binding. The test for ensuring this is not a simple one:

	    <testAssertion id="BP2017" lg=”xpath20” >
	    <target>//wsil:descriptionFile/wsdl:definitions/wsdl:binding
	    [wsoap12:binding]</target>
	    <prerequisite>tag:BP2404($target) eq 'pass'</prerequisite>
	    <predicate>not(.//wsoap12:body/@use != 'literal')
	    and (count(.//wsoap12:body) = count(.//wsoap12:body/@use)) and
	    ((not(.//wsoap12:*/@style != 'rpc') and
	    not(.//wsoap12:operation[not(@style) and not(../../wsoap12:binding/@style)]))
	    or (not(.//wsoap12:*/@style != 'document')))</predicate>
	    ...
	    </testAssertion>

Several test assertions relate only to document-literal type. Once it is known that a binding is of either above type, the test to distinguish an rpc-literal from a document-literal is fairly simple. In the following test assertion that applies only to document-literal type, this simple test - used here in the target expression - guarantees that only document-literal bindings will be selected:

	    <testAssertion id="BP2111" lg=”xpath20” >
	    <target>//wsil:descriptionFile/*:definitions/wsdl:binding
	    [not(.//wsoap12:*[@style = 'rpc'])]</target>
	    <prerequisite>tag:BP2017($target) eq 'pass'</prerequisite>
	    <predicate>not(.//wsoap12:body[@parts and contains(@parts," ")])</predicate>
	    ...
	    </testAssertion>

The other form of chaining (b) is done in the Predicate expression. This allows for defining "meta-level" test assertions that wrap entire groups of test assertions by summarizing their expected outcome in a single logical expression. Such a meta-level test assertion can then be referred by other test assertions in their prerequisite condition, when this entire group of tests must be passed. For example, consider BP1214 listed in the next section. This test assertion targets a SOAP message, but needs to access a contextual document: the interface binding definition that governs the content of this message. Before executing BP1214, it is clear that the binding definition must be verified. A meta-level test assertion can "summarize" all the tests that ensure this correctness for rpc-literal bindings:

	  <testAssertion id="BP-rpc-bindings" lg=”xpath20” >
	  <target>//wsil:descriptionFile/*:definitions/wsdl:binding
	  [.//wsoap12:*[@style = 'rpc']]</target>
	  <prerequisite>tag:BP2017($target) eq 'pass'</prerequisite>
	  <predicate>(tag:BP2404($target) eq 'pass') and
	  (tag:BP2406($target) eq 'pass') and (tag:BP2020($target) eq 'pass') and
	  (tag:BP2120b($target) eq 'pass') and (tag:BP2117($target) eq 'pass') and
	  (tag:BP2118($target) eq 'pass')
	  </predicate>
	  ...
	  </testAssertion>

The above test assertion may then be used as prerequisite for BP1214, over the binding definition related to its message target, i.e. the "binding" element parent of the "operation" element selected by the XPath expression in the variable $myOpBinding:

	  <testAssertion id="BP1214">
	  <var name="myOpBinding"> ...</var>
	  <prerequisite>tag:BP-rpc-bindings($myOpBinding/..) eq 'pass'</prerequisite>
	  <target type="message:SOAPmessage" >/wsil:testLog/wsil:messageLog/wsil:message[...]
	  </target>
	  ...
	  </testAssertion>

Execution Semantics

The general mode of execution for above TAs, is of a conventional forward-chaining rule engine. This is indeed necessary due to the chaining of test assertions, and departs radically from how other XPath-based rules systems are processed (e.g. Schematron or CAM do not handle chaining). The set of test assertions that have no references to other TAs is executed first over all candidate targets. This first set of test results for all possible target instances is recorded. In the next iteration of the engine, only TAs that have all their references resolved over the first set of results are executed in turn. Their results are added to the initial result set. Subsequent iterations add to the previous result set until iterations cannot augment anymore this result set which is considered stable. At this point, all possible validations have been made over the material under test, and they are ready for an (html) test report generation.

In the XPath-extended XML markup of the TAG model, test assertions can be conditionally chained as rules to create dynamic test suites, the result of which can also be manipulated by higher-level test assertions. This approach is addressing the need to integrate validation of various XML artifacts, with validation of combinations of such artifacts (consistency across documents). Indeed, this requires composing and orchestrating test cases and test suites in a modular way.

Implementation Considerations

The automation of TAG methodology is leveraging both XLST2.0 and XPath2.0. However, end-users (TA designers) only need to know about XPath to write test assertions. XSLT is only used for the execution engine. A two-phase processing of above test assertions has been implemented by the authors, as often used when XSLT is the target execution language.

The first phase amounts to generating an XSLT-hardcoded test suite for a particular set of test assertions. The second phase amounts to executing this test suite. In addition to increased performance, the two-phase approach allows for advanced parameterization features in test assertions at different levels, with the use of variables:

	  <testAssertion id="BP1214">
	  <var name="myOpBinding">//wsil:descriptionFile/wsdl:definitions/wsdl:binding
	  [.//wsoap12:*[@style = 'rpc']]/wsdl:operation
	  [@name =  fn:local-name-from-QName(node-name($target/soap12:Body/*[1]))]</var>
	  <target type="message:SOAPmessage" >/wsil:testLog/wsil:messageLog/wsil:message[...]
	  </target>
	  ...
	  </testAssertion>

This test assertion verifies that the message is conforming to some aspect of its binding definition.

Two of the authors have developed test suites for Web Services profiles developed by the WS-Interoperability consortium (http://www.ws-i.org) using the XPath2.0-extended TAG model. These test suites include test cases that involve a combination of documents (WSDL, Schemas) and sequences of messages. About 250 test assertions were developed for three WS profiles. The entire test suite execution process (Phase 1 + Phase 2 + html rendering of the test report) is handled by stylesheets.

Prior to this one author had been leading a similar test tool development process for WS-I based on conventional programming languages (Java, C#) [4]. The advantage of the recent approach using XPath and XSL are:

Originally, the authors attempted to use XPath1.0 as expression language for TAs. This was not sufficient to handle complex correlations of XML-fragment (either intra-document or cross-document) required by WS-I test suites. XPath2.0 provided new features that significantly enhanced the expressive power of target / prerequisite / predicate expressions, such as quantified expressions, iterations and an extensive library of functions. Advanced correlation patterns inside Predicate could be expressed in a declarative way as a set of nested quantified expressions, making it possible to assign and reuse variables at each level.

XSLT2.0 proved a suitable script language to implement a forward-chaining test assertion engine thanks to features such as next-match(), while the authors have not been successful at trying this with XSLT1.0.

Along the line of managing complexity by leveraging the composability of test assertions and of their execution (chaining of test assertions, parameterization, meta-level assertions), a future enhancement will allow a test assertion to define “byproducts” , i.e. XML fragments produced by their execution (in addition to the main outcome “pass”, “fail” etc.) that can be reused in Prerequisite or Predicate of referring test assertions.

Schematron

Schematron is a simple pattern and rule language well-focused on document testing. It leverages XPath functions and expressions, and can be implemented using XSLT. Rules in Schematron can be seen as serving similar purpose as test assertions. However the authors, after initially attempting to develop WS-I test suites with Schematron 1.5, had to give up mostly due to its restricted rule execution semantics: there is no support for "logical" rule chaining, and in a pattern only one rule - the first that matches the context - will execute. This form of "if-then-else" chaining applies to the context matching and not to the result of the rule itself, unlike what is expected in conventional rule-based systems.

Schematron has been designed around the idea that the entity under test is the document, while in our test assertion engine it is before all an XML fragment, subset of some document. Each fragment (target) is systematically identified according to a well-defined scheme for its target type. This identity - defined by an XPath expression - is not only used for detailed diagnostic information in the test report, but is central to the rule chaining mechanism of the test engine, i.e. for deciding of the order of the tests on this target. Schematron allows for detailed and dynamic diagnostic information that has the ability to fully identify a subset of a document, but this does not play any role in the rule processing mechanism. Some valuable convenience feature had to be added to test assertions in the form of variables, for handling of complex Predicate expressions or to parameterize test assertions. Such variables are also supported in Schematron 1.6. On the modeling side, Schematron introduces a hierarchy of constructs (assertion, rule, pattern, phase). In contrast, the presented approach is based on a flat model relying on a single construct - test assertion - composable at different levels, but subject to the same execution semantics.

Although Schematron can be written against any XML document, it is primarily intended for XML instances. Namespace handling becomes difficult when writing rules against an XML schema, and when there is a namespace prefix in a value of an attribute or element. Schemas often need be tested against naming and design rules [5]. Rules are label specific, i.e., there is no inheritance. If there is a type hierarchy, separate rules have to be written for every type even if the intention is the same. In Schematron 1.5 rules cannot be reused or combined programmatically, although ISO Schematron has more provision for reusability such as abstract pattern and include statement.

In conclusion, although Schematron is well-positioned for document validation through its lifecycle and is sufficient for many test cases, it has been designed more in the spirit of validating content in a type-checking mode (an extended schema). Its rules are intended for detecting patterns in documents and not to be executed along a test suite processing model requiring tight control on which tests are executed and when based on previous tests results. We believe the concept of test suite is appropriate when considering a combination of diverse XML artifacts including XML-formatted data of non XML source.

OWL

OWL by itself is not a rule or test language. It rather allows for declarative semantic models. The handling and mapping of ontologies is an important aspect of validation [6]. OWL reasoner can determine if there is a conflict based on the characteristics of a particular object instance – in our context, a test target - based on a semantic definition of class membership. OWL's “open world assumption” makes it difficult to use it to validate documents. The open world assumption states that even if an object instance has not declared one of its property, the reasoner cannot assume that it does not exist. Typically, a programming routine needs to be written to “close the world” by explicitly asserting that this object instance has no such property.

Because OWL is designed for acceptable computational time, its expressivity has been limited. Even in OWL Full (the most expressive level of OWL language), the semantic expressivity is limited to constraints around cardinalities and a few relationships between class and properties such as transitive, inverse, and uniqueness. Expressing arithmetic relationships between properties is virtually impossible. There are also few reasoner that can perform OWL Full reasoning and more importantly datatype reasoning (particularly user defined datatype necessary to validate against a range of values or a code list). OWL reasoners typically process data in triple representation, which is memory-greedy. Validating a few megabytes of document on a typical desktop with one to two gigabytes of memory almost becomes impossible (industrial strength reasoning engine and memory or storage management - such as Oracle RDF database - would be needed).

XBRL

The XBRL conformance test suite is worth consideration more as a typical use case than a reusable tool. It is an example of validating a (set of) complex document(s) with advanced semantics, that must comply with various rules in addition to schema compliance. It also encompasses the testing of processors supposed to produce such documents, illustrating how document-testing and processor-testing are intertwined. The “minimal” conformance suite focuses on the document validation while “full” conformance targets XBRL processors. Full conformance involves other documents than the main document being processed, and relies on output documents (Post-Taxonomy Validation infosets) that reflect the processing semantics. Minimal conformance generally contains at least one test for each appearance in the specification of ‘MUST’ that are not already enforced by XML Schema validation. The structure of the test suite is based on the OASIS XSL Conformance Suite. The structure of each individual test is simple:

Each test case is described by a "meta-level" XML file that refers to associated test material. Some XML files are describing the expected outputs for each test case. The overall test engine is an ad-hoc stylesheet that runs the tests. The pass / fail decision for each case is based on the comparison of canonical forms of actual output and expected output. A first assessment allows to conclude that each test case could be scripted as a test assertion. When specialized operations are needed (like file canonicalization using infoset.xsl, or file comparison) these could be wrapped as xsl functions used in the test assertion expressions. Assuming a two-phase testing process (running XBRL processor, then validating results), the test assertion engine described here could handle the validation phase, which relies on XML documents.