How to cite this paper
Hamilton, Kate, and Lauren Wood. “Schematron in the Context of the Clinical Document Architecture (CDA).” Presented at Balisage: The Markup Conference 2012, Montréal, Canada, August 7 - 10, 2012. In Proceedings of Balisage: The Markup Conference 2012. Balisage Series on Markup Technologies, vol. 8 (2012). https://doi.org/10.4242/BalisageVol8.Wood01.
Balisage: The Markup Conference 2012
August 7 - 10, 2012
Balisage Paper: Schematron in the Context of the Clinical Document Architecture (CDA)
Kate Hamilton
Kate Hamilton is a Senior Information and Process Analyst at Lantana
Consulting Group. She has a long history in pointy-bracket modeling and implementation,
always with a focus on supporting new adopters
and on design that meets the needs of all users. Her degree is in the history and
philosophy of science, technology, and medicine. She also grows and markets
heirloom varieties of vegetables. These interests may be related.
Lauren Wood
Lauren Wood is a Senior Consultant with the Lantana Consulting Group,
working predominantly on standards in XML-based healthcare. She is also Course
Director for the XML Summer School. She has been part of the XML world for many
years, working first as an SGML consultant for the publishing and aerospace
industries, then on SoftQuad's authoring tools. In between she has been on many
technical committees and chaired the XML Conference for 5 years from 2001-2005.
At Sun Microsystems she was part of the identity and privacy group, taking part
in and managing research projects.
Copyright © 2012 by the authors. Used with permission.
Abstract
The Clinical Document Architecture (CDA), created by Healthcare Level 7 HL7 is widely used in healthcare. Its scope is any clinical document or
report. The (single) CDA schema that is used to validate all of these reports is
derived from a UML model. The element names reflect specializations of various
concepts, while the attribute values can refine element meaning, add flavor to the
parent/child relationship, reverse the subject and object of a compound expression,
negate the meaning, or explain the absence of a value. Separately-defined prose
constraints represent the requirements for individual document types such as a
Procedure Note or a public-health accounting of bloodstream infections. These
report-specific constraints are, of course, not defined in the general CDA.xsd
schema. Although the element-attribute relationships can be tested using the schema,
the value-driven conditional and alternative rules are best tested using Schematron.
We create Schematron and use it in conjunction with the CDA schema to confirm that
the CDA documents conform to the relevant specific report constraints and
requirements. The Schematron must itself be tested to ensure that the combination
of
W3C Schema and Schematron correctly checks the rules and that the Schematron error
messages point comprehensibly to the real error.
This paper presents the reasons for using Schematron for this validation, some of
the processes used to test the Schematron during its development, and
challenges.
Table of Contents
- Introduction
- Aspects of the CDA Model
-
- Uses of Attribute Values
-
- Relationships
- Negation
- Null Flavors
- The Price of Power
- Vocabulary for those Attribute Values
- Constraints, Value Sets, Alternatives and Conditionals
- Why Schematron validation is Needed to Supplement Schema Validation
- Creating the Schematron Validation
- Testing Schematron Validation
-
- Testing requirements
- The testing process
-
- Overview
- Details
- Testing challenges
- Conclusion
Introduction
The Clinical Document Architecture (CDA) Release 2 is derived from the Healthcare
Level 7 (HL7) Reference Information Model (RIM). It defines a formal model and semantics
for concepts found in a clinical document — acts, such as procedures, substance
administrations, and observations of laboratory findings; entities such as people,
places, devices, and drugs; relationships such as participation and causality; and
document metadata.
CDA is undeniably complex, covering as it does the full range of clinical documents
for multiple countries and regions. Its derivation from a Unified Modeling Language
(UML) representation of healthcare concepts reflects that complexity.
A CDA Implementation Guide (IG) defines a specific implementation of CDA by
specifying how to use these building blocks to express a particular kind of clinical
document — a report on infection in hemodialysis patients, or the history and
physical taken during a visit to a physician.
The CDA model is instantiated as one W3C Schema (version 1.0) that covers all these
related clinical report types (also called implementations). The model is very
expressive but there are technology-determined gaps between the UML of the model and
the
XML Schema expression of it, and then further gaps between the generic XML Schema
that
covers all clinical documents and the specific type of document described in a given
implementation guide for a specific purpose. Some of these gaps could perhaps be filled
if CDA used W3C XSD 1.1, or RELAX NG, to define the schema, but neither of those options
are likely to be possible in the near term, given the practicalities of implementations
of complex standards that are used across the world in critical healthcare systems.
People who are guaranteeing to a healthcare organization that the documents they
deliver contain the right information for a specific purpose, and expressed using
the
right syntax, need to know that the validation we provide for testing will pass all
good
files and fail all bad files. This means we have to test our validation mechanism,
and
that mechanism has to be in addition to the basic CDA.xsd (1.0) validation.
The validation mechanism we choose has to fulfill a number of criteria. It must be
easy for non-XML experts to use to test the files that come out of their
implementations, and it must be able to be used in many ways. It must cover the gaps
between
the prose definitions in a specific implementation guide and the generic XSD schema
that
is used for a large number of implementation guides as much as possible. The goal
is to
make validation available in appropriate forms to guarantee the quality of the XML
documents that the public health organizations receive from hospitals and other
healthcare organizations.
This paper introduces Clinical Document Architecture (CDA) concepts to show why
Schematron validation is needed to supplement schema validation, discusses how we
currently produce and test the Schematron validation, and explores some challenges.
We
are interested in other approaches to quality management and testing that we could
investigate to supplement our current methods.
Note
The documentation examples used in this paper are taken from the HL7
Implementation Guide for CDA® Release 2 - Level 3: Healthcare Associated Infection
Reports, Release 7 (US Realm) (HAI R7 IG), available at
http://www.hl7.org/dstucomments/.
Much of this work was carried out for the Lantana Consulting Group.
The authors appreciate the comments made by the anonymous peer reviewers as well
as Rick Geimer and Liora Alschuler from Lantana.
Aspects of the CDA Model
Those seeking to represent clinical documents in XML face the same choices as in many
other areas: where do we place the line between a centrally-mandated model that omits
data that only a few participants need to record, and a free-for-all in which no two
documents are modelled in the same way? CDA addresses this in a two-pronged approach
— the abstract model is expressed as elements; attribute values refine the
meaning, with heavy reliance on public vocabularies. These vocabularies are slightly
different than the ones often assumed in an XML context; they are not concepts described
in an XML schema but rather an ontology or set of codes that can be (and are) described
in many different formats. In many ways they play a role similar to that played by
elements in some schemas -- they associate semantic meaning with the data. Healthcare
professionals who specialize in vocabulary can spend just as long arguing over the
precise meaning of a
term that is to be defined in a codeset, or over which one to use in which context,
as XML
schema designers spend in arguing over what name to give a particular element in its
context. We will come back to vocabularies again throughout this paper.
What does it mean to say that attribute values refine the meaning of an
element?
CDA has two aspects: a text-heavy document aspect, called the narrative block, that
is
recorded in HTML-like elements; and an interoperable, machine-processable aspect with
more precise semantics, called clinical statements or coded entries. Coded entries
do not record presentational aspects such as section,
paragraph, table, list, or figure. Rather, they record specializations of the abstract
concepts of
entities, acts, and relationships. These specializations have XML
names: a participant is a specialization of the entity concept;
procedure and substanceAdministration are specializations of an act;
component-of, is-reason-for, is-cause-of are types of relationship. Some of these
examples are elements, others are attributes.
In many applications of XML, attribute values are not central to interpretation.
Some of us were taught, when first learning a pointy-bracket syntax,
that an element name classifies the content and an attribute provides
additional, secondary information:
<animal coatColor="brown">dog</animal>
In CDA attributes have a stronger role: rather than providing supplementary
information, they usually continue refining the taxonomic distinctions
made by elements.
In the procedure element below, the code element refines its parent's meaning
by specifying the kind of procedure, using a value from a specific vocabulary. The
vocabulary
is identified in the codeSystem attribute by a dot-notation object identifier (OID):
<procedure>
<!--ID of procedure -->
<id root="2.16.840.1.113883.3.117.1.1.5.1.1" extension="232323"/>
<code codeSystem="2.16.840.1.113883.6.96" code="423827005"
displayName="Endoscopy"/>
</procedure>
The refinement of meaning can have multiple levels. The previous example captures
"A
procedure; what kind of procedure? An endoscopy." The next example shows a
waterfall-like nesting of questions and answers: "A participant; what kind of
participant? A location; what kind of location? A service delivery location; what
kind
of service delivery location? A Medical/Surgical Critical Care unit."
<participant typeCode="LOC">
<associatedEntity classCode="SDLOC">
<!--ID of facility -->
<id root="2.16.840.1.113883.3.117.1.1.5.1.1" extension="9W""/>
<code codeSystem="2.16.840.1.113883.6.259"
codeSystemName="HL7 Healthcare Service Location Code" code="1029-8"
displayName="Medical/Surgical Critical Care"/>
</associatedEntity>
</participant>
CDA has two more general specializations of the act concept,
the observation elements and act elements.
<observation classCode="OBS" moodCode="EVN" negationInd="false">
<code codeSystem="2.16.840.1.113883.6.96" code="50373000"
displayName="Body Height"/>
<value xsi:type="PQ" value="180" unit="cm"/>
</observation>
This captures "An observation; of what? A body height; what height was observed? 180cm."
A consequence of the semantic role of attributes in CDA XML is that the words "value"
and "code" have several usages: the value element,
its value attribute, the value of that attribute (which may be a code), the value
of the code element's code attribute (which is always a code), or -- which is usually
clear from context -- the value of some other attribute. (Ordinary speech is similarly
challenged in distinguishing between the abstract concepts, which are UML classes,
such as Act, and XML elements in the CDA schema, such as act.) To cut through that
confusion, don't think about the XML first! Focus on the clinical content -- what
is being expressed? -- and consider the XML elements and attributes as packaging.
What is to be expressed? A body height.
That's an observation. An observation of what? body height (code element: code attribute).
What was observed? 180cm. (value element: datatype is physical quantity, value is 180, unit is cm)
Uses of Attribute Values
In CDA, attribute values can have implications for the node tree, primarily
through alternatives and through conditional requirements.
Relationships
The range of
relationships in clinical content goes far beyond
child containment, so the model interposes a wrapper that can carry
information about the relationship. Here we’re recording the micro-organism
cause of a positive blood culture:
<observation classCode="OBS" moodCode="EVN" negationInd="false">
<code code="ASSERTION" codeSystem="2.16.840.1.113883.5.4"/>
<statusCode code="completed"/>
<value xsi:type="CD"
codeSystem="2.16.840.1.113883.6.277" code="1955-4"
displayName="Positive blood culture"/>
<entryRelationship typeCode="CAUS" inversionInd="true">
<observation classCode="OBS" moodCode="EVN">
...
</observation>
</entryRelationship>
</observation>
One powerful attribute, @moodCode, expresses something akin to mood in
English verbs: it can change the sense of a substanceAdministration element from
prescription (an intent) to application (an event).
Negation
To record that something did not happen or was not done, CDA provides a
negation mechanism — this is also an attribute value. This patient experienced
no adverse reaction:
<observation classCode="OBS" moodCode="EVN" negationInd="false">
<code codeSystem="2.16.840.1.113883.5.4" code="ASSERTION"/>
<statusCode code="completed"/>
<value xsi:type="CD"
codeSystem="2.16.840.1.113883.6.96" code="281647001"
displayName="Adverse reaction"/>
</observation>
This has great expressive power when used in combination with relationships
(the cause of the fever was not the bacterium).
Of course, that is not the same as not
knowing whether the cause of the fever was the bacterium....
Null Flavors
Unlike many paper forms and database tables, CDA makes a strong distinction between
a value and
the reason a value is not recorded. Such reasons are recorded in a @nullFlavor
attribute. Here, we haven’t asked for the patient’s birthdate (perhaps the
patient arrived unconscious and without his wallet):
<dateOfBirth nullFlavor=”NASK”/>
For an elderly person living in a remote village, the appropriate nullFlavor
might be “UNK” — the question was asked, but
the answer wasn’t known.
The Price of Power
The price of this expressive power and interoperability is complexity, of course.
Nevertheless this provides a reasonably concise expression of the very large world
of clinical documents that is the model's scope. Every layer has a role; any
collapsing of the model leaves some body of information out. The act relationships
and moods are elegant: convert an intent to administer tachytherapy into a report
of
having done it by changing the moodCode from INT to EVN, and look forward to
processing the a volume of XML documents to compare the number of intents to the number
of events, or to
report on the average elapsed time between intent and event.
Vocabulary for those Attribute Values
Controlled and widely-used vocabularies are crucial to making this approach work.
There are several vocabularies covering every aspect of healthcare, from units of
measure to precise descriptions of body parts as a surgeon would view them). These
public vocabularies are crucial for interoperability.
There are many public vocabularies in the healthcare realm: for example, SNOMED
CT is a core general terminology with more than 311,000 active concepts
organized into hierarchies that is commonly used for clinical findings and body
parts; RxNorm provides
normalized names for clinical drugs and ingredients. There are, of course,
overlapping vocabularies with concepts that almost, but not quite agree with each
other, so in practice many healthcare systems need to support multiple vocabularies
to cover all the cases.
These vocabularies are made available under differing licensing terms, and in
different formats. For the schematron testing purposes we create custom XML files
with only the terms (codes) that are relevant to the specific implementation guide.
The format we use has entries like this:
<code value="413495001" displayName="ASA physical status class 1"
NHSNdisplayName="Normally healthy patient"
codeSystem="2.16.840.1.113883.6.96"/>
<code value="413496000" displayName="ASA physical status class 2"
NHSNdisplayName="Patient with mild systemic disease"
codeSystem="2.16.840.1.113883.6.96"/>
<code value="413497009" displayName="ASA physical status class 3"
NHSNdisplayName="Patient with severe systemic disease, not incapacitating"
codeSystem="2.16.840.1.113883.6.96"/>
<code value="413498004" displayName="ASA physical status class 4"
NHSNdisplayName="Patient with incapacitating systemic disease, constant threat to life"
codeSystem="2.16.840.1.113883.6.96"/>
<code value="413499007" displayName="ASA physical status class 5"
NHSNdisplayName="Moribund patient, < 24-hour life expectancy"
codeSystem="2.16.840.1.113883.6.96"/>
</system>
The many-digit numbers are globally unique object identifiers (usually abbreviated
as OID). These identifiers are the preferred method of identifying objects in HL7
standards such as CDA, and are used for everything from sets of vocabulary (e.g.,
the ValueSet definition above) to chunks of the implementation guide, known as
templates (referenced in the pattern id in the schematron snippet). HL7 has an OI
registry, available at http://www.hl7.org/oid/index.cfm, with more information about the design
and use of OIDs.
One thing to note about OIDS: there is something of a structure in that the
left-most number is considered the root and the right-most number the leaf node on
the tree. OIDs are assigned to organizations at a particular sub-tree level, and how
that organization chooses to arrange its sub-tree depends on that organization. It
may choose to have a logical structure for its OIDs, or not.
Constraints, Value Sets, Alternatives and Conditionals
In any healthcare record there are rules about which information must be present.
In CDA convention these are represented as constraints on the CDA model. The
constraints have a formal prose representation that is published in a document called
an Implementation Guide because it defines an implementation of CDA.
For example, an observation representing an
adverse reaction:
5. SHALL contain [1..1] code (CONF:11542).
a. This code SHALL contain [1..1] @code, which SHALL be selected from ValueSet
2.16.840.1.114222.4.11.3391 NHSNAdverseReactionTypeCode DYNAMIC (CONF:4698).
We generate this prose representation from a database. A constraint is associated
with
a context (observation) and is recorded in data such as "conformance verb" (SHALL),
"value",
"value conformance", and "value set".
A value set is a set of coded concepts, drawn from one or more public vocabularies,
that are appropriate for the context. In the example above, the value set members
are types
of adverse reaction. The concepts in the previous example, showing patient status,
are members of a value set named ASAClassCode.
Constraints that express alternatives are common in some implementations of CDA.
One necessary usage is to require that
a code element contain either (a) both the code
and codeSystem
attributes OR (b) a nullFlavor
attribute.
Value-driven conditional rules arise for specific content situations;
for example, if the procedure being recorded was a cesarean (the relevant code/@code=2115-4
),
the report must also specify the estimated maternal blood loss.
Why Schematron validation is Needed to Supplement Schema Validation
As we've seen, much of the meaning of a CDA document resides in an element's attribute
values, which are used to
-
refine the meaning of those elements (rather than merely to describe the
object, as in <dog color=”brown”>
)
-
expand the varieties of relationship beyond what’s available from the XML
tree,
-
vary the verb mood,
-
switch the subject and object of a compound expression,
-
negate, and
-
explain the absence of a value.
These tools can build remarkably complex sentences. “Marie’s grandmother, who is
her
legal guardian, said Marie had pneumonia when she was six, which went untreated and
is a
possible explanation for the scarring on her lungs; however, Marie’s mother denied
this
and her father was unsure.”
In any healthcare records there are report-specific rules about which data must be
present. Since so much of the content in CDA is recorded in attribute values, these
rules amount to value dependencies, which are not adequately expressible in W3C Schema
validation. Some of the report-specific rules could be tested with a custom W3C Schema,
but not all, and, in practice, many of the most important report-dependent rules cannot
be checked by even a custom W3C Schema.
The two main problem areas for validation are alternatives and value-dependent conditionals.
As we saw above, one commonly-used construct in CDA is to require that a code element
contain either the code and codeSystem attributes (with optional displayname and
codeSystemName attributes), OR a nullFlavor attribute. The CDA Schema allows all the
relevant attributes to appear on the code element, in any
combination. As a result, a valid document instance might populate the code attribute
without
the codeSystem attribute, or populate both the code and nullFlavor attributes. Both
combinations are inherently meaningless, but the CDA Schema
can’t check for them.
The conditional rules that arise for specific content situations can be expressed
as
if [some XPath] then [some other XPath]
For example,
If procedure/code/@code="1234" (a specific type of procedure), then performer/id must be present.
Since the type of procedure is recorded as an attribute value, even
a custom XML Schema can’t check this requirement.
Schematron covers the gap. We use a two-step validation approach: first against the
CDA XML Schema file (CDA.xsd), and then against a Schematron file and custom vocabulary
file that tests the rules that cannot be expressed in the
CDA.xsd. Currently we are using Schematron 1.5 and XPath 1, for compatibility/historical
reasons; we are gradually moving to ISO Schematron and thence to XPath 2.
Creating the Schematron Validation
A typical schematron section looks like this:
<sch:pattern id="p-2.16.840.1.113883.10.20.5.6.41-errors" name="p-2.16.840.1.113883.10.20.5.6.41-errors">
<sch:rule context="cda:observation[cda:templateId/@root='2.16.840.1.113883.10.20.5.6.41']">
<sch:assert test="count(cda:statusCode[@code='completed'])=1">shall contain 1..1 statusCode=completed
"Completed" (CodeSystem: 2.16.840.1.113883.5.14) (CONF:2282)</sch:assert>
<sch:assert test="count(cda:code[@codeSystem='2.16.840.1.113883.6.1'][@code='41852-5'])=1">shall contain
1..1 code=41852-5 "Microorganism Identified" (CodeSystem: 2.16.840.1.113883.6.1) (CONF:2281)</sch:assert>
<sch:assert test="@classCode='OBS'">shall contain 1..1 @classCode=OBS "Observation"
(CodeSystem: 2.16.840.1.113883.5.6) (CONF:2279)</sch:assert>
<sch:assert test="@moodCode='EVN'">shall contain 1..1 @moodCode=EVN "Event"
(CodeSystem: 2.16.840.1.113883.5.1001) (CONF:2280)</sch:assert>
<sch:assert test="cda:value[@xsi:type='CD']">shall contain 1..3 value, which SHALL be selected from
ValueSet 2.16.840.1.114222.4.11.3194 STATIC (CONF:2283)</sch:assert>
</sch:rule>
</sch:pattern>
This shows the standard set of attribute/attribute value testing, and an example of
the vocabulary testing that is such an important part of testing the validity of CDA
documents.
Testing Schematron Validation
Testing requirements
The first reaction of most people is to wonder why you test the Schematron, when
everyone knows the Schematron is used to test the document instances. The second
reaction is to say “of course” — we need to ensure that the combination of XML
Schema and Schematron that we create correctly confirms the validity of all the
files that conform to the constraints, while flagging any errors.
We also need to ensure that all Schematron error messages point to the real error
that needs to be fixed. Many of the parser error messages that arise from validation
errors are difficult to understand, particularly for those who are not XML experts.
This adds a testing requirement — not only does the Schematron have to fail a
bad test file, but it has to fail it with an understandable message.
As we discussed earlier, the CDA schema does not
catch quite all the CDA errors people might make, and of course it does not validate
rules that constrain CDA; so our testing concentrates on the those aspects of the
Schematron
validation.
The database that stores the constraints and exports prose for the Implementation
Guide
also generates the basic schematron validation file.
We make the most of the fact that Schematron error messages can be more tailored:
the error message we generate cites the constraint prose just as it appears in the
Implementation Guide.
In general the items that the CDA XML Schema doesn't validate,
such as alternative attributes and value-dependent conditionals, currently can't be
recorded as computable constraints and thus can't be
generated from the database. These aspects of validation are written by hand, and
therefore need to be thoroughly tested before delivering the resultant Schematron
file to the customer. Since the automatic generation process is constantly beng
developed and upgraded, we also need to do regression testing on new releases of the
Schematron export.
When we are creating Schematron for a client, as opposed to testing the
Schematron generation system ugrades, the automatically-generated Schematron
requires only spot checks to make sure nothing went wrong with the generation. The
hand-coded portion requires far more in-depth testing.
The testing process
Overview
We use the phrase “good-test files” for the test files that ought to pass
validation. “Bad-test files” are those which should throw errors.
The first step is to create good-test files — combinations of elements,
attributes, and attribute values that are allowed. Once the Schematron passes
those correctly, it’s time to ensure it fails the bad-test files, and fails them
in the correct way. This is where the bulk of the testing work comes in.
We take the good test-files as a base, and make them incorrect by deleting a
required element, or setting an attribute value that is not allowed at that
point. The Schematron has to fail all the bad-test files, with a reasonable
error message, at the right spot. Then we create more bad-test files, taking the
documentation of the report type as a guide, to cover any combination of wrong
choices that a user could make.
We don’t want to go overboard with the number of test files though, as we do
need, at some stage, to ship the Schematron. Thus we are constantly looking for
ways to ensure better and more time-efficient coverage of the possible error
conditions while maintaining confidence of complete coverage. One approach is
to create a complete set of alternatives
for each constraint we want to test. That gives us perfect confidence on coverage
(with some challenges for tracking which file tests which condition) but requires
a
very large number of test files. We add efficiencies by making carefully-chosen
assumptions such as:
The mechanism that generates a test for membership in a value set is applied
in the same way wherever used; we will only test it once. This significantly reduces
the
number of test files we must create, but having a dozen or so such rules complicates
the
business of drawing up a test list.
Details
Creating the good-test files requires a solid understanding of the report
rules. In practice, this means that creating the good-test files also tests the
quality of the documentation. The best way to test this is for the person
who creates the test files to be someone other than the person who wrote the
documentation.
The prose representation of a CDA constraint has precise meanings for
every part of the sentence. The constraint is documented in reference to an XML
fragment in CDA, and the element and attribute combination are defined in terms
of XPaths. Each constraint has a conformance number (the 10304 and 10907 listed
here). In the following snippet, the topic is the administration of a drug:
SHALL contain consumable/manufacturedProduct/manufacturedMaterial/code (CONF:10304).
a. In an Evidence of Infection (Dialysis) Report,
i. If the antimicrobial started was Vancomycin,
the value of @code SHALL be '11124' Vancomycin [CodeSystem: 2.16.840.1.113883.6.88 RxNorm].
ii. Otherwise, the value of @nullFlavor SHALL be 'NI'. (CONF: 10907)
This specifies: in the context of this substance administration, there must be a
code element
at the bottom of that XPath chain. If the report is of type “Evidence of
Infection (Dialysis)”, we're only collecting statistics about one drug: if the antimicrobial
started was Vancomycin, then the
code element must have an xsi:type attribute with value ‘CE’, a code attribute
with value ‘11124’, and a codeSystem attribute with value
‘2.16.840.1.113883.6.88’. (It may also have a displayName attribute with value
‘Vancomycin’ and/or a codeSystemName attribute with value ‘RxNorm’.) Otherwise the
code element has neither of those attributes; instead it must have the attribute nullFlavor
with value ‘NI’.
To test this, we need a good-test file that shows when Vancomycin was
started, a good-test file that shows when it wasn’t Vancomycin, but something
else, and a good-test file that shows when no IV antimicrobial was started. That
makes three separate good-test files, and at least that number of bad test files
to test the number of ways in which things can be wrong.
This is a relatively simple constraint; we have some that run to several
alternative branches, each with a number of options. Independent values can be
tested separately (recording which antimicrobial was started does not depend on
whether the person administering it washed their hands), but some values
interact even though widely separated in the CDA document (maternal blood loss
after a caesarian section cannot logically be recorded for a male patient, nor
can a male patient logically be treated in a pre-natal ward).
Ensuring reasonable test coverage is one of the big issues we face. For one
project we created 91 good and 133 bad test files by hand to test one
medium-sized chunk of a report. Not surprisingly, we’ve started looking into
ways to generate test files as variations of base sample files to save time. We
also don’t want to create more test files than we need; testing unnecessary
combinations costs time and effort, delaying delivery without benefitting the
customer.
We need to itemize the combinations and variations, determine which we will
test, and ensure that at least one good and one bad test file exists for each of
those combinations and variants that can't be tested by the CDA Schema. We need
to create a few bad test files to ensure the CDA.xsd validation is triggered by
the validation process, but not for every possible error. We need to be sure
that the error message thrown by the Schematron for each bad file is reasonable;
this involves keeping track of the file name and the error message(s) through
iterative development cycles.
Currently we keep track of the files in a spreadsheet, using
carefully-constructed filenames that indicate what is being tested. (The
file-naming conventions aren’t strictly necessary, but they are a significant
practical help to the person creating the test files and the person checking the
test files.) The test files are commented to show what is being tested, and
where the error (if a bad test file) is. The comments reference the conformance
numbers in the documentation. We track and improve error messages that are
inaccurate or unhelpful. The spreadsheet goes through iterations to match the
Schematron development iterations.
Testing challenges
We have a number of challenges apart from creating the test files. Many of
these are typical testing challenges, such as how to assess the results quickly,
and what the best type of testing system is (which depends to some extent on
what the various developers are used to).
A bigger issue is the best way to indicate how many errors should be present
in a given bad test file. Any given single error in an XML document can
potentially be the cause of more than one error message. When the Schematron
finds a different number of errors in an XML file to the expected number, it
could be due to error(s) in the Schematron, or error(s) in the XML. Once the XML
file has been checked, what remains are the Schematron error(s). And the number
of those may change as the Schematron is developed.
There are a few ways to tell the error-testing system how many errors go with
a particular file. Tony Graham published a poster at XML Prague 2012
(http://www.mentea.net/resources/schematron-testing-framework.pdf) discussing
his framework, which uses processing instructions in the XML file itself (along
with XProc). We use a spreadsheet with the filename and expected number of
errors (along with some custom Java code that invokes the standard JUnit testing
framework). Both systems then use ant to run the tests against all the test
files and report results. If the number of expected errors remains stable,
either method should work well.
Conclusion
We have shown why we need to test Schematron files in the context of healthcare
standards, how we use the standard testing methods of good files and bad files, and
discussed some of the challenges we find. We welcome feedback, suggestions for
improvement in the process, and comments.