Haupt, Stefanie, and Maik Stührenberg. “Automatic upconversion using XSLT 2.0 and XProc: A real world example.” Presented at Balisage: The Markup Conference 2010, Montréal, Canada, August 3 - 6, 2010. In Proceedings of Balisage: The Markup Conference 2010. Balisage Series on Markup Technologies, vol. 5 (2010). https://doi.org/10.4242/BalisageVol5.Haupt01.
Balisage: The Markup Conference 2010 August 3 - 6, 2010
Balisage Paper: Automatic upconversion using XSLT 2.0 and XProc: A real world example
Stefanie Haupt
Stefanie Haupt is currently finishing her education for an M.A. degree in Literary
Criticism, Text Technology, and Sociology at Bielefeld University. Her main research
interest
focuses on markup and schema languages, with emphasis on XML databases and querying.
Maik Stührenberg
Maik Stührenberg studied Computational Linguistics at Bielefeld University. After
working
for four years as research assistant at Giessen University in different text-technological
projects, he is now a Ph. D. student and research assistant at Bielefeld University.
His main
research interests include XML schema languages and specifications for structuring
and querying
multi-dimensional annotated data.
All too much of the data on the Web appears in unstructured presentation-centric formatting
that isn't suited for structured searching and retrieval. Upconversion to a more data-centric
information storage format offers a potential for many new uses of the data. The starting
point
of our work is a collection of HTML documents containing video game reviews. Our goal
is to
describe a target XML format that supports certain elements and attributes containing
information that we consider valuable. Furthermore, the conversion process itself
should be
carried out automatically by means of an XProc pipeline. We conclude our paper with
a
demonstration of typical benefits of the highly structured data that results from
our
conversions.
Vast collections of information are stored in HTML files distributed over millions
of Web
pages through the Internet. Among these quite valuable data can often be found; however,
HTML
does not offer a large pool of semantically motivated elements or attributes for annotating
arbitrary data, since the language was originally created for hypertexts. Although
CSS
microformats Suda (2006) may be used to add semantic value to structuring
elements (e.g. div and span), most information is buried underneath a
"tag soup" of td, p or div elements that allow no
inference about their content. In contrast, we can have information that is highly
structured in
terms of very specialized XML markup using a document grammar (DTD SGML, ISO 8879:1986,
XML 1.0, XSD XML Schema Part 0: Primer or RELAX NG RelaxNG, ISO/IEC 19757-2:2003)) that allows for easy retrieving of very specific information. A real
world example where the origin of our data is a collection of (sometimes even invalid)
HTML 4.01
HTML 4.01 Web pages storing documents of video game reviews is a good
candidate for demonstrating how value can be added through better markup. Our goal
is to
transform these into fully structured and valid XML instance documents that allow
different
queries about the information. Since we are confronted with several hundred reviews,
an automated
conversion process is valuable. As an additional goal, we would like to stay in the
realm of XML
techniques; for example, we would like to avoid using non-XML-aware software such
as
general-purpose scripting languages (e.g. Perl, Python).
The data
Information content
Video games are a part of today's culture and are available in a huge variety in terms
of
supported game system, genre and — of course — quality. Finding a game that fits both
one's
hardware requirements and favored genre is a relatively easy task to accomplish, but
basing the
decision to buy a specific game only on the text written on the back of its case is
daring at
least. Impartial (more or less) reviews of video games may help to clarify if the
money is well
spent in the long run by providing rating systems for features such as graphics, sound,
atmosphere or overall score (usually higher scores are better). The team of the German
Mag'64 Web site [1] has tested video games for over eight years, gathering over 1500 reviews, each
consisting of a single HTML Web page. Each document contains information about the
game being
tested, the review, including a general judgement, and images and screenshots. This
information
is quite valuable since among the provided items are general ones such as the title,
system, or
publisher, but in addition more specific items such as number of players, genre, age
rating and
difficulty. The review consists of running text' while the final verdict and pros
and cons are
summarized in a tabular view. The data we have to deal with consists generally of
two types of
reviews, which we call "Type A" and "Type B". Type A was used during the years 2001
through
2004, while Type B was introduced in the Autumn of 2004.
Technical analysis
From a technical point of view the data is stored in HTML Web pages. Because HTML's
original task is to structure hypertexts, it lacks specific elements and attributes
for
annotating the information we are interested in. Furthermore, the markup of our test
data is
very focussed on presentation, that is, general HTML elements such as div,
p, td are used for physically structuring the information according
to a given layout. While the two review types, A and B, do not differ regarding their
information content, there are differences in the markup techniques used.
Type A
The Type A review was originally used as part of an HTML frameset. While one frame
contained a menu for navigating through the whole service, the second frame stored
a single
review in the form of a HTML Page. This page lacks an HTML Doctype declaration, and
typical
copy and paste errors can be found, including end tags without preceding start tags,
wrong
attributes, etc. The img element for embedded graphics lacks the required
alt attribute. [2] Furthermore, no information about the character encoding is given, which leads to
encoding errors since German umlauts and other special characters were used.
This markup we have to deal with is very presentation-focussed: semantic markup such
as
h1 or h2 that could be used for structuring the text is not used at
all. The title of the game can only be found in the running text or in the graphic
image
referred by the img element — and sometimes in external cheats or tricks documents
that are referred to from the review page (the term "CHEATS: JA" in Figure 2). All useful information is buried deep inside HTML's table elements, and the
page lacks any meta elements for storing additional information. Spacing between
different parts of the text was introduced by using HTML's entity, while the
whole
markup is layout oriented, using font, i and u elements.
Sometimes font elements with identical formatting options are embedded into each other
resulting in a tag soup. Emphases are arranged solely by selecting "size 3" fonts.
The running text of the review is distributed among different table elements,
establishing a print-like layout. Each review begins with two blocks containing
meta-information, such as system, genre, number of players, etc.
The Type A review ends with a tabular overview, consisting of the "pros" and "cons"
of the
game.
Type B
The Type B reviews were established in the Autumn of 2004, coinciding with the release
of
the Nintendo DS® handheld console. Since this
videogame console introduced some features that were unknown before (e.g. split-screen
and the
stylus input device), a new HTML template for reviewing video games was adapted. As
a new
meta-information item, an age rating was added, and the running text was subdivided
by
headings.
Most of the HTML pages contain a doctype declaration (incorrect for HTML 4.01), a
reference to an externally declared CSS stylesheet and information about the character
encoding
(ISO-8859-1 — although the specified encoding is sometimes not correct, since some
documents
are encoded using the Windows-1252 charset or even UTF-8). In addition to the external
CSS
file, local formatting using attributes such as marginwidth, bg-color
or border can still be found. In general, the HTML pages are not valid according
to the W3C validation service. Figure 3 shows the mixture of different
formatting options used.
A positive difference from the Type A is the fact that the title of the game appears
(together with the platform it was released for) in HTML's title element.
Important information such as price or age rating are hidden inside a single div
element (Figure 4), divided by line breaks.
In contrast to the Type A reviews, subheadings are included; however, these are not
marked
up by HTML's inherent h1 through h6 elements but by using formatting
elements such as b and font.
Both review types show HTML's inherent lack of support for highly structured data.
Although our example application deals with document-centric texts, the data under
observation
contains important information that should be marked up explicitly.
Highly structured data
Our goal is to create an XML markup language capable of structuring the video game
reviews
of both Type A and B that have been discussed. This format should be used as representation
format for the output of the conversion process that will be presented in the section “Upconversion” and could be used as a storage format for future review applications.
Since we have already stated the input documents are often invalid (sometimes even
not
well-formed) and important information is buried inside HTML table elements, having
a document grammar for both validating the conversion process's output format and
providing
explicit markup of the important information is quite important for us. For these
reasons, the
use of a capable of full text search engine was not taken into account. We have chosen
XML schema
in favor of XML DTD because of its datatype library and especially for its support
of
user-defined simple and complexTypes Walmsley (2002). A RELAX NG schema (in
combination with the XML schema datatype library) would have been another option,
however, the
broader support for XML schema supplied by the XSLT processor used during the conversion
process
tipped the scales for us (Figure 5).
Each game can be identified by a unique xml:id attribute, further optional
attributes correspond to genre and subgenre, supporting an enumerated
list of possible values which should help avoiding typical errors such as typos. Children
of the
game element are the title and platforms elements, the
latter consisting of at least either one handheldGameConsole or
videoGameConsole, allowing to combine reviews of the same video game released on
multiple platforms [3]. Both elements are derived by extension of the globally declared complexType
consoleType, sharing common information present in stationary and handheld game
consoles (see Figure 6 for a graphical overview of the shared
information).
The release element stores information about the date of release (using an
xs:date Type Attribute), the different languages and price. Children of the
languages element are spoken, text and
handbook elements, depicting information about the parts of the game that have been
translated. The price element has a currency attribute that uses an
enumerated list of possible values according to ISO Country Codes, ISO 4217:2008.
An optional image element can be used to represent box pictures or screenshots
of the game reviewed.
As mentioned above, the handheldGameConsole and videoGameConsole
elements are derived from the complexType consoleType by extension. Although the
additional elements techSpecs and saving use the same names, their
content models are different with respect to the video game console, since, for example,
the
requirements for storing save games are different between handheld and stationary
consoles. Only
the videoGameConsole element allows for the compatibleInputDevices
child element. Most of these elements use enumerated lists to eliminate possible typos
and to
ease the acquisition of new reviews.
The main part of the review is stored underneath the review element that
consists of the mainText and conclusion elements and further optional
screenshots and that has a date attribute and an author attribute
group. The running text is subdivided into optional headers and paragraphs, allowing
a fine
grained division of text parts and representing both review types.
The conclusion element is used to store both further text (e.g. in a form of a
final verdict similar to the Type B reviews) and the tabular-like lists of pros and
cons,
followed by the final score element. Scoring can be expressed either via numeric
values (using the percent child element with its attributes graphics,
sound (optional), multiplayer (optional) and overall) or
through text, since both variants can be found in our sample data.
This grammar can not only be used to store the information coded in both review types
but
also is highly flexible for future extensions. Possible future extensions of the schema
may
include XSD 1.1 assertions, for example, to ensure that multiplayer scoring information
is only
allowed when the maximum number of players is greater than "1".
Upconversion
Our upconversion process begins in the typical manner by using XSLT 2.0 / XPath 2.0
Kay (2008). Because it requires multiple steps and must be applied to many files, we
have encapsulated it in XProc.
XSLT 2.0 benefits
In his paper "Up-conversion using XSLT 2.0" Michael Kay points out the great advances
XSLT
made when shifting to XSLT 2.0, and he provides a real-world example that makes heavy
use of the
new features. The key features which produce benefit for upconversion are in short
schema-awareness, support for regular expression processing, better manipulation of
strings, and
advanced grouping possibilities. So tasks that formerly were often solved by using
a general
purpose scripting language like Perl or Python, by loading XML modules can be done
equally well
or better with XSLT 2.0 [See Kay (2004) for an elaborated example]. Our
upconversion of the reviews mostly makes use of regular expression processing and
string
manipulation.
The documents are preprocessed into well formed XML using HTML Tidy. [4] For the upconversion, both functions as well as named templates are used widely.
The
following snippet demonstrates the massive clean-up the stylesheet performs. It is
taken from
the extensive main template, which uses a variable to hold the string with
information about the genre of the reviewed game (Figure 7). This
string is checked for both Type A and Type B data equally but it is applied differently
with
respect to the structure.
This variable is then checked against regular expressions to assign the respective
value from the defined enumerated list. Figure 8 demonstrates the
assignment of some genres and a sub genre, implemented using case differentiation
that takes
advantage of the order of the test expressions.
Because the data varies a lot throughout the transformation, many case
differentiations are used. To find the title of some documents information stored
into external
documents has to be taken into account. In Figure 9, a linked "cheats"
or "tips" document is accessed to extract the game title that is hidden in the backlink
to the
review document.
Throughout the transformation many more requirements are met in carrying out the
upconversion. The examples above are simply illustrative of the process without going
into
complete detail.
Pipelining with XProc
XProc a new standard for automating processes like ours through an XML pipeline has
been
developed by the W3 working group XProc. It has reached the status
of W3C Recommendation on 11 May 2010 after being advanced to Proposed Recommendation
in March
2010. The specification had been downgraded from Candidate Recommendation to Working
Draft again
in January to solve some issues. It has reached a fairly stable level now, and a book
on XProc
by Norman Walsh is in progress.[5] For our desired all-in-one XML solution, XProc is first choice to handle the
pipeline.
The pipeline should process the documents that are stored locally in the filesystem
recursively (Figure 10). There are documents other than game reviews
(e.g. cheats and tricks), and we need some of them to extract the titles of games,
but most of
these documents are discarded. One problem here is that while we can say from the
filename what
is most likely not a test, but not what actually is.
The pipeline will apply the following tasks to each HTML document:
Use HTML Tidy to transform the HTML input into well-formed XML
Apply the XSLT script to the output of the former task using an XSLT 2.0
processor
Validate the output files according to the XML schema
Separate valid from invalid documents
Provide a log of valid documents
XProc suits these needs well, and, as an XML language, ensures perfect XML
compatibility. For processing we use XML Calabash version 0.9.21.[6] As another option, Calumet 1.0.11,[7] was taken into account, but since Calumet currently does not support XPath 2.0, we
stick to XML Calabash. We prepared the documents so the encoding of the files is either
ISO-8859-1 or UTF-8 and the special characters are masked as numeric entities for
the moment.
Otherwise there would be encoding errors in the result XML documents. Since the pipeline
shall
take HTML documents as input and shall process all of them in sequential order some
preparatory
steps are used to make the documents accessible inside the XML pipeline. Figure 11 provides a simplified overview of the first steps of the
pipeline.
We chose p:declare-step as root element for good control of input and
output ports. Both are set to allow any number of documents. Since parameters are
to be used for
XSLT transformation, we need the optional input port "parameters" - because it is
the only
parameter port in the pipeline it is primary by default. The source directory HTML
is bound to a
variable and made accessible for the step p:directory-list, which here returns the
system-folders in c-namespace (Figure 12).
To advance deeper into the structure we use nested p:for-each loops; of
course, the output port needs to be set to accept sequences. Next we list the subdirectories,
consisting mainly of game-folders (Figure 13).
Now we loop over the game-folders (not shown due to space restrictions) and prepare
the files for accessibility. First we add the base-uri to get the complete filepath
using
p:make-absolute-uris. Then we add slashes using p:string-replace to
ensure accordance to the file protocol. To make sure the file is accessible for the
p:http-request step we rename the element c:file to
c:request.[8] Furthermore, we need to add the proper attributes for the
p:http-request step to work. Since there is no server involved and we do not want
to work with binary data, we need to add the attribute override-content-type and
attach the value text/html (Figure 14).
Now we can process the HTML documents in sequence. We use a filter to exclude
documents which are not reviews and will not help us to find game titles (Figure 15). These documents may be reader reviews that follow no
certain structure, hardware reviews, or other texts. Files that may help us to find
missing game
titles contain these abbreviations: opt|chea|tipp|herz|guid|pass.
For the filtered documents, the second and, therefore, the main part of the pipeline
is initiated (Figure 16). If something goes wrong during the
upconversion, we want to be able to check in which step and what the reason may be,
so each of
the main steps has its output stored apart from each other. We nest try-catch
clauses to ensure the flow of the pipeline.
The variable file holds the URI of each file. It will be available
throughout the loop and not only serve to get each file but to store each file in
its given
folder. So first we convert these files that pass the filter through HTML Tidy via
p:exec, which can take non-XML input and provides safety (Figure 17). We could use p:unescape-markup in conjunction
with Tagsoup 1.2[9] or HTML Tidy as an alternative solution here, but as XML Calabash so far only
implemented Tagsoup for reading HTML and the results from HTML Tidy and Tagsoup differ
slightly,
we stick to p:exec. Calumet supports both HTML Tidy and Tagsoup for this step, but
as we are using XPath 2.0 we cannot use this option. We set source-is-xml to false
and result-is-xml to true. By default, result lines are wrapped, and the output of
this step is also wrapped to ensure wellformed XML documents on the output port. We
negate
wrap-result-lines and unwrap the output of the step. (Note that the arguments for
HTML Tidy need to be in a single line.)
The output of this step is saved to folder "Tidied" as "filename.xml" and chained
to the
next step p:xslt. As a precaution, this step along with the connected saving
procedure is encapsulated into a try group. If any of this fails, we record the tidied
file to
the folder "Transform-failed". The p:xslt step takes three input ports, one for the
stylesheet, one for the XML document and one for parameters (Figure 18). The filepath needs to be provided to the stylesheet to ensure reaching the documents
that
will be consulted for missing titles. The filename and system folder are processed
inside the
transformation as well.
If the transformation and the saving process can be executed successfully, the output
of
this step serves as input for p:validate-with-xml-schema (Figure 19). Depending on the output of this step, the documents are
saved separately. Valid documents can be found in the 'Schema-Valid' folder and the
invalid in
the 'Schema-Invalid' folder. (During the programming of the XSLT-Transformation, invalid
documents give hints for expressions in need of improvement.)
The last steps of the pipeline follow after the loops and take the result of the
loop
started in Figure 13. Here we create an XML document which takes the
c:result elements returned by the step directoryloop and lists them
for an overview (Figure 20).
This pipeline takes approximately half an hour to process the data, and is relatively
independent of CPU speed on an average actual system. It results in 1573 schema-valid
files.
The result of the upconversion process
Figure 21 shows an excerpt of an instance coded in the target output format
according to the XML schema. The critical information is marked up with the help of
appropriate
elements or attributes. Conversions of a game (i.e., the release on different platforms)
are
supported, as well, by separating the general information such as title and genre
from the
platform for which the review is written. The verdict contains the list of "pro" and
"con" items
and the score (depending on the input review type, subdivided into single figures
for game
graphics, sound, multiplayer and overall) in a highly-structured form that allows
easy access to
relevant criteria.
Benefits of highly structured data — searching for the game according to your
flavour
The result instances of the automatic upconversion process discussed in the section “Upconversion” contains highly structured information. All relevant and important data
that was formerly hidden inside HTML's table element or as part of the running text
can be accessed via XPath or XQuery expressions Chamberlin et al. (2004), allowing for
easy retrieval of reviews of games of certain types or according to certain criteria
such as
genre, price, and score. While the original structure of the Mag'64 Web site offered access to the review based on either the video game system
or the name of the game, a full-text search engine was not implemented. We have
developed some sample XQuery queries that allow for a different kind of retrieval
of game
reviews.
Alternative access to the reviews
The query genres.xq uses two parameters, genre and platform, to search for
games of a certain genre on a specific platform by using a collection of all valid
XML instance
documents. Figure 22 shows the output of the genres.xq
with the value "Wii" for the platform parameter and the value "Puzzle"
supplied for the genre paramater. Since this query was originally developed as a alternative
access mechanism, the information returned is very sparse. However, in combination
with
(X)HTML output containing hyperlinks to the respective review page, it would be
sufficient.
Finding a game according to specific features
Sometimes a user searches for games that support certain technical features, such
as online
content, multiplayer, etc. The techspecs.xq query uses the parameter platform and
techspec to retrieve only the reviews of games that include the provided feature.
Figure 23 shows an example result.
A more elaborated example: a wish list
Kids love video games these days, and often they leave their parents behind when it
comes
to choosing the right game for a present. We will demonstrate the benefits of highly
structured
data in this example. Consider a seven-year-old child with a Nintendo DS® who wants to get a racing game for his system. The parents might agree
but formula additional constraints: the game to be bought should have a score of at
least 70%
and should be appropriate for kids of his age. Furthermore, the difficulty should
not be too
high.
For this query different parameters have to be taken into account: the platform, the
genre,
age rating, score, and difficulty. The shoppingList.xq query provides all these
parameters (Figure 24). Using Saxon as XQuery processor with the
following call results in the output shown in Figure 25.
The results are sorted according to the score in descending order (with 100 representing
the best value). Each instance element contains the age rating, score, and
information about the difficulty, encoded in attribute values. Child elements are
the title and
the review notes, consisting of the "pros" and "cons" of the game. The notes
element, in particular, may contain information that is subjective; it may occur that
our
example parents will judge a certain feature higher or lower than the reviewer did
(or even
think of a "con" as a "pro").
Conclusion
The results of our work are of many kinds: first, the newly introduced features such
as
regular expressions and string manipulations qualify XSLT 2.0 as a full-fledged conversion
tool
for transforming weak structured data into a highly structured format. Second, if
a
transformation process has to be carried out multiple times and if other processing
is involved,
automation by using the XProc pipelining language is highly recommended. Both the
XProc
specification and the supporting software tools are ready for a productive environment.
Furthermore, the output of the upconversion clearly shows a high potential in terms
of
flexibility and of the ability to retrieve certain information, as shown by our example
applications using XQuery.
We are certain that minor problems such as the one caused by the character encoding
will be
fixed during the ongoing development of XProc software. From our point of view, future
modifications could result in a XSD 1.1 compatible XML schema supporting more video
game systems
or textual content that is not review related, such as cheats, hints, or walk-throughs.
Both the
XSLT script and the XQuery queries could be modified in how they interact with each
other. For
example, the distinction of different cases that is carried out by the XSLT script
could be
reformulated as pipeline step, allowing for a more maintainable XSLT script.
In general, the realization of the pipeline and query system as a Web service in conjunction
with a native XML database would result in an alternative search and retrieval mechanism
that
would indeed search for the game according to your
flavour.
References
[Chamberlin et al. (2004)] Chamberlin, D., D.
Draper, M. F. Fernández, M. Kay, J. Robie, M. Rys, J. Siméon, J. Tivy, and P. Wadler,
XQuery from the Experts: A Guide to the W3C XML Query Language. Pearson
Education. Addison-Wesley, Boston, 2004.
[HTML (ISO), ISO/IEC 15445:2000] Information technology — Document description and processing languages — HyperText
Markup Language (HTML). ISO/IEC 15445:2000, International standard, International
Organization for Standardization, Geneva, 2000.
[ISO Country Codes, ISO 4217:2008] Codes for the representation of currencies and funds. ISO
4217:2008, International standard, International Organization for Standardization,
Geneva,
2008.
[Kay (2008)] Kay, M. XSLT 2.0
and XPath 2.0 Programmer’s Reference. Wiley Publishing, Indianapolis, 4th edition,
2008.
[RelaxNG, ISO/IEC 19757-2:2003)] Information technology - Document Schema Definition Language (DSDL) — Part 2:
Regular-grammar-based validation — RELAX NG. ISO/IEC 19757-2:2003, International
standard, International Organization for Standardization, Geneva, 2003.
[SGML, ISO 8879:1986] Information Processing — Text and Office Information Systems — Standard
Generalized Markup Language. International standard, International Organization for
Standardization, Geneva 1986.
[Suda (2006)] Suda, B. Using
microformats. O'Reilly, Sebastopol, CA, USA, (2006).
[Walmsley (2002)] Walmsley, P. Definitive XML Schema. Prentice Hall PTR, Upper Saddle River, NJ, USA,
2002.
[XML 1.0] Bray, T., J. Paoli, and C. M.
Sperberg-McQueen, Extensible Markup Language (XML) 1.0. W3C
Recommendation 10 February 1998. World Wide Web Consortium. http://www.w3.org/TR/1998/REC-xml-19980210, 1998.
[8] We will need p:http-request, although we work on the filesystem. This is
because p:data, which one could expect here, is not a step and therefore does not
accept options.
Chamberlin, D., D.
Draper, M. F. Fernández, M. Kay, J. Robie, M. Rys, J. Siméon, J. Tivy, and P. Wadler,
XQuery from the Experts: A Guide to the W3C XML Query Language. Pearson
Education. Addison-Wesley, Boston, 2004.
Information technology — Document description and processing languages — HyperText
Markup Language (HTML). ISO/IEC 15445:2000, International standard, International
Organization for Standardization, Geneva, 2000.
Codes for the representation of currencies and funds. ISO
4217:2008, International standard, International Organization for Standardization,
Geneva,
2008.
Information technology - Document Schema Definition Language (DSDL) — Part 2:
Regular-grammar-based validation — RELAX NG. ISO/IEC 19757-2:2003, International
standard, International Organization for Standardization, Geneva, 2003.
Information Processing — Text and Office Information Systems — Standard
Generalized Markup Language. International standard, International Organization for
Standardization, Geneva 1986.
Bray, T., J. Paoli, and C. M.
Sperberg-McQueen, Extensible Markup Language (XML) 1.0. W3C
Recommendation 10 February 1998. World Wide Web Consortium. http://www.w3.org/TR/1998/REC-xml-19980210, 1998.
Walsh, N., A. Milowski, and H. S.
Thompson, XProc: An XML Pipeline Language. W3C Recommendation 11
May 2010, World Wide Web Consortium. http://www.w3.org/TR/2010/REC-xproc-20100511/, 2010.