Balisage Paper: Life, the Universe, and CSS Tests

Antenna House Regression Testing System (AHRTS)

Out of the box, AHRTS makes a pixel-by-pixel, visual comparison of the differences between the PDF (or image) files in a ‘base’ directory against the same-named files in a ‘new’ directory. It produces an overview PDF report listing the state of each test file plus, for each test file with differences, an individual PDF report that contains some data about the ‘base’ and ‘new’ test files plus, for each page with differences, the individual PDF report has a page: showing the ‘base’ page; a composite of the ‘base’ and ‘new’ pages with the differences highlighted; and the ‘new’ page.

AHRTS can be run from a GUI or through the command line. As Figure 1 shows, its additional inputs are an ‘ahrts.properties’ file for controlling the operation plus the XSLT stylesheets for the overview and individual reports.

Figure 1: AHRTS block diagram

Figure 2 shows part of an AHRTS overview report with its indications of which files showed differences.

Figure 2: AHRTS overview report (detail)

Figure 3 shows one page from an individual report. The same page from the ‘base’ and ‘new’ files are shown on the left and right panels, respectively. The center panel is an overlay of the ‘base’ and ‘new’ pages with their differences highlighted.

Figure 3: AHRTS individual report differences page

Figure 4 illustrates the overlaying of the ‘base’ and ‘new’ results and the highlighting of their differences.

Figure 4: AHRTS difference reporting

Customized AHRTS Reports

When I started working with AHRTS, it was to check the effect of my changes to XSL-FO processing. I didn’t want to look through pages of results to spot the ones with differences, so I used AHRTS and I used the Jenkins Continuous Integration Server to automate the running of both the formatter and AHRTS.

AHRTS generates its listing of differences as an XML file, and its PDF reports are produced by using XSLT to generate XSL-FO that is formatted using a built-in version of AH Formatter. Since the presentation aspects come from the XSLT, I also made an alternative XSLT stylesheet that groups tests by their results. From there, it wasn’t much more effort to add counts of each result type.

Figure 5 shows a portion of the overview report produced using the ‘alternative’ stylesheet that is included with AHRTS 1.4.

Figure 5: Alternate AHRTS overview report (detail)

With AHRTS 1.4, it’s now possible to include metadata from the ‘base’ and ‘new’ PDFs in the individual report, as Figure 5 shows.

Figure 6: Alternate AHRTS individual report (detail)

Setting the XSLT stylesheets to use is done in the AHRTS GUI’s ‘Settings’ tab, in the AHRTS properties file, or on the AHRTS command line.

CSS Test Suite Results

When it was time to check the CSS features defined in some of the newer CSS modules that were stable enough to be implemented, we looked again at how the CSS WG reports its results.

The CSS WG has a comprehensive test suite of about 17,000 test files (and growing) and has at least two test harnesses for looking at tests in a browser and reporting results. The test harnesses obviously aren’t unusable when producing PDF.

Figure 7 is a screenshot of a page from the CSS test harness in action. The buttons for selecting the status of the test result are highlighted.

Figure 7: CSS test harness

The CSS WG reports test results as one of five categories 1:

Since I was already using Jenkins to run both AH Formatter and AHRTS, rather than adding or writing yet another application, I wanted a simple way to use Jenkins to collect CSS test results. I made a version of the XSLT for individual reports that added a set of links, one for each CSS test result state plus a sixth that just copies the test’s PDF file. Each link triggers the same ‘testresult’ Jenkins job but provides different parameters, most noticeably the parameter indicating the test result.

Figure 8 shows a portion of an individual AHRTS report with the links for the test results highlighted.

Figure 8: AHRTS individual report with CSS result links

The ‘testresult’ Jenkins job simply runs an Ant task that appends the supplied parameter values as a new line in a log file and also copies the test’s PDF to the test job’s ‘base’ directory^[1]. The next run of the AHRTS job uses the log file information to display the test’s state alongside the indication of whether the current result is the same as the base. And, since I already had similar plumbing for counting tests with differences, the summary report also showed counts of the reports with each state.

Figure 9 shows a portion of an overview report showing both differences found by AHRTS and test result statuses.

Figure 9: AHRTS overview report with CSS status (detail)

HTML Report

However, and there’s always a ‘however’, I was now told that my colleagues in Japan wanted an HTML report. Producing an HTML version of the current overview report was straightforward, and it re-uses some of the existing XSLT modules that are more concerned with logic than with presentation. I was then told that they wanted a report in a format similar to that which had been used previously when testing SVG and MathML support, and they provided a copy of their current CSS test results.

Figure 11 shows a portion of the HTML report being produced by staff in Japan.

Figure 11: Manually produced HTML report (detail)

Their report just recorded the state of the test as ‘OK’ or ‘NG’ (No Good). The real ‘however’, however, was the possible additional comment about the test, issue number, and status of the issue’s resolution.

Adding the extra fields to the PDF for an individual report was straightforward. Instead of using just links, the test results were captured using an Acrobat form.

So far, so good, but this had four problems:

The encoding issue was the killer issue. It completely ruled out Jenkins for collecting test results, even though Jenkins was still wanted for compiling the code and running the tests. Collecting test results in a text file had always been a temporary solution. The intention had always been to move to using an XML database once the data was complex enough to justify doing so. The data still wasn’t particularly complex, but the need to preserve the Japanese text made a good reason to change.

eXist-db

So the project moved to using eXist-db 6. I chose eXist-db partly because I was more familiar with it than with BaseX, but also because I’d had more contact with the eXist-db developers so I knew who to ask if I had problems. I did have problems, but eXist-db has an active and helpful mailing list as well as developers who respond quickly to GitHub issues.

My approach to developing the eXist-db code was initially to copy and modify one of eXist-db’s demo apps. This worked, but the eXist-db documentation has evolved over time, and older documentation advises separate XQuery modules in the ‘modules’ collection, whereas newer documentation favors (almost) all XQuery code for the app in a single ‘modules/app.xql’ file.

Figure 12: App ‘About’ page and default page for a new application

The initial attempt to use eXist-db was by inserting a link to the eXist-db ‘app’ in the PDF of an individual test result. I had also tried making eXist-db return a 206 HTTP response code and no response body (to avoid one of the problems with Acrobat) but I couldn’t get that to work.

Loading

A sequence of Jenkins jobs runs the CSS formatter on the CSS tests then runs AHRTS to compare the latest result with a set of ‘base’ PDF files. The Jenkins job that runs AHRTS also uploads the AHRTS-generated XML files to eXist-db.

The XML for the overview report begins:

<overview date="2017-05-05T20:36:42.638+01:00"
    overview-report-title="Vxx-ref-70-ahrts-csswg-test-pdf #252" test-set="reports">
  <compare name="WOFF2-UserAgent=Tests=xhtml1=available-001.xht" module="WOFF2-UserAgent"
      missing-input-file="false" error-detected="false"
      individual-report-unicode-safe-filename-pdf="000000001_97421a4c.pdf"
      fatal-error="false" difference-detected="true"
      individual-report-pdf="report-WOFF2-UserAgent=Tests=xhtml1=available-001.xht-97421a4c.pdf">
    <warn name="pdf-annotation">どちらのPDFにも注釈タグが含まれています。
    タグは、レポートに埋め込まれたPDFには表示されませんが、比較に含まれます。</warn>
  </compare>

This XML is augment before being uploaded to eXist-db to add the log from AH Formatter and to pre-compute some values.

The individual PDF files from AHRTS are also uploaded into the database. Storing PDF is arguably not a good use for an XML database, but it is much simpler than storing the PDFs elsewhere on the application server and then using URL rewriting to access them. My colleagues in Japan operate their own eXist-db instance, which was set up for them by their IT support staff with whom I have no contact. Since no-one in the Japan office had used eXist-db before, keeping the database installation as simple as possible is, for the moment, more important than shaving a few milliseconds off the time to serve a two-page PDF file.

Summary view

It is straightforward to generate a summary page from the overview XML. Early versions of the application generated a single HTML page with results for every test. Following a request by my colleagues in Japan, more recent versions present one module at a time, as shown in Figure 15:

Figure 15: Summary page

In theory, every top-level directory in the CSS tests corresponds to a same-named CSS Recommendation or Working Draft. In practice, some of the directory names differ from the short name of the module they test. Also, CSS 2.1 has nearly 10,000 tests, so the subdirectories of the CSS2 directory – CSS2/colors, CSS2/fonts, and so on – are treated as separate modules just to keep things more manageable^[2].

Individual test results

A sample testresults.xml file containing the information recorded about the results of a single test is shown below:

<testresult date="20170630" ahf-version="AH Formatter Vx.x A0 for Linux64 : x.x.0.29482 (2017/06/27 09:25JST)">
    <d2/>
    <g4>OK</g4>
    <comment>r28137：NG（Letters of the "Don't Panic" aren't friendly enough.）
r29557：OK</comment>
    <issue>12345</issue>
</testresult>

The format of the XML was determined, firstly, by the information that was already being recorded by my colleagues in Japan (see section “HTML Report”) and, secondly, by needing a simple, ‘XForms-able’ form for the XML. eXist-db ships with two XForms implementations 7 – XSLTForms, which works client-side, and betterFORM, which works server-side. Indeed, the template XForms instance existed before the first results could be added to the database.

Once the XML format was fixed, however, two simple XSLT stylesheets were written to convert the pre-existing log file and HTML results into testresult.xml files that were then uploaded to eXist-db to bring the database up-to-date.

Fatal Attraction

Files with fatal errors are sometimes the most interesting tests. However, they’re rather less interesting if you don’t know why they failed, and are totally uninteresting if you don’t know that they exist.

AHRTS compares PDF output from the CSS formatter, but it has nothing to work with if the formatter aborts with a fatal error because of a problem in its source. Reporting files with fatal errors to eXist-db and including the logs from all tests required more interdependency between Jenkins jobs and between Jenkins and eXist-db.

Firstly, the Jenkins job that runs the formatter had to be modified to save the formatter’s log. Secondly, the Jenkins job that runs AHRTS was modified to add XSLT transforms that augment the AHRTS overview XML to add compare elements for tests with fatal errors and add log elements to (almost) every compare.

The first attempt at saving the formatter log saved the log from the entire test suite as one file and used XSLT to split the text when adding log elements to the overview XML. However, some of the tests generated control characters in the log – for example, the "\f" in \format is a hexadecimal character reference that produces a literal Ctrl-o in the log. That could be handled by switching the XSLT processing to use XML 1.1, which allows control codes in the form , etc. The literal control code wasn’t a problem for the unparsed-text() function, but even the unparsed-text() function and XML 1.1 couldn’t cope with an unpaired Surrogate Pair character code. To avoid a problem with one file affecting all logs, the current processing: saves the log from each test as a separate log file; prepends and appends markup to each log to make the log text be in a CDATA section in a log element; then accesses the XML logs from XSLT by using collection(). Tests with fatal errors now show up in the eXist-db app, and only a handful of tests don’t also have the log from running the test.

Making changes to the XML before uploading it to eXist-db is a slippery slope. The same Jenkins job now also runs more XSLT to group the compare elements by module and to pre-compute and annotate the compare elements with the result of a per-compare calculation that was previously done on-the-fly in eXist-db.^[3]

Import and Export

As stated above, my colleagues in Japan also operate their own eXist-db instance with their own copy of the app. The Japan office maintains the master copy of the test results, so it was necessary to add a way to export results from my database for import into their database. eXist-db has XQuery functions for reading and writing Zip files 8, so this was quite easy.

Running Jenkins from eXist-db

The eXist-db app copies a test’s PDF output to the appropriate AHRTS ‘base’ directory using the same ‘testresult’ Jenkins job that was used back when updating was done using links in the AHRTS individual report PDF files. This, too, is straightforward since the eXist-db app can make a HTTP request to Jenkins to remotely trigger execution of the Jenkins job.

(:  Get Jenkins to copy the 'new' PDF to the 'base' PDF directory :)
declare function common:jenkins-update($job as xs:string, $test as xs:string) {
let $config-uri := concat($common:ahrts-data-home, $job, '/jenkins-config.xml'),
    $config as element(jenkins-config)? :=
        if (doc-available($config-uri))
            then doc($config-uri)/jenkins-config
         else ()
    return
        if (exists($config))
            then let $external-destination :=
                        concat('http://', $config/host, ':', $config/port, '/job/',
                        $config/updatejob, '/buildWithParameters'),
                     $copy-uri := concat($external-destination,
                                    '?',
                                    'RESULT=copy&PDFDIR=',
                                    encode-for-uri($config/pdfdir),
                                    '&TESTNAME=',
                                    encode-for-uri($config/testname),
                                    '&EDITION=',
                                    encode-for-uri($config/edition),
                                    '&VERSION=',
                                    encode-for-uri($config/version),
                                    '&NEWPDF=',
                                    encode-for-uri($test),
                                    '.pdf'),
                     $get := httpclient:get($copy-uri, false(), ())
                 return $copy-uri
        else ()
};

XForms or Bootstrap?

When developing an XML project that uses a web browser, it’s hard to avoid thinking that you should use XForms. eXist-db ships with two XForms implementations, and so the first versions of the eXist-db dutifully used XForms for collecting input. However, eXist-db also ships with the Bootstrap, which is the HTML, CSS, and JavaScript framework that provides the look-and-feel of a large proportion of sites on the Web. Two of the three provided templates for an eXist-db app install Bootstrap in the new app, and, as shown in Figure 12, this app’s pages that don’t need forms do use the Bootstrap-based eXist-db design. eXist-db also provides an "HTML Templating Module" 11 that makes it easy to generate HTML pages, but none of the XForms examples in the documentation use it.

Bootstrap can also style HTML forms 12. Partly to provide a consistent ‘look’ for the app, but mostly because the XForms pages looked dated compared to the Bootstrap pages, all but one of the original XForms in the app have been replaced by HTML forms that are styled using Bootstrap. Can you pick which of the preceding screen shots uses an XForm?^[4]

It is possible to achieve a ‘mostly-Bootstrap’ appearance by combining Bootstrap classes for structural elements with XForms markup for the form fields, for example:

<div class="form-group">
    <label class="col-xs-2 col-sm-2 control-label">Import File</label>
    <div class="col-sm-10">
        <xf:upload id="upload1" ref="file"
            mediatype="application/x-zip-compressed"
            accept="application/x-zip-compressed">
            <xf:filename ref="@filename"/>
            <xf:mediatype ref="@mediatype"/>
            <xf:size ref="@size"/>
            <!--<xf:send ev:event="xforms-value-changed" submission="save"/>-->
        </xf:upload>
        <span class="help-block" style="margin-bottom: 0">Select the test results export file to import.</span>
    </div>
</div>

but the end result is still unsatisfactory since some parts of the form don’t quite line up correctly and some aspects of the styling, such as the appearance of the buttons, can’t be worked around, as shown in the following figure.

Figure 19: Bootstrap and XForms ‘Import’ buttons

HTML Templating

The HTML templating mechanism 11 is separate from Bootstrap. It provides a convenient mechanism for generating HTML pages since:

The theory, from the documentation, is that:

My practice, however, has tended towards putting a minimum in the HTML:

<div xmlns="http://www.w3.org/1999/xhtml" data-template="templates:surround"
  data-template-with="templates/report-page.html" data-template-at="content">
    <div class="col-md-12" data-template="app:individual">
        <div class="row">
        <div class="col-xs-10 col-sm-10">
        <h1 data-template="app:individual-title">Generated page</h1>
</div>
<div class="col-xs-2 col-sm-2">
                <a id="poweredby" href="index.html"/>
</div>
        </div>
                <div data-template="app:individual-form"/>
                <div data-template="app:individual-nav"/>
                <div data-template="app:individual-pdf"/>
                <div data-template="app:individual-log"/>
        </div>
</div>

and doing more in the XQuery:

(: Populate $model for an individual page. :)
declare
    %templates:wrap
function app:individual($node as node(), $model as map(*), $job as xs:string, $test as xs:string) {
    let $report as element(report)? :=
            doc(concat($app:data-home, $job, '/compareOutput/', $test, '/base_vs_new.xml'))/report,
        $result := $report/analysis/result,
        $base as xs:string? := $result/inputfile[@version = 'base']/string(),
        $new as xs:string? := $result/inputfile[@version = 'new']/string(),
        $compare := doc(concat($app:ahrts-data-home, $job,
                               '/reports/digest_vs_reports.xml'))/overview/compare[@name eq $test],
        $prev as xs:string? := $compare/preceding-sibling::compare[1]/@name/string(),
        $next as xs:string? := $compare/following-sibling::compare[1]/@name/string(),
        $pdf as xs:string? := $compare/@individual-report-pdf/string(),
        $log as element(log)? := $compare/log
    return
        map { "report" := $report,
              "compare" := $compare,
              "prev" := $prev,
              "next" := $next,
              "pdf" := $pdf,
              "log" := $log }
};

(: Page title for an individual page. :)
declare function app:individual-title($node as node(), $model as map(*), $job as xs:string, $test as xs:string?) {
    let $report := $model("report")
    return
       (<h1 class="main-title"><a href="index.html">{$config:expath-descriptor/expath:title/text()}</a></h1>,
        <h3 class="report-title">
            <a href="summary.html?job={encode-for-uri($job)}">{
                if (exists($report))
                    then $report/@overview-report-title/string()
                else $job
            }</a>
               </h3>,
       if (exists($test)) then <h2>{translate($test, '=^', '//')}</h2> else ())
};

I find that the templating mechanism works quite well and is easy to use once you get the hang of it. One problem, however, is that the keys of the map entries are not validated, so a typo where the key is defined or anywhere where it is referenced can lead to a mysterious empty sequence simply because the keys don’t match.

‘=’ in file names

AHRTS can currently only compare files in two directories, and it does not look into subdirectories. Most test suites – including the CSS test suite – have files arranged in subdirectories. To get around this difference, the Jenkins job that runs the formatter would write the PDFs to file names where ‘=’ – which does not appear in any test file names – was used in place of the directory separator. This worked fine before eXist-db was used. However, ‘=’ needs to be escaped in parameters in URLs, and the ‘=’ in file names exposed multiple bugs in eXist-db. For example, collections with names containing ‘=’ can’t be opened in the eXide editor’s ‘Manage’ interface, and the only way found for deleting them is by using eXist-db’s Java interface.

To their credit, the eXist-db developers responded quickly to the initial bug reports, but fixing them all will take time. It was simply be more reliable to just use a different separator character sequence, but doing that required a lot of renaming of files on Jenkins’s file system and renaming resources in eXist-db. Since the PDF files couldn’t be renamed programmatically, the collection that contained them had to be deleted and a whole new set of PDFs uploaded.

CSS Tests now Web Platform Tests

In the first half of 2017, the CSS Working Group migrated from their tests from their own GitHub repository to being under a subdirectory of the Web Platform Tests project. Several of the modules were renamed during the migration, so it was necessary to migrate the corresponding test results to new URIs to match.

It is unlikely that export files from before the changeover will ever be needed again but, just to make sure that nothing is lost, the code for importing test results now handles both old file names containing ‘=’ and old, pre-WPT module names and maps them to current usage.

Localization

The current ‘app’ is almost completely localized for both English and Japanese. Some localization into English was necessary because the ‘D2’ states were initially only provided as Japanese text. Some of the localizations into Japanese were rolled back at the request of my colleagues in Japan because they found the English easier to understand than the notionally equivalent nouns and verbs that I’d plucked from an online English–Japanese dictionary.

eXist-db has a localization library 13 that uses files with a format that is very similar to Java XML property files.

<catalogue xml:lang="ja">
  <msg key="Comment">コメント</msg>
  <msg key="Date">日</msg>
  <msg key="Diff">相違</msg>
  <msg key="Issue">発行</msg>
  <msg key="Test">テスト</msg>
</catalogue>

Localizations are applied using i18n:text elements in a mechanism similar to the HTML templating mechanism:

<tr>
  <th><i18n:text key="Test">Test</i18n:text></th>
  <th><i18n:text key="Diff">Diff</i18n:text></th>
  <th>D2</th>
  <th>G4</th>
  <th><i18n:text key="Comment">Comment</i18n:text></th>
  <th><i18n:text key="Issue">Issue</i18n:text></th>
  <th><i18n:text key="Date">Date</i18n:text></th>
</tr>

This works well enough, but language selection (in my opinion) is not straightforward, and the standard library does not provide the option of selecting the language from the browser’s Accept-Language header. At present, the app uses its own version of the i18n library that is based on a versoin 14 by Wolfgang Meier, one of the eXist-db developers. This version can use either the Accept-Language header or a language setting configured in the app.

Using elements to handle localization isn't helpful for localizing attribute values and the functions provided for use from XQuery require specifying both the path to the localization files and the current language, plus repeating the text in the i18n:text element’s content and its key attribute seemed redundant, so I made some convenience functions that wrap the regular i18n processing, work in attribute values, get the localization files’ path and language from the app’s configuration, and require only one copy of the text, for example:

<input type="text" class="form-control" name="issue" id="issue"
  placeholder="{common:i18n-text('Issue numbers')}"
  value="{$testresult/issue/string()}" accesskey="i"/>

No way has yet been found to localize the messages popped up by the BetterForms XForms implementation.

Dashboard

As stated previously, the mass of tests is divided into modules to make the work more manageable. Information about the modules and their relative priorities was initially maintained as a wiki page, but that was later migrated to a spreadsheet. When my colleagues in Japan wanted to also see the priorities in the eXist-db app, I both added a mechanism to paste tab-delimited text from the priorities spreadsheet into the app and provided a dashboard summarizing the results for each module and each priority level.

Over time, however, my colleagues in Japan have requested additional information on the dashboard for combinations of test result status values that are useful to them, as shown in Figure 20.

Figure 20: Dashboard

And despite their being the impetus for dividing the summary view by module, they also request views of all tests with particular status combinations, as shown in Figure 21.

Figure 21: New summary views