Balisage Paper: Discovering a Long Dormant Bug in XML 1.0

Introduction

Bald-faced Hornet Queens hibernate through the winter to found a new colony in the spring.

Figure 1: Hibernating Queen

Cicadas live underground for up to 17 years before they emerge by the billions in a thunderous summer night symphony.

Figure 2: 17-year Cicada

Is it possible for a bug to lie dormant for almost 30? Yes, yes it is.

Apparently, I first noticed the problem in 2005 since that's when I added a to-do comment into the XOMHandler class indicating that the lines of code for managing system identifiers in DOCTYPE declarations needed a little bit more thought: [Harold 2005]

    if (publicID != null) { 
        internalDTDSubset.append(" PUBLIC \""); 
        internalDTDSubset.append(publicID); 
        internalDTDSubset.append("\" \""); 
        internalDTDSubset.append(systemID);       
    }
    else {
        // need to escape system ID????
        internalDTDSubset.append(" SYSTEM \""); 
        internalDTDSubset.append(systemID); 
    }

Then I promptly forgot about it. More recently, I ran GitHub Copilot across the entire XOM codebase, specifically instructing it to find all the assorted to-do comments and file issues for each one. [XOM Issue 340] Many of the comments it found were moot, already fixed, or just not all that important, but a couple were actually significant, and this was one of them. [XOM Issue 358]

The Problem

Looking at this code two decades later, I didn't recall exactly how system identifiers were supposed to be escaped so I dug into the XML specification. This is a little harder than it used to be because the W3C really tries to hide the actual original XML 1.0 specification from 1998. But I did finally find it. And when I did, here's what I saw. According to section 2.4:

The same text appears unchanged in the Fifth Edition [XML 1.0 Recommendation Fifth Edition] and in the XML 1.1 Second edition. [XML 1.1 Recommendation Second Edition]

A system literal is clearly not a comment, not a processing instruction, not a CDATA section, and not a markup delimiter. Therefore, an ampersand or less than sign that appears inside one should be escaped. Crystal clear, right?

Unfortunately, if you happen to also read section 2.3, you'll see this:

Emphasis added.

This isn't an academic point. Less than signs don't appear in all that many URLs, but ampersands appear in URLs routinely, especially ones that have query string components. Imagine a parser reading a DTD that encounters the URL http://www.example.com/foo?bar=baz&wik=wek

How should it decode that system identifier? As http://www.example.com/foo?bar=baz&wik=wek or as http://www.example.com/foo?bar=baz&wik=wek

These aren't the same. And if you think that's obvious, what about:

http://www.example.com/foo?bar=baz&wik=wek

Is there one algorithm that covers both cases? Do we just have to special case URL decoding based on what comes after the ampersand in a system literal?

Fortunately, we can fall back on the BNF. According to production 11: [XML 1.0 Recommendation First Edition]

[11] SystemLiteral ::= ('"' [^"]* '"') | ("'" [^']* "'")

This does allow & and < to appear unescaped in system identifiers. If you're uncertain about that, compare this production with the similar and immediately preceding production for attribute values which does explicitly rule out ampersands and less than signs: [XML 1.0 Recommendation First Edition]

[10] AttValue ::= '"' ([^<&"] | Reference)* '"' |  "'" ([^<&'] | Reference)* "'"

It seems the preponderance of the evidence does point toward not escaping ampersands and less than signs in URLs. The text that suggests otherwise is simply and error, and I'll come back to that. But first…

How do different parsers behave?

As we know all too well, parsers don't always adhere to the strict letter of the XML specification, even when it's much less ambiguous than this. [XML Conformance Test Suite] So I typed up a couple of very simple documents, one that escaped system literals in the internal DTD subset and one that didn't. First, we need the simple DTD shown in this listing:

                  
<!ELEMENT root EMPTY>

What matters is not so much what's in this DTD, but what its name is. This is saved in a file named root&nothing.dtd. The ampersand is the critical bit. Do we reference that as a raw & as in the first listing below or as an escaped & as in the second listing below?

                  
<!DOCTYPE root SYSTEM "root&nothing.dtd">
<root />

                  
<!DOCTYPE root SYSTEM "root&amp;nothing.dtd">
<root />

A parser should find the DTD with one of these documents and fail to find it with the other. To find out which, I validated each with some common parsers to see which they preferred. The results are summarized in Table I. For non-validating parsers like expat, we expect a parser that does not unescape system literals to report both examples as well-formed and one that does unescape to report the first as malformed.

The XML Conformance Test Suite

The W3C XML Conformance Test Suite is a bit of an afterthought. [XML Conformance Test Suite] It post-dates the original spec by some years. Nonetheless, it has some claim to normativity. Does it address this case? Unfortunately, no.

There are no test cases in the suite (so far as I can find) that contain ampersands or less than signs in system literals. This is neither a negative nor a positive case.

It is perhaps worth noting that the test suite does contains explicit negative ("not well-formed") tests that verify parsers reject unescaped < and & characters within attribute values as required by production 10. (ibm-not-wf-P10-ibm10n01.xml and ibm-not-wf-P10-ibm10n02.xml).

A Proposed Erratum

The text of the spec "The ampersand character (&) and the left angle bracket (<) MUST NOT appear in their literal form, except when used as markup delimiters, or within a comment, a processing instruction, or a CDATA section." is not a well-formedness constraint and therefore not a requirement. The BNF takes precedence and fortunately agrees with parser behavior. Thus, we can classify this as a bug in the spec. We can add the missing case and issue an erratum:

Or can we? Errata for the XML 1.0 and 1.1 specifications are approved and issued by the W3C XML Core Working Group. Unfortunately, this working group dissolved 10 years ago. [XML Core Working Group] I'm not sure it's worth reconstituting it just to do this, but it does feel like this is a hole in the W3C process. XML 1.0 is still very much alive and still a very successful technology. Developers, both human and machine, consume the specification and rely on it to answer questions like this. And this is clearly a less than ideal situation.

Just maybe a test case can be added to the XML Conformance Test Suite that covers this. It's a little tricky due to different rules about characters allowed in file names on multiple platforms, and the desire not to load the test case DTDs from live web servers. Nonetheless, this is only a minor hurdle and should be manageable. The public-xml-testsuite mailing list doesn't seem to have had any activity since 2014, and my attempt to subscribe bounced with "Address not found", but perhaps it can be resurrected. [XML Conformance Test Suite Mailing List]

Specifications in active use in the world today need some form of support for when new problems are uncovered or for when the external situation changes in a way that somehow renders the specification now inaccurate. Consider, for example, time zone shifts, new currencies, or even new laws that somehow cause problems for existing specs. I'm not aware of any cases where these have happened to XML, but I know of many cases where these have happened in other arenas of technology.

While the stability of the XML specification is of critical importance, we should be prepared for the possibility that the specification is not just less than perfect, but contradictory and unimplementable. This is highly unlikely to happen in the core parts of the specification that have been battle tested for nearly three decades now. However, as this example demonstrates, there are still pieces no one's looked at closely enough. We should have a way to fix these.