Balisage Paper: The XSLT/XPath/XQuery 4.0 Standards: a High-Level Perspective

About this Paper

The sheer number of new features added to the 4.0 specifications^[1] makes it very difficult to provide a concise summary of what's new. Even in the release notes^[2] for Saxon 13, which implements many of the new features, we resorted to the headline summary "More than 175 features from the draft 4.0 specifications are now available...." There are various ways of tracking the detailed changes: the specification itself contains a "changes" summary below the section heading of any section that contains substantive changes, and these link to the GitHub Pull Requests^[3] that contain the detailed text of each change. Test cases in the extensive test suite^[4][5] (also available on GitHub) are also linked to pull requests, and pull requests that have been implemented in a released Saxon version include a GitHub label "In Saxon 12" or "In Saxon 13".

As a taster, I published a list of 101 Fun things to do with QT4^[6]: none of them described in any detail, but a list intended for easy reading where, if particular features whet your appetite, you can go to the specs and investigate further.

So I won't make any attempt in this paper to provide a comprehensive list of new features. Instead, I will take a step back, and try to discern some overall themes.

What is the status of the work? It's six years since I first proposed the QT4 initiative [Kay 2020], and four years since the QT4 community group started work to flesh out a definition of the 4.0 versions of XSLT, XPath, and XQuery. The work is ongoing and subject to revision and change as the work proceeds.

At a process level, there is still some work to do to decide under what auspices the work should be published. W3C no longer has a formal XML Activity, and there are no official working groups chartered to take the XML standards forward. QT4 is not alone in making technical progress through the informal mechanism of W3C Community Groups^[7], but these do not have the authority to publish official Recommendations. As the work nears completion, we can expect to see the politics get resolved, but meanwhile the work is presented to the world to accept or reject based on its technical merits.

Process is important, but the significant parts of the process to date have been those that facilitated the technical collaboration to produce a high quality specification, rather than the process for getting an official stamp of approval on the final publication (including a formal consultation process). Because process is important, I'll share my thoughts on the subject.

The current state of the QT4 standards

We can divide this into three parts.

I'll consider these three questions in the following subsections of the paper.

Standardization in the XML community

The XML community's attitude to standards is driven by the nature of XML itself. As a standard for transmission of data between heterogenous systems, and even more so as a standard for long-term storage of information of lasting value, XML instrinsically needs to be very rigorously standardised, and moreover, changes to the standard over time are generally unwelcome, as proven by the failure of the 1.1 version: the cost to the community of adopting a change exceeds the benefit. The same observations can be made for JSON, and it is notable that for both XML and JSON, only very small changes have been attempted once they had become established in widespread use. Even fixing widely-acknowledged bugs (such as the fact that JSON allows unpaired Unicode surrogates, or the fact that XML fails to say exactly what constitutes a valid namespace URI) has proved remarkably difficult.

This community attitude to standardisation naturally extends to closely associated standards such as XML Schema. The need for standards in areas such as transformation and query languages is perhaps not as strong; there is no intrinsic reason why multiple programming languages and APIs (and successive versions thereof), should not thrive just as happens in the field of general-purpose programming languages. What's different here is the architecture of the web: if stylesheets are to be executed in the browser, then the stylesheet language needs to be supported by every browser, and until recently, the browser environment has been a barrier to innovation because the platform is tightly controlled by a cabal of browser vendors. The standardisation culture for the XSLT environment is thus conditioned partly by the original motivation of client-side transformation, which demands rigid standards, and partly by the area where it has seen most success, as a server-side domain-specific transformation language, where there is more room for variety and innovation: a culture that has enabled newer versions to thrive in a way that newer versions of XML and JSON could not.

Lessons from history

QT 4.0 is, as the number suggests, the fourth phase of standardisation in this area. The 1.0 standards were produced very quickly, and proved highly successful, though they had many technical flaws. Each subsequent phase has taken longer: seven years to version 2.0, a decade to version 3.0, perhaps even longer (who knows) to version 4.0. On many measures the standards have improved in quality: they are more precise, less ambiguous, better supported by test suites, there are certainly fewer gaps in functionality, and fewer usability quirks. But some of the new features that were introduced have not achieved widespread adoption^[15]. Can we look at the features that were successful, and those that weren't, and spot any trends?

My personal attitude is that the success of a standard, and of new features in a standard, is more related to the cost of adoption than to the benefit of adoption. The most significant innovation in XSLT 2.0 was schema awareness, and twenty years later it seems that only a small minority of the user community is using the feature. This is not because there is no benefit: our experience suggests that when developing complex XSLT-based applications, schema awareness gives many productivity benefits through easier debugging. But there is a significant up-front adoption cost. Writing the first version of your code takes longer, even if you get to a complete debugged application more quickly. Similar observations apply to some of the big ideas in 3.0, such as streaming and packages, both of which deliver significant benefits that only a small number of enlightened users are taking advantage of.

By contrast, the things that have been most successful have been those that could be adopted incrementally without any major change. In XSLT and XPath 2.0, grouping, regular expressions, XSLT functions, date and time manipulation, conditional expressions. In 3.0, maps, xsl:iterate, accumulators. Higher-order functions occupy a middle ground: the cost of adoption here is primarily the challenge of learning a new concept in computer science and software engineering, and adoption has steadily increased as the same basic idea has become familiar in other languages such as Java, C#, and Javascript.

I think that the vast majority of changes in QT4 are incremental in this sense: once users discover that the feature exists and is available in their chosen products, they will be quick to take advantage of it.

The standardization process

Who are the stakeholders in the standards process?

I will identify four: the users, the implementors, the funders, and the standards organizations themselves. What we end up delivering depends on how these four groups of stakeholders play the game, and that in turn depends on what they are trying to achieve.

What I'm going to say here draws not just on experience with QT4, but on other experience within W3C and before that with other standardization projects: Codasyl, ANSI, BSI, ISO, ECMA, in some of which I was a direct participant and in others merely an interested observer. And this over a timespan of fifty years.

The mechanics of the standardization process have of course changed immeasurably over that timeframe. In the Codasyl days I would type a proposal on a typewriter, make 40 copies, and address 40 envelopes, making sure they arrived three weeks before the next meeting, which was always somewhere in the US and lasted a week. Compare that to the process of creating a pull request on GitHub, and getting feedback an hour later. But although the mechanism may have changed, the factors that make some proposals successful and others unsuccessful have not.

On most of the groups I have worked with, users have acted primarily as reviewers rather than initiators. A genuine user (by which I mean someone who actually does hands-on development and knows the specs well) is a great asset, even if review and verification is the only contribution they make. Unfortunately some user organisations instead send a supernumerary, someone who is no longer actively engaged and whose technical knowledge is rusty; these people can be time-wasters. The QT4 activity has been mercifully free of time-wasters, though we do of course get the occasional contribution from someone outside the core group who is unaware that the feature they want is already available.

By "implementors", I mean those participants who are actively involved in creating implementations. I don't mean representatives of the companies who market products, I mean the designers and developers who actually cut the code. Very often there are only two or three genuine developers taking part. That's not necessarily a bad thing: the original XQuery 1.0 work was hampered by having perhaps eight or nine people in this role, many of whom could have done a good job as chief designer of the language, but all of them having different perspectives on what kind of language they wanted to create. It tends to be the developers on the group who come up with most of the ideas. Ideally they not only develop implementations of the standard they are working on, but also have projects in which they develop applications using the technology, so they can put themselves in the mindset of users. They typically also have their own source of feedback from their own user community.

In my experience the developers on the group, although they work for companies that are competing with each other, often get on very well with each other: they love sharing ideas and often build on each others' ideas to create something that none of them would have come up with individually. This collaborative competition is highly beneficial to improving the state of the technology for everyone. There can be tensions, however, if very capable people share different visions of the end product.

All groups can suffer from having participants who are poor team players. These people may be very able technically, but they can get so passionate about their good ideas that they lose any ability to see other points of view and make compromises. They can also lose sight of the fact that selling their ideas is essentially a human process that depends on understanding where other people are coming from. The chair of one standards group, who in his day job was an industrial psychologist, once took me aside and told me that a colleague of mine had (in my absence) presented a proposal in such an antagonistic manner that the group rejected it out of hand, even though the proposal actually had considerable technical merit.

The funders — the people who pay the salaries and hotel bills of the participants — have their own agenda, but in my experience, they are often rather vague about what they actually want to achieve. It's hard to convince them that they are in a joint venture with their competitors from which everyone gains an advantage. They may well be under the misapprehension that by taking part, they will be able to get features into the spec that benefit them and harm their competitors: an argument which, in my view, is nearly always deluded. The real business case is that the activity grows the size of the market and they get their fair share of that growth.

Too much input from the funders can have detrimental effects. When vendors start pushing the standard in a particular direction, the pressure is often unrelated to any genuine understanding of user requirements. Whether the pressure is from managers in the vendor organization, or their marketing people, or the IT directors investing good money in the vendor's products, the chances are that the real needs of hands-on developers using the technology have got lost somewhere along the way. This can easily lead to complex features being added that no-one ends up using, while simple practical improvements are overlooked.

There's a contradiction I've observed in all the groups I've worked with. The fine detail of a new feature gets an enormous amount of detailed attention, and ends up being very well specified and documented (far better, in my experience, than the specification of any proprietary software product). Meanwhile, the requirement behind the feature often receives little scrutiny. If someone claims that their users need the feature, it's very hard to challenge that statement; standards groups gather their requirements from a small and self-selecting sample of users. Certainly, I've never seen any serious attempt to quantify the cost benefit of a feature, or to apply any rational process to prioritization of different features.

Finally, the standards organizations: W3C, ECMA, ITU, IETF, WhatWG, or whatever. What is their role and motivation? Almost invariably, these groups start as a collaboration between vendors, often vendors whose common interest is to reduce the dominance of an existing monopoly supplier. If they are successful at all, the dominant supplier later senses which way the wind is blowing and joins the group. Very often the group is formed because established venues for doing the work have become fossilized: they have built up sufficiently heavyweight decision-making processes that the active parties decide they are better off starting something new. Whether the rest of the community likes it or not, it is implementors who drive the technology forward. User input is useful, but the real costs are borne by implementors, and they are only going to keep implementing a standard if it's in their interests to do so. There was a stage in the 1980s when official "de jure" standards bodies became totally disconnected from implementation activities, and it led to the creation of numerous standards that weren't worth the paper they were written on. As the internet sprang to life, it became clear that implementors had the upper hand, and that standards were often written as a post-hoc exercise once the defining products had already become established. That's true regardless whether the products in question were open source (like the early email software, or like browsers) or whether they were proprietary (like Microsoft Office).

What does all this mean for QT4? We've been fortunate. There have been no misguided inputs from funders disconnected from hands-on users, there's a general feeling that everyone is pulling in the right direction, and we benefit from the reviews and comments of a wider user community that understands the technology well and knows its limitations. We have developed our own very effective and economical tooling based on GitHub, XML, XSLT, Gradle, and the proprietary DeltaXignia product (formerly DeltaXml). The absence of a sponsoring standards organisation (unless you count the very hands-off W3C, which makes no attempt to control the direction) hasn't hampered us so far, though it might be useful to re-establish a branding relationship in order to build credibility as we come to publication stage. But we might find that at that stage, politics creeps in.

We do have a problem knowing when to stop. We're all perfectionists who want to get it right, and we tend to prioritize quality over completion dates.

One of the obstacles W3C put in our way during the development of the 3.0/3.1 versions of the standards was the requirement to demonstrate the existence of three independent implementations. We scraped through that requirement on XSLT 3.0: we had two implementations actively developed by working group members (Saxon and Exselt, the latter having since been withdrawn from the market), and there was a third implementation, Altova Raptor, which followed the standard closely even though the vendor played no part in the W3C process (which in a sense is actually good evidence that the spec is solid). Unless there's a mystery implementor shadowing every move we make, we're not going to be able to satisfy that requirement for XSLT 4.0: does this matter? My view is that a specification created using this community process is going to be of much higher quality than if a vendor like Saxonica had tried to go it alone, and that's what matters to the user community.

I'm confident that there will be other implementations in the future even it there aren't today: at this stage of the game, people want to see a standard set in stone before they start the expensive process of implementation^[16]. There are even signs that AI technology is getting close to the point where given a good enough specification and test suite, the process of constructing an implementation can be largely automated. We're not there yet, but progress has been dramatic.

Meanwhile it remains very much the case that continuous implementation during development of the standard is necessary both to verify that features are practically implementable, and to provide early feedback on their usability.

And of course, the continuous and collaborative development of a comprehensive test suite is another key ingredient - but one which probably merits a separate paper.