Balisage Paper: The Allure of Gothic Markup

The Legalistic Applications of SGML

SGML required documents to come with an declaration of their structure. Brittleness was built into the system. Documents were not meant to exist by themselves, but rather as part of an SGML application, defined as such:

The SGML Handbook notes that SGML applications needn't actually be "applications" as most computer users expect them:

But while processing should not be constrained, document structure must be. SGML documents must contain a DOCTYPE declaration, which must in turn reference (or include) a DTD, and those DTDs rapidly became the battleground over which the users of SGML fought.

Ground Rules: XML

Those battles - or perhaps it is nicer to say negotiations - carried over into the XML world. Although XML allowed documents to go forth boldly naked without a DTD, the expected approach for its use still involved prior agreement. Citing Lewis Carroll's Humpty Dumpty, the W3C's Dan Connolly warned of the dangers of diverse semantics:

Phrasing things more positively, Jon Bosak and Tim Bray described the work ahead and why people were eager to do it:

Businesses and developers took up that challenge, and thousands of committees blossomed. XML "solved" the syntactic layer, and it was time to invest millions in defining structure.

Pinning Down Butterflies with URIs

The world of agreements wasn't enough for XML's keepers at the W3C. Tim Berners-Lee's Semantic Web visions required globally unique identifiers for vocabularies. In the period when Berners-Lee considered XML a key foundation for that work, that meant spackling URIs into markup to create globally unique markup identifiers and build them into vocabularies.

As the SGML community had ventured into hypertext, they too had found difficulties in sharing structures across vocabularies and recognizing those structures. They lacked pretensions of building a single global system, however, and had proposed a very different route: architectural forms.

While there was some work done to bring architectural forms to XML - most visibly David Megginson's XML Architectural Forms - architectural forms lost a bitter battle inside the W3C. While confidentiality makes it difficult to tell precisely what happened, public sputtering suggests that architectural forms' vision of adaptation to local schemas did not not appeal to the Director of the W3C's intention of building globally-understood vocabularies identified by URIs. Instead, the XML world was given a mandate to create vocabularies with globally unique identifiers.

Architectural forms were a limited set of transformations, still deeply intertwined with the contractual expectations of DTDs. The stomping they received at the W3C, however, was a sign of static expectations to come.

The First Hit is Free: Augmented Infosets and the PSVI

As more and more developers took Bosak and Bray's promises seriously, XML reached field after field, migrating far beyond the document-centered territory SGML had considered its home. While the ideas of agreement and validation were popular in many areas, DTDs did not feel like an appropriate answer to many developers. Many developers' expectations had been shaped by databases, strongly typed languages, and fields that wanted more intricate specifications of content.

The W3C responded with XML Schema, a pair of specifications defining a language for specifying deterministic document structures and content, supporting associated processing. Like its DTD predecessor, XML Schema's validation process modified the document, adding default values for attributes (the classic case). Going beyond what DTDs had done, it also annotated the infoset of the reported document with type information accessible to later processing.

While press releases are not usually a great place to learn about the details of specifications, they are an excellent place to learn about what those specifications are meant to do. In the case of XML Schema, there are even a few to choose from. At the outset, 1999's press release was excited about the potential for establishing standards for many kinds of transactions:

In 2001, when the XML Schema Recommendations arrived, the press release described schemas as finally fulfilling the promises XML had previously made:

It wasn't just that XML Schemas would "make good on the promises of extensibility and power", but, as later specifications demonstrated, that they would provide a foundation for further work in processing XML as strongly typed data. The Post-Schema Validation Infoset, effectively a type-annotated version of documents that passed validation, became the foundation on which XSLT 2.0 and XQuery would build. XML itself didn't require that you use schemas of any kind, but the core toolset incorporated more and more assumptions based on schema capabilities, without any separation of concerns. "XML" practice clearly incorporated XML schema practice.

As schema proponents had hoped, the technology took off, becoming a central component in a rapidly growing "Web Services" ecosystem, largely built with tools that made it (relatively) easy to bind XML data to program structures using the type information provided by schemas. Schemas served not only as documentation and validation tools but as tools for structuring code. (They also served as configuration for XML editors of various kinds.)

However, as the limitations and intrusions of XML Schema became clearer, alternate approaches appeared. RELAX NG was in many ways a simpler and better thought-out approach to creating comprehensive schemas, while Schematron's rule-based approach offered a very different kind of testing. Examplotron built on Schematron's ideas to create a very different model of a schema, re-establishing the value of using sample documents for conversations about document interchange.

The Nature of Gothic

New fields like to flatter themselves by styling themselves after older ones. While computing is often engineering (or plumbing) minus the structured training, it more typically compares itself to architecture. Like architecture, it hopes to achieve some form of grace in both its visible and invisible aspects, creating appealing structures that will remain standing. While we may have calmed down a bit from the heroic architect model of Howard Roark in The Fountainhead, markup language creators still expect to be able to lay things out as plans and have them faithfully executed by others who will live up to our specifications.

Writing a century before computing's emergence, John Ruskin's The Nature of Gothic (a chapter of The Stones of Venice) offered a very different view of the proper relationship between craftsman and architect. This is long, but even Ruskin's asides have parallels to computing practice, so please pause to study this quote more than its Victorian prose might otherwise tempt you to do:

What has architecture to do with markup?

In the virtual world, markup creates the spaces in which we interact. It creates bazaars, agoras, government buildings, and even churches. Markup builds the government office, the sales floor, the loading dock. Markup offers us decorations and distractions. Markup is our architecture.

The concerns which apply to architecture also apply to markup. Markup process, often deliberately, parallels that of the contemporary design approach. Define a problem. Develop a shared vision for solving it. Hire experts who create plans specifying that vision in detail, and send them to the "workmen" to be built.

Ruskin's friend William Morris, in his Preface to The Nature of Gothic, warns of the costs of that style of work.

Architecture has largely ignored these concerns, and computing, alas, has taken its lessons from that ignorance. Let us instead take Ruskin's worker free of alienation - valuing savageness and supporting changefulness - as a goal worth achieving.

Patterns for Conversation

Christopher Alexander has the strange distinction of being an architect more revered and imitated in computing than in his own field. When, during an XML conference in Philadelphia, I stopped at the American Institute of Architects to buy A Pattern Language and The Timeless Way of Building, the cashier informed me that I must be a programmer, because "only programmers buy those books. Architects don't."

Not Required

Ever since XML 1.0 permitted documents without a DOCTYPE declaration, it has at least been possible to work with XML in the absence of a schema. While in most of my travels I have only found people using this freedom for experimental purposes or for very small-scale work, conversation on xml-dev did turn up some people who simply avoid using schemas. They do, however, seem to use standard formats, but test and document them by other means. These practices are often criticized, especially when the content leaves those often closed systems [Beck 2011].

Even in the best of these cases, though, throwing off schemas is not enough if the expectations of fixed vocabularies remain behind, as Walter Perry warned over a decade ago in [Perry 2002]

Leaving Gaps

Even the most obsessively controlling schema vocabularies allow developers to leave some space for growth. ANY in DTDs and xs:any in XML Schema are the classics, and variations allow a mixture of precision and openness.

Support for these gaps, however, varies wildly in practice. Both tools and culture push back against the open models. Tools that came from the strictly defined expectations of object definitions have a difficult time dealing with "underspecified" markup. The culture of interoperability testing often encourages maximum agreement. Do open spaces make it easier to create new options, or do they just create new headaches when it's time to move from one version of the a defined vocabulary to another? Gaps create tension with many supposed best practices.

Generic Parsers, not Parser Generators

Although it is certainly possible to write parser generators that lock tightly onto a particular vocabulary and parse nothing else, it happens less often than might seem likely. There are tools for creating parser generators, like [XML Booster], and there are certainly cases where it is more efficient or more secure than processing the results of a generic parser. However, judging from job listings and general conversation, parser generation has had a much smaller role in XML than it has had, for example, in ASN.1. (I've had a difficult time in ASN.1 conversations even convincing developers that generic parsers were possible and useful.)

Data binding tools, of course, can produce tight bonds even when run on top of a generic parser, but XML's explicit support for generic parsing has at least created the opportunity for looser coupling.

Peace Through Massive Overbuilding

Some vocabularies have taken an "everything but the kitchen sink" approach. Two of the most popular document vocabularies, DocBook and the Text Encoding Initiative (TEI), both include gigantic sets of components. While both provide support through tools for those components, many organizations use subsets like the DocBook subset used for this paper. While subsets vary, having a common foundation generally makes transformations easy and fallback to generic tools an option. The TEI pizza chef [TEI Pizza Chef], which served up custom DTDs, typically a TEI subset, stands out as a past highlight of this approach.

Building a vocabulary so large that most people work with it only through subsets may seem excessive, but it opens the way to conversation among users of different subsets. In many ways, this is similar to (though operating in the reverse direction of) Rick Jelliffe's suggestion that:

So long as the superset model is broad enough, peace can be maintained.

Peace Through Sprinkling

Rather than building supersets, some groups have focused on building the smallest thing that could possibly work for their use cases. Dublin Core [DCMI] is probably the most famous of these, though a variety of annotations from [WAI-ARIA] to [HTML5 Data Attributes]. These approaches offer a range of techniques for adding a portion of information used by a certain kind of processor to other vocabularies. They allow multiple processors to see different pieces of a document, though frequently there is still a unified vision of the document managed by those who sprinkle in these components.

Peace Through Conflict

While I cited Connolly 1997 above, I halted the quote at a convenient point. There is more to that argument - still schema (DTD) focused, but acknowledging conversation beyond the original creation point:

While there has been more visible competition among schema languages than among vocabularies specified with schemas, there are constant overlaps among vocabulary projects as well as some direct collisions. "Acclamation" may be too strong a word, as steady erosion seems a more typical process, but there is certainly motion.

Accepting Failure

Resisting System-B is easiest, perhaps perversely, in a corner of the software universe that has long hoped to make System-B's "design by specification" possible: functional and declarative programming. These styles of software development remove features that add instability to imperative approaches, often in the pursuit of mathematical provability, reliability and massive scale. These design constraints, though intended to maximize industrial-scale processing of information, also make possible a wide range of more flexible approaches to handling information.

The paradigmatic application of these tools in the markup world lies in the technologies we call stylesheets, or style sheets, depending on who is editing at any given moment. While Cascading Style Sheets (CSS) and Extensible Stylesheet Language (XSL) were frequently seen as competitors when XML first arrived, both offer similar capabilities in this regard. They both are (or at least can be) excellent at tolerating failure, with little harm done.

The key to that tolerance is pattern matching - selectors for CSS, XPath for XSLT. If patterns don't match, they don't match, and the process goes on. XSLT offers many more features for modifying results, and is more malleable, but neither of them worry much if a document matches their expectations. At worst they produce empty results. XSLT is capable of operating more generically, and of working with content it didn't match explicitly. The XSLT toolset can support reporting and transformation that goes beyond the wildest dreams of schema enthusiasts - and can do much more useful work than validation and annotation along the way.

Pattern matching is also central to a number of explicitly functional languages. While they were built for things like mathematical provability, "nine nines" reliability, and structured management of state, those constraints actually give them the power needed to go beyond XSLT's ability to process individual documents. Erlang's "let it crash" philosophy, for example, makes it (relatively) easy to build robust programs that can handle flamingly unexpected situations without grinding to a halt. Failures can be picked up and given different processing, discarded or put in a queue for different handling.

A calm response to the unexpected opens many new possibilities.

Valuing Errors

Years ago, Walter Perry said in a talk that often the most interesting communications were the ones that broke the rules. They might be mistakes, but they might also be signs of changing conditions, efforts to game the system, or an indication that the system itself was flawed.

Errors and flaws have become more popular since. While much of the effort poured into test-driven development is about making sure they don't happen, a key side effect of that work is new approaches to providing meaningful error messages when they do happen. "Test failed" is useful but incomplete.

In distributed systems, errors aren't necessarily just bugs, an instant path to the discard bin. While the binary pass/fail of many testing approaches encourage developers to stomp out communications that aren't quite right, turning instead to the meaningful error messages (and error handling) side of that conversation can be much more fruitful.

Mechanical Turks

After decades of trying to isolate computing processes from human intervention, some developers are now including humans in the processing chain. After all, it's not difficult to treat such conversations as just another asynchronous call, especially in an age of mobile devices. Not everything has to be processed instantly.

Amazon developed the [Amazon Mechanical Turk] service, named after an 18th chess-playing "machine" that turned out to have a person inside of it. It looked like brilliant technology, and was, if humans count. Amazon adds digital management to the approach, distributing "Human Intelligence Tasks" across many anonymous workers. Facebook uses similar if more centralized approaches to censor photos. [Facebook Censorship] The Mechanical Turk model has led to some dire work situations [Cushing 2012] in which humans are treated as cheap cogs in a computing machine, as a System B industrial approach seeks cheap labor to maximize profit.

Horrible as some of these approaches are, they make it very clear that even large-scale digital systems can pause to include humans in the decision-making process. It isn't actually that difficult. Connecting these services to markup processing, however, requires interfaces for letting people specify what should be done with unexpected markup. "Keep it, it's okay" with a note, or an option to escalate to something stronger (perhaps even human-to-human conversation) may be an acceptable start.

JSON Shakes it Up

While XML seemed to be conquering the communications universe, even finally reaching the Web as the final X in AJAX, many developers dreamed of an escape from its strange world of schemas, transformations, and seemingly endless debates about data representation. Douglas Crockford found an answer uniquely well-suited to the Web, extracted in fact from the JavaScript programming language itself. [JSON] (JavaScript Object Notation) rapidly became popular with JavaScript developers. JSON had an innate advantage in that it could bypass same-origin requirements, but its use has spread far beyond those situations.

JSON uses a different syntax, but much more importantly, the nature of the conversation also shifted. Perhaps because it comes from a free-wheeling JavaScript background, expectations of structure have always been loose. Coordination can happen, but reuse and modification is a more common pattern than formal structuring. Many JSON formats are created by single information hubs, rather than across groups of providers, and conversion to internal formats is just a normal fact of life for JSON data consumption.

JSON, while somewhat less readable to humans than markup, was both easy to work with in a JavaScript context and compatible with (actually a subset of) the [YAML] data serialization supported by a few other languages. JSON was just a data format, a means for developers to pass information from one program to another. Although JSON schemas and JSON transformation tools exist, they are relatively minor corners of JSON culture.

Despite those glaring absences, JSON use continues to expand rapidly. It replaced XML as a default format in Ruby on Rails, and dominates current Ajax usage. Perhaps more striking, it is becoming more common in public use, exactly the territory where prior agreement was deemed most important [Bye XML]. It hasn't replaced XML in that space yet, but is claiming a larger and larger share. Documentation and samples, it seems, is enough.

So why stick with markup and not just leap to JSON's more open approach? Mostly because of the tools described in the previous section. Markup understands transformation and decoration better than JSON. Despite its largely schema-free world, JSON is still primarily about tight binding to program structures. The schemas are invisible, often unspecified but they still exist when a document is loaded.

However, JSON programmers do plenty of transformation internally. They base their expectations of schemas more on sources than on documents, and some have gone so far as to establish simple source-testing regimens that warn them of change. Source-based versioning is also common in the world of JSON APIs. Rather than the URI of a namespace changing or the details of a schema, new versions of APIs are often simply hosted at new URLs, with the changed content coming from a new location to give developers time to adapt.

JSON's curly braces may earn sneers from XML developers who prefer their angle brackets, but JSON is doing more with less.

Polyfills

XML was supposed to reach the browser. Mostly, it didn't, but the current state of the browser has much to teach XML. Some of that is ugly, of course. Even without a hard focus on schemas, the power of the browser vendors and the continuing insistence on standardization have limited possibilities. Here we see that it is not only the formalization of schemas but the cultural values surrounding them that create brittleness and stifle experiments.

However, that very brokenness, failed versioning, and the lingering hangovers of old software led to the development of a new software pattern, the polyfill [Sharp 2010]. Polyfills are JavaScript code (sometimes combined with HTML and CSS) that quietly extend a browser to support JavaScript libraries that are missing or markup that it doesn't understand. Building cross-browser polyfills is tricky, but [Osmani 2011] not that much more difficult than creating cross-browser frameworks. Even the limited world of HTML5-specific polyfills is vast [Polyfills List].

While the HTML+CSS+JavaScript architecture is extremely flexible, there are some barriers, largely created by browser makers' efforts to optimize bandwidth and processing. Efforts to create a picture element - combining concerns of responsive design and human interaction - have faced challenges around timing, pre-loading in particular. Media processing is (as usual) one of the hardest challenges in working with the browser environment. Establishing communications and respect between vendors and those using their products in perhaps even more difficult.

The W3C and browser vendors are working to address those challenges as well as create new frameworks that make polyfills easier to build and more efficient. The Shadow DOM [Shadow DOM] and Web Components [Web Components] work both aim directly at making polyfills a more standard part of web development environments. Google's recent work on [Polymer] is an example of a browser vendor pushing hard in this space.

Separately, the Extensible Web Manifesto [Manifesto 2013] encourages this work as a way to shift much work done in JavaScript to work done in markup:

While XML processing models are typically different, offering no tools to extend processors at the document creator's discretion, this approach could prove useful in situations where processors have chosen to place more trust in users. It also clearly offers an option to developers tired of waiting for the W3C, the WHATWG, and the browser vendors to add markup functionality to HTML5.

Relational Decline

Structured data has had a difficult few decades in general. While XML's schemas defined structure, relational database purists (most notably Fabian Pascal) heaped scorn on XML's daring to step outside the sharply defined boundaries of RDBMS tables and joins. Much of the pressure for XML Schema's insistence on deterministic structures and strongly typed data came from communities who considered the constraints in 1990s RDBMS practice to be a good thing - but XML's very success was a key factor in making clear that the relational model was not the only possible story for data.

The challenges of scaling within the constraints of Atomicity, Consistency, Isolation, and Durability (ACID), led to several rapid generations of change in the database community. While there are probably more relational databases deployed today than there were when XML appeared, the NoSQL movement has ended the era when developers only chose among relational databases unless their project was extremely unusual.

This shift has little direct effect on markup processing, but it does reduce the cultural pressures to only create data structures conforming to a well-known schema.

RESTfulness

REST is a communications style based on HTTP's limited number of methods, treating those constraints as a virtue teaching us to build with few verbs and many nouns. There is nothing in REST-based work specific to schemas - schemas (or the lack thereof) are a detail of the work that happens inside the local processors.

However, in contrast to their more RPC-based predecessors, which emerged from the CORBA and object-oriented worlds, this lack of specification is still a significant opening. A minimal set of verbs makes it much easier to process a much larger set of nouns, with fewer expectations set up front.

Strictly Local Uses of Schemas

Some developers and organizations see schemas as a limited-use tool, applied primarily in a local context to reflect local quality assurance and document creation needs.

So long as authoring applications expect schemas as input, schemas will be necessary. There are other ways to do quality assurance, of course, but schemas are common. Will developers resist the temptation to apply schemas more broadly than these cases, when tools and practice point them that direction?

Transition Components - DSDL and MCE

Much of the best thinking in markup schemas has worked under the banner of DSDL. RELAX NG, Schematron, and some more obscure pieces demonstrate more flexible alternatives to the W3C's XML Schema. Namespace Validation Dispatching Language (NVDL) finally offers tools for mixing validation approaches based on the namespace qualifiers applied to content. Document Schema Renaming Language (DSRL) offers a simple transformation approach for comparing documents to local schemas. These parts are still tightly bound to schema validation approaches, but they at least add flexibility and add more options.

Markup Compatibility and Extensibility (MCE), coming out of the Open Office XML work, finally asks hard questions about different degrees of "understanding" a document:

As Rick Jelliffe describes it:

Examplotron: A Bridge?

Examplotron Examplotron is in fact a schema language, but it is also a possible bridge between expectations from the Age of Schemas and other possibilities. It uniquely combines communications through sample documents with the possibility of validation processing, and seems like a base for describing further transformations. To the extent that a schema technology could work successfully within System A, Examplotron is clearly the strongest candidate. (Schematron isn't too far behind, but lacks the document-sharing orientation.)

Negotiation Style

Negotiation in the schema sense is typically a gathering of opinions, working together to hammer out a schema for future communications. To avoid the cost of holding that conversation frequently, it makes sense to gather as many possible voices as possible, though that "as many" is often slashed by requiring that the voices be 'expert'. After all, too many people in a conversation also adds delay, often for little benefit. The result is a formal structure that can be used to share data, a form of contract among the participants and anyone else who wishes to join that conversation.

Most forms of negotiation, however, are much less formal than a diplomatic council or standards organization. People bargain constantly, and not just over prices. The forms of communications within business change constantly, often fitting the chaotic model of shared spreadsheets much more readily than the formal structure of relational databases. While spreadsheets carry all the headaches of individual files that are easily misplaced, there is nothing inherently superior about a tightly-structured centralized database. The advantages of databases emerge almost exclusively when large quantities of data need to be shared according to a formal process. Telling a story with a database typically requires sifting and selecting its information to be presented in a different form.

The negotiation style of spreadsheets typically works according to a simple process: "send me what works for you, and I'll see if I can make it work." There may be massive gaps of spreadsheet prowess, business understanding, tools choice, or sheer power between the participants, but having a concrete basis for the conversation generally eases those, or leads to a request for simplification and a broader conversation.

Exchanging XML documents has a similar concrete effect. Marked up documents are not that difficult to explore so long as you have a rough familiarity with the language used for the markup or willingness to consult a dictionary. Just as there might be questions about how a spreadsheet is structured, there can be conversation about markup structure choices. Just as the contents of a spreadsheet can be questioned, there is room to discuss whether the model the spreadsheet uses for a particular problem is the appropriate one. Just as with spreadsheets, there is rarely need to ask for a deeply formalized representation of the underlying model.

XML has one major advantage over a spreadsheet, however - it is far, far easier to extract information from XML documents than from spreadsheets. Writing code that extracts content from person A's spreadsheet and reliably places it in person B's different spreadsheet is possible, but difficult. Changes make it even harder. Spreadsheets weren't built with that in mind. Markup was.

Extending it ourselves

In one context, this is easy. Because the HTML world allows developers to send code along with their markup, the situation in browsers is actually simpler than the world where markup is the sole content of messages. Markup in the browser is surrounded by opportunities to explain what it is and even to do something with it. (Markup plus logic is at least as capable as spreadsheet cells and logic, after all!)

The picture polyfill debacle seems to tell the story of callous browser vendors who have their own bad ideas blowing off the people who use their projects most intensively. In the long run, however, it is more of a pointer to the way things can be. CSS and JavaScript have developed to the point where they can tell the story of markup, with HTML necessary - strictly speaking - for a very few tasks like form fields. ARIA can provide metadata supporting accessibility, and if things are really tough, XSLT (and JavaScript implementations of it) are now available to transform documents in whatever markup vocabulary to an HTML equivalent, perhaps enhanced with SVG or Canvas.

Browser vendors get lots of press, good and bad, for implementing new features and new APIs. The quiet reality, though, is that there is no longer any good reason for them to control our markup vocabularies. We can do that, and we can do it today.

Changes Inside

Supporting a more flexible negotiation style in cases where documents travel without supporting logic requires a major break with prior models of development.

Most software for processing documents created by markup - not all, but by far the majority - follow a consistent process. A parser reads a document, checking its structure against syntactic and possibly structural expectations. During or after that reading, if the document passes, the data is bound to internal structures, possibly using additional information added to the document by the testing process.

Perhaps most important, however, isn't the nature of the parser (generic or not), the schema type used, or the nature of the data binding. The most important aspect of this is that each receiving process only supports a single or a few vocabularies. (The primary exception is for completely generic processing, which doesn't care and generally doesn't know what kind of markup it's dealing with. This is common in editors and some kinds of filters.) Extending that support, changing the vocabulary expectations, requires programmer intervention.

Avoiding programmer intervention has been the foundation of data processing for decades. Much of the joy around schemas celebrated that schemas themselves could be used to create code for handling markup vocabularies. Yes, of course you still needed to add business logic to that code, but generated code would reduce the time needed to write, and adoption of common schemas would even allow the sharing of at least a portion of the business logic code over time.

The design choices in schema and schema culture come from the dream of a machine that would run by itself, with only periodic human intervention when updates were necessary, plus the usual maintenance now considered the cost center of IT.

So what might a different model look like?

The machine doesn't run by itself. Human intervention is routinely acceptable, and facilitated by a toolset that assumes that humans are available to do the mapping previously performed by schemas, data-binding, and similar approaches. All participants in a conversation have their own processing structures, but those structures are explicitly adapted to internal needs, not external conversation. (Yes, it is possible that one of those internal needs will be to facilitate external conversation, but that need should not set the terms in most cases.)

Human intervention isn't required for every message, however. Over time, such a processing system will develop a library of known transformations to and from external vocabularies and in particular to and from known senders. Intervention arises when new or ambiguous information arrives. A new party has joined the conversation, or a previously known sender has added or deleted sections of a document. In short, intervention is necessary when prior mapping failed.

Intriguingly, this approach can fulfill one of the failed promises of XML Schema:

Schemas can flag information as outside of the schema and still pass it on, but it has no way to determine the 'safety' of such information or to encourage change, unless someone is monitoring the schema processing closely. By contrast, such cases are not anomalies in the mapping system - they are just day to day business.

But wait, isn't there still a schema? Isn't the transformation from external representation to internal structure a hidden schema? HTML5 style? Perhaps, especially if there some kind of document describing that transformation that can be shared. However, unlike the schemas that currently dominate the markup world, these transformations don't describe a single document structure. They describe a set of possible mappings between a document structure and a specific internal structure. That substantially reduces the prospects for (abuse) reuse.

Also, while it is possible that a transformation would return a binary succeeded/failed response like the valid/not-valid result of most schema processing, there is a much broader range of possibilities. Partial mappings may be completely acceptable. Failures may simply be prompts to create new mappings.

The cost of programmers, though. What about the cost of programmers? Their time isn't cheap, and we have other things for them to do! Even Bray and Bosak cautiously warned us that programmers were necessary, and schemas would only reduce the amount of time they needed!

These transformations aren't usually that difficult. Simple transformations are easily designed through basic interfaces. Complex transformations may require template logic on the order of XSLT. Especially difficult transformations, particularly of compact markup styles might require an understanding of regular expressions.

Except perhaps for especially twisted regular expressions, however, none of these skills requires what we pretend today is a "rock star" programmer. There are many contexts, but the home context is relatively simple, and transformations themselves are more intricate than deep. While establishing this model might require regular contributions from experts, and having experts available is useful, over time the need for expertise should decline. Transformation should not be as difficult as, say, natural language translation.

Rehumanizing Electronic Communications

Changing tracks is difficult, especially when an entire technology culture has been built on values of industrial and bureaucratic efficiency aimed at minimizing human involvement in small but constant processing decisions. Pioneers created vocabularies for the purpose of controlling a market and leaping ahead of competition, hoping that the resulting brittle processing model would make it too difficult to refactor their advantage away.

However, rather than striving for maximum automation, developers have the opportunity to aim for systems that are both more flexible and more human than current models allow. Developers can build on a model of automating what humans hate to do, rather than automating everything. The three great virtues of the Desperate Perl Hacker displaced by XML's arrival - laziness, impatience, and hubris - can take their rightful and continual place in a community of processing rather than disappearing in a cloud of efficiency. Automate what is convenient for humans communicating, not entire processes.

That changes the role of markup experts as well. Instead of consulting on vocabularies and leaving others to implement them, we have to take a more active role on a smaller number of projects:

Some of us work this way already, of course.

Working this way also allows to us focus on one of Alexander's most general patterns in his question for local adaptation: site repair.

This means continuous mending, continuous repair, by people close enough to the ground to see what needs work. Continuous refactoring has entered the programming lexicon, and needs to become a dominant part of the markup lexicon.

In a world saturated with the message of industrial efficiency and in a computing culture deeply soaked in expectations of standardizing away as many interactions as possible, this Alexanderine model will be at least as difficult to achieve as it is in architecture. Possible on the margins, yes, perhaps in experiments, but not to be taken seriously. That is a normal starting point for this conversation, and this paper just an opening.

Markup workers of the world, unite! You have nothing to lose but your chains.