Balisage Paper: Abstract generic microformats for coverage, comprehensiveness, and adaptability

When descriptive schemas work

The short answer is that how well a descriptive schema works in operation depends not only on its fitness for the data and task (i.e., whether its coverage is appropriate and it is adequately comprehensive), but also to the extent that document encoders have editorial control over the information, both in its content and organization. This should not come as a surprise, since indeed the roots of XML technologies are in publishing systems in which it is more or less imperative to exercise such control, and the promise of externalizing, rationalizing and simplifying these control mechanisms is much of the point of descriptive markup. The classic application of this principle, of course, is in the publishing of a manual, technical series or periodical, in which authors or contributors (on the one hand) and production staff and designers (on the other) assent to creative decisions of editors, and document markup can serve transparently to represent editorial intent. When control is centralized in this way, the circle is complete: once an encoding format is defined, content can be selected, composed and modified to conform to its capabilities. Requirements for change and adaptability are managed in balance against other needs for consistency and stability. System operators resort, when necessary, to editorial policy-making to forestall, prioritize, and schedule any innovations likely to have a destabilizing effect.

The limits of description: the case of the book

The inverse is also true. Descriptive schemas work well to the extent that control over the data is centralized and consistency in the application of markup can be enforced. But when it is not so centralized, they do not work so well. This can happen anywhere, but it happens most notably and inescapably in a community of use which shares a common schema or tag set (using that term to identify a family of related schemas) without sharing local goals and objectives. Maybe they have an interest in a common encoding standard for purposes ostensibly of document interchange; similarly, they may wish to take advantage of common knowledge and a shared tool base. Of course, this is exactly the case with the big general-purpose document encoding standards and community initiatives, such as TEI, Docbook, and NLM/NISO.

The limits of a descriptive encoding can be readily illustrated by considering information artifacts in the wild – that is, as actually found, and apart from any considerations of how they may be encoded electronically (or not) by the authors, editors, designers and publishers who have composed and produced them. Texts in the wild – even if we limit consideration to those texts we are content to designate as books – are not perfectly regular, but only mostly so. Indeed this semi-regularity is an important (perhaps the most important) general feature of books, inasmuch as significant variation within pattern might serve as a definition of information itself: the structures of books cannot be entirely regularized without sometimes destroying the book. Sometimes this works at a very granular level.

Figure 1: Specialized structures across a series of books

Michael Kay's XSLT 2.0 and XPath 2.0 Programmer's Reference is a convenient example of a book whose internal structures are more elaborate than may be typical of a generic book. Yet unless we wish to say this book is not a book, but something else, a viable book model has to support such structures either natively or by means of extensions to be introduced by the author or publisher. It is possible that the markup needed here could be useful across all the books of the series to which this book belongs; in any case, it might be useful to the publisher and to application designers who wished to design interfaces to this content if it were.

Figure 2: Specialized structures across a single book

Stafford Beer's Platform for Change (1975), like many other literary or quasi-literary productions, is a fascinating example of a book whose structure is peculiar to itself. Beer's volume is not unusual in having figures and diagrams, although one appears on the inside of the dust jacket; but it does have structured prose (it is not verse, or so the author asserts), and its chapters are differentiated by type, as indicated by page colors.

Does a generic book model need to be able to accommodate works of this kind? We can agree to say no, this book is out of scope: the coverage problem again. Or say that although it is a book, we will force-fit it within available semantics (risking tag abuse to deal with inadequate comprehensiveness). Or we have to find a way of dealing with its special requirements: the adaptability problem.

For what it's worth, even unextended TEI does not provide elements directly capable of marking up this text, as it has no elements for lineated (structured) prose, only verse.

Figure 3: Specialized structures within a single book

Thomas Carlyle's Sartor Resartus (1833) should be required reading for all students of semiology. It is mostly conventional in structure; or at least, much of its point is in how it observes some of the more elaborate conventions of nineteenth-century scholarship. But an epitaph pops up in the middle. Students of literature know that such things are less uncommon than you might think, even leaving aside Tristram Shandy. (Illustration from Google Books.)

Again, we have either to extend our coverage of what's a book (or what's in a book), or provide a way for a markup language to be adapted to purposes like this.

Three examples appear in Figures 1-3, showing this phenomenon at the level of the series (the Wrox Programmer to Programmer series, shown here by Michael Kay's XSLT 2.0 and XPath 2.0: Programmer's Reference^[2]); the level of the book (Stafford Beer's Platform for Change); and within the book (the epitaph appearing in Sartor Resartus). Of course, many more examples could be shown to demonstrate how every book – and every structure within a book – is potentially a one-off, a thing unique unto itself. In part, this reflects the fact that books as functioning information objects can and must be designed to serve sometimes very distinct and peculiar purposes, which may even be intended specifically to differentiate them from other books. One of the core requirements addressed by book design, that is, is that a particular design may be distinctive, at least in part, not only in its features but even in the requirements it addresses, which will be different from the requirements addressed by any generalized design. This reflects a different balance of power in book publishing from journal publishing: the center of gravity for design decisions shifts from publisher to author – which is why, as a class, books are less tractable for descriptive markup than journals that maintain a consistent structure (expressed in their design, layout and page formatting) across articles.

Of course, the irony here is that this will be true of books as a class even while individual books or certain genres, types or formal series of books might well be suitable and profitable objects for description. (And so we generally deal with this problem by limiting our coverage to particular books, leaving books in the abstract out of scope.) Because coverage – what, exactly, constitutes a book, and what sorts of documents are not books – is not well defined, the comprehensiveness of any markup applied to nominal books will also be at issue. In practice this means that descriptive encoding hits a wall, since to whatever extent we try and provide appropriate handling of every structure we see, our markup language grows into a Tower of Babel: a single megalo-description supporting this differentiation ultimately fragments under its own weight, since any instance may have structures peculiar to it, while no instance includes all (or even many of) the structures necessary for the others. While this may be a fascinating state of affairs to a media critic or markup language designer, it is no basis for a common information standard that needs to normalize description at some level, if only to enable and regulate processing.

This paradox results from the foundational insight of descriptive encoding. The operative dogma is that we can and should clearly distinguish presentation from content, where the organization of content is regular and in some way normative, even while the presentation and indeed application of that content may vary. (In effect, this is a model/view/controller paradigm applied to documents, in which the document itself is a model that can be subjected to multiple views in one or more applications as controllers.) As noted earlier, this tends to be the case within collections of documents that can have such regularity imposed on them – consistently designed works such as journal articles or technical manuals, that is, where control of both content and composition is top-down and centralized.^[3] So the design of markup languages is focused on defining that regularity and normativity, in a sense leveling the differentiation into those classes of things (we try to identify these with element types) that can be adequately, if not always comprehensively, distinguished with an encoding system.

Yet in order to do this, we have to pretend that some distinctions between things which we choose to identify as the same type are not, and cannot be, significant. And we never know which one or another of the distinctions to which a particular markup system has a blind spot will turn out to be important as soon as we turn the page of a book or of any work outside our controlled set. In other words, depending on the type of work, anomalous structures may be more or less unusual, and regarded as more or less anomalous; but when they occur, they can be very anomalous. The structures of actual books, that is, have a fractal quality, inasmuch as what constitutes an anomaly (as opposed to merely another regular if unusual feature) depends on the level of scale – and yet there is no level at which anomalies disappear.^[4]

Different markup languages deal with this problem differently, depending on the extent to which their design targets a distinct genre or application domain. But in every markup language, we can see this stress between coverage and comprehensiveness on the one hand, and adaptability on the other. To whatever extent coverage is well defined and comprehensiveness is achieved within it – by virtue of the meaningful distinctions being made and the incidental ones elided – adaptability also becomes important – because a distinction never made in one context, or made only incidentally, becomes meaningful in another. (In contrast, precisely because it is relatively uncomprehensive across a very wide coverage, a markup language such as HTML needs to adapt less; the price for this, of course, is that its semantics are weak.) But adaptability by its nature threatens the integrity of the boundaries (the coverage) that make the semantic labeling function of a markup language useful.

The root of the problem here is in a question not often asked about descriptive markup (except, possibly, at Balisage), namely what is it that markup should be intended to describe. Sometimes, naïve proponets of descriptive markup have been content to argue that what is being described is the thing itself, but this of course begs the question. A successful markup application works precisely because it does not seek to capture every distinguishable feature or detail of anything it describes, but instead represents and abstracts from it. The model it offers is an optimization: more accessible, tractable and useful, at least for certain purposes (including automation), than the thing itself. This reduction of differences to (supposed) essentials reflects the way, in information processing systems, everything we do is on the basis of the differences and differentiations (and similarities) we can register and retain within and between different data objects, as is well established in information theory from Saussure [Saussure 1916] to Shannon [Shannon 1948].^[5] And this means that we necessarily make assertions (by our names and labels) that erase some distinctions between things, precisely so that we can assert others. It is in recognition of this that in SGML terminology, an element name is called a generic identifier.^[6]

This erasure of distinctions by calling different things by the same names, as long as they are alike in certain crucial aspects – crucial being defined in the relation between an information type and its application domains – is the balancing factor that prevents descriptive markup from going out of control and failing in its mission sine qua non, namely the automatability that follows on our ability to know in advance the limits we will work within. For a markup application to be viable, in other words, it must have a fairly limited set of elements, whose semantics are known and respected and whose structural arrangements are predictable (indeed this is the entire point of a schema).^[7] To the extent this set grows and changes (either because new elements and attributes are added or because the semantics of old ones are elaborated), viability is endangered, to the point where (if the rate of change outstrips the capability of the rest of the system to change with it) applications of the language become unsustainable and the language itself becomes, at best, a literary artifact.

Of course, this limitation inhibits adaptability, insofar as an element set defined in advance will (ipso facto) fail to account for any semantic features of the information type that are newly defined or recognized, whether accounting for new phenomena to be described, addressing new applications, or for any other reason. And this is the crux of the paradox, the precise dilemma on whose horns the problem of descriptive markup is stuck.

A proposal

So here is a straw-man proposal for a set of such elements and attributes:

It is important to stress that this element set would be intended to supplement, not replace, the basic core elements of a capable document-oriented tag set – although it is also conceivable that a valid (and useful) document might contain nothing but these element types.

One limitation of abstract generic elements is if they cannot be provided with arbitrary attributes. How severe an encumbrance this would on the general use of abstract generics for ordinary purposes can only be known by trying it.

But at least two possible workarounds can be imagined: (1) where attributes are needed to provide subclassing, either or both the @class or @type assignment of the abstract generic can be overloaded. (2) In general, child elements or element structures can be recruited for use to carry values that might otherwise be given in attributes.

An example: an approach to drama

An example in TEI markup:

<stage>Enter EGEUS, HERMIA, LYSANDER, and DEMETRIUS</stage>
<sp>
  <speaker>EGEUS</speaker>
  <l>Happy be Theseus, our renowned duke!</l>
</sp>
<sp>
  <speaker>THESEUS</speaker>
  <l>Thanks, good Egeus: what's the news with thee?</l>
</sp>
<sp>
  <speaker>EGEUS</speaker>
  <l>Full of vexation come I, with complaint</l>
  <l>Against my child, my daughter Hermia....</l>
</sp>

The same example could be tagged as follows in a schema that supported abstract generic elements:

<div type="drama">
<line type="stage">Enter EGEUS, HERMIA, LYSANDER, and DEMETRIUS</line>
<pair type="sp">
  <line type="speaker">EGEUS</line>
  <ab type="speech">
    <line>Happy be Theseus, our renowned duke!</line>
  </ab>
</pair>
<pair type="sp">
  <line type="speaker">THESEUS</line>
  <ab type="speech">
    <line>Thanks, good Egeus: what's the news with thee?</line>
  </ab>
</pair>
<pair type="sp">
  <line type="speaker">EGEUS</line>
  <ab type="speech">
    <line>Full of vexation come I, with complaint</line>
    <line>Against my child, my daughter Hermia....</line>
  </ab>
</pair>
</div>

It is important to note here that stylesheets or display engines that don't know what to do with these attributes could nevertheless do something reasonable with the elements. For processes that need the more specific semantics, they are available – as they are available to be validated using Schematron or by way of a transformation into a more tightly defined schema.

Merging with a host language: tight or loose?

At least two different ways of integrating a set of abstract generics with a host language can be considered:

It seems reasonable to suppose that both these approaches could be useful for different reasons and in different contexts.

Validation and validibility

Introducing custom semantics in a secondary layer using attribute values raises the problem of how to validate them. Obvious alternatives for validation of microformats include RelaxNG and Schematron. Additionally, transformation pipelines can be used to formalize the relation between a microformat defined using abstract generic elements, and more tightly defined elements in target markup languages.

Of possibly greater importance, however, than the means of validation, might be the shift in emphasis on stages of validation necessitated by the presence of abstract generics in the model – since the very idea is to relax both the demand and the expectation for uniformity and consistency of element usage across system boundaries. Validation will no longer, in other words, be as simple as thumbs-up vs thumbs-down against a common schema, but will also entail more nuanced specifications and tests.

Interchange and interoperability

Isn't this an invitation to chaos, as users are encouraged to introduce their own idiosyncratic usages of the abstract generic elements?

Whether allowing microformats based on abstract generic elements would in fact make it harder than it presently is to achieve interoperability or even interchange across organizational boundaries depends on what we take that baseline to be. To the extent that stronger typing of schema-validated element types now supports interchange, obviously to rely on labeling abstract generics to assert the same semantics in the instance would be a step backwards. However, it should be considered whether experience has not shown that beyond a very basic level (hardly beyond the level of XML well-formedness), interchange cannot be guaranteed by schema validation alone in any case, at least for properly descriptive markup languages, not markup applications that are essentially serializations of object models bound to applications (such as XSL-FO or SVG).^[13]

There is, in other words, a difference between the mythic ideal of interchange, and the actual fact of it; and in fact, when it is achieved at all, data interchange at scale happens as a deliberate outcome of a negotiated process, in which a shared tag set may be a useful starting point, but will not by itself be a guarantor of success. It might be better if we considered transparent or blind interchange to be a functional requirement to be considered among others – and considered in its particulars as they apply to specific parties to interchange rather than simply asserted as an abstract principle that remains unrealized. Doing so would mean that we would be free not only to consider the actual costs and benefits of achieving interchange, but also to balance this requirement against others, such as the need for expressiveness and adaptability simply so a tag set can address local goals that are, after all, sometimes more pressing.

In view of this, allowing document creators and editors the means to extend the semantics of their markup openly would be better than the status quo, in which they do this anyway, but hide their practices behind the premises of interchange offered by standard semantic tagging. When standard semantic tagging is used in non-standard ways, in other words, it is not standard any more. This is why we call it tag abuse, because such practice not only mislabels the contents with elements designed for other purposes; it also degrades the semantics of the misused element type even where it is used properly.

Since abstract generic elements are designed to accommodate the very behaviors that constitute abuse when used on properly descriptive elements, they would (at least to the extent they are used) should relieve the pressure to do so. And to the extent problems with interchange and interoperability will exist across systems that use the abstract generics differently, it is reasonable to wonder whether it does not actually reflect the requirements for description itself, which would otherwise have been hidden behind creative tag use and misuse. That is, to the extent a system uses abstract elements instead of abusing other elements, its tag usage will be more open and transparent – while the abstract generics provide a base level of interoperability even without any implementation of their local semantics.

Thus, when parties to interchange find that they don't know how to handle the semantics of one another's abstract generics, they will have a basis for identifying, isolating and analyzing the issues. Already, markup systems engineers who are not deploying their own schemas or schema customizations are doing this all the time, and successfully, with HTML. (The success does not come without effort; the point is that it is possible at all.) To introduce abstract generic elements and encourage the development of microformats on that basis would not be to license bad behavior so much as it is to adopt and appropriate the good motivations that might, under too strict a regiment, lead to it.

Addressing production workflow concerns

One of the virtues of abstract generic elements with microformatting is that it fits well within a production model in which XML encoding happens later, after authors and designers have made commitments to structures in their content that serve expressive and rhetorical purposes rather than the goals of any technology as such. In other words, if a tag set is to be a suitable target format for upconversion of arbitrary content (such as is produced in word processors or derived by OCR from page images), it needs to be arbitrarily flexible, if only to capture structures at the edges of normal. Then too (and for the same reason), most authors who have worked in XML have also had to confront limits in the expressiveness of the tag set in use, and would welcome an opportunity to use more abstract tagging, with attributes to signal semantics for possible application downstream, when the situation warrants it.

Yet both these scenarios raise the question of what should happen in a production system when these elements are encountered: clearly, these capabilities are somewhat at odds with processing requirements, since to hide semantics in attributes is in no way the same as extending an application framework to support them. Yet (as noted above) there is a great advantage in having a set of elements designated especially for this purpose, as opposed to the present state of affairs, in which this already happens, but in a way much less amenable to improvement, because the elements being appropriated are already designated for other purposes.

Again, layered validation, where only the basic structures are validated in the core schema, and supplementary validation logic is provided elsewhere (in schemas, Schematron, XSLT or custom logic), provides a way to negotiate the challenges here. The abstract generic elements are already provided with a baseline of functionality by virtue of their mappings to common display types. Provide for alerts in the system when unknown labels or combinations are used, and developers can then address them more specifically, whether by extending processing, retagging, or pushing the questions they raise back for editorial consideration.

Microformats as a driver for evolutionary change

A set of abstract generic elements can offer a way for users to mark up structures of arbitrary description and complexity as they are encountered. But if this concept works in the field, it will be due to the willingness of users and application developers to work to support it. Is half a loaf really better than none?

Experience with HTML shows that local users are able, and eager, to take good advantage of such capabilities, given the right level of support by generic tools. Furthermore, this reservation has its counter-argument built into it: anything that rewards users for taking such responsibility – confronting practical issues of schema design and document modeling and getting hands-on experience with transformations – will be a good thing.

While good design is accomplished by design, markup technologies do not develop in a technological or cultural vacuum, and no design can foresee exactly the ways in which it can be put to use. In this kind of world, we can rely also on evolutionary mechanisms – selecting and promulgating the most successful from among those attempted – to help us distinguish the best available solutions to any set of problems. But this requires alternatives from which we can select.

In a widely deployed tag set, variations in tagging practices are certain to occur; and in an environment in which many different approaches can be tried, some will persist and spread. While including a set of abstract generic elements will make any tag set better fitted for use across a disparate community, the argument for fitness does not end with this. Abstract generic elements may also serve as useful laboratories for tagging practice; and on this basis, common microformats for common kinds of data and common markup problems may emerge, and even micro-standards, perhaps, maintained much like the core standards although across more localized application domains within and across organizations.

It should be stressed that the aim here is to serve bottom-up as well as top-down design for XML applications in use – not because XML schemas don't work, but because they do. Precisely because, and insofar as, communities are able to take advantage of shared practices and tools, the needs of authors, editors and publishers to design their own models will continue to be felt. Giving them the means to do this will enable and empower them, while maintaining the integrity of the community standards they share.