Balisage Paper: Your Standard Average Document Grammar

From clay tablet to PDF

There appears to be a set of structural features common to the majority of text documents that have become a part of the way the human race has recorded textual information over the millenia. As I have shown elsewhere, it is apparent that from clay tablets to PDFs, we have slowly evolved various models of a document that have many features in common [Flynn14, Ch.1]. Part of this may be due to the need — until recently — to agree upon a generalized physical representation for the document that others would recognize, but this could not have been done without there being a mental model of the document to start from. It is not known if anyone actually sat down at the dawn of writing, or even at the dawn of printing, to decide that certain features are what makes up a document,^[1] but we can see evidence of such decisions in the design of commands and structures in older markup systems such as RUNOFF, Scribe, [S]GML, LaTeX, and others which inherit their paradigms.

Strictly speaking, a document grammar (in the case of XML, for example, a DTD, W3C Schema, or RNG Schema) is a set of definitions and declarations for modeling a class of text documents. It defines the components of the documents they describe, as well as the rules governing their presence in the documents of that class [Tekli11] — a similar application has been noted in linguistics [Power03]. However, we are more concerned here with the document components themselves, and with the rules governing their arrangement, than with the expressive power of the particular grammatical notation used to describe them.

Feature set

The set of features for the derived grammar is expanded below, but we should first deal with what it does not describe.

There are many classes of document structures that do not or cannot follow a generic model but have their own: those which are too short to exhibit much in the way of structure; those which are intended as ephemeral or singular; and those which by convention of their nature require a specialist structure. But even amongst these, some of the features may be present, even if (for example) in the metadata rather than the text body.

The point of standard and average as described above is that such a grammar should be able to cover enough of the spectrum to be a useful pattern or model in a majority of cases, and that this fit should be generally accepted by the user community. There will nevertheless be some specific factors which must be considered in testing this acceptance:

Naming is also important, and has been the topic of much discussion over the years on XML-related mailing lists. Not only are names a prerequisite of any concrete instantiation, but we need them informally as handles during discussion, so they may as well be meaningful in the language of that discussion. This raises other linguistic and cultural questions, but in essence we are simply requiring agreement that the feature we refer to as a title is in fact the title of a document (or section, or whatever) as commonly understood, and not a mosquito or a bottle of beer.

Because of the traditional separation of concerns between logical and physical in dealing with document markup, the visual appearance of a grammatical feature is not generally relevant. However, for the purposes of usability and — as here — illustration, when features are given an appearance, it is common to use one of the widely-accepted styles.

The salient features of a Standard Average Document Grammar are summarized in Figure 1 to Figure 4. There may be disagreement over the presence or absence of some specifics, but enough of these appear to occur in enough instances of otherwise disparate types of document to make it worth inclusion.

Figure 1: Identification

Naming and explaining Title Subtitle Author Summary

The features in Figure 1 are often regarded as metadata, as they typically stand outside the running text. It is nevertheless seems to be accepted as part of the function of the grammar that it should label the document (title), link to an authority outside the document (author), and provide an overview or synopsis (summary).

Figure 2: Formation

Hierarchical structure Preamble Major Division Subdivision Minor Divisions Postamble

The core structure of a document appears most commonly as a hierarchical nesting of divisions, with each level able to reoccur as siblings (Figure 2). As encoders of documents are well aware, this does not hold true for many early documents, and even for some contemporary ones, but it is sufficiently true elsewhere for it to be useful as a model, and is sometimes imposed upon otherwise unstructured or semi-structured documents to make them usable in conventional modern contexts. In formally-published documents, especially books, there is usually material preceding and following the hierarchical structure (prefaces, forewords, indexes, appendices).

Figure 3: Text Content

Recurrent, reusable, ordered Paragraph List Item Table Figure Quotation Image Notation

While the function of a hierarchical structure is to provide a referential framework within which the author can develop or express an argument (at the least, something like introduction, exposition, analysis, and conclusion), the text itself uses a set of building-blocks to present that argument (Figure 3), of which a small subset seems to be widely used.

The critical point about these building-blocks is that they occur and reoccur many times. While the components of the hierarchical structure which contain them may reoccur as often as needed as siblings (that is, at their own level), they cannot occur out of depth (that is, you cannot have a subsubsection as a child of a chapter), whereas the building-blocks of content can occur and reoccur at any level within the hierarchical structure. Whatever about the constraints imposed by the hierarchical model, this distinction seems to be a key aspect of document grammars.

Figure 4: Reference

Completion Internal Link External Link Signifiers importance relevance sequence

Unlike the other features in Figure 1 to Figure 3, where at least one of them must be present, otherwise you have no document at all, the reference features are entirely optional, and are used at the author’s discretion according to sense (Figure 4).

In the detail of running text, there may be a need to link components within the document for reference or to link to other documents elsewhere. While these features perform a closely related function, an internal reference can be checked immediately, so it is dependent, whereas a link to another document is independent, as it cannot be known at the time of writing if the reader will have access to the document concerned.

Signifiers are ways to express some special nature of a feature, so that it takes on a quality which impresses itself on the reader. Emphasis or terminology are probably the most frequently-used in continuous text; specifiers of sequence occur in structures like numbered lists and the titles of sections.

Adopt, Adapt, Build^[6]

In this author’s experience, the core set of features, or one very similar, is where most concrete instantiations of document grammars appear to have started, as far back as the days of SGML DTDs. Additional features, and deviations from the norm, are legion, and may be specialist within a field or topic, or introduced for practical, technical, or political reasons — it is these which distinguish one implementation from another. The ease (or otherwise) with which a particular type of document can be modified seems to depend largely on the original authors’ intentions:

Not all parts of a document grammar may be equal to the task: in some cases it may be hard to modify the metadata but easy to modify the hierarchy; in others the reverse. There is also significant debate (not a part of this analysis) about the extent to which modifications should allow or deny a user the right to continue to claim that they are [still] using the type of document they started with.

Drawing the line

If there is anything we can learn from a Standard Average Document Grammar, it seems to be that it’s a convenient term for a phenomenon which needs more accurate measurement. One way of looking at it would be to pursue the pseudo-statistical theme and construct values for concrete use cases, with their distance from the theoretical SADG as a measure of divergence.

When an organization or individual considers using an existing document grammar, there will eventually be a pain point at which they in effect say, No, that really isn’t how we see things here, we need something closer to how we work. From that point on, it’s a case of adaptation: new names, perhaps, or a new structure, or an extended or contracted content model. If such a fork is public, it may attract additional users, particularly if it is designed for a vertical market. Takeup and the amount of divergence from the original can be measured.

Some will never get to that point, and will use an existing grammar unadapted, or perhaps with only the most trivial of changes to, say, attribute value lists. In these circumstances, we are effectively adding to the number of use cases at the mode (the most commonly-occurring value of an average).

Those who elect to build their own grammar are in effect initially located beyond some as yet undetermined measure of deviation, although if the resulting structures end up bearing enough similarity to the SADG, the grammar may be considered have added to the base of contributory systems.

In this author’s experience, the adaptations of existing grammars are undertaken for multiple reasons, but often related to not enough or too many or not what we call it:

Cutting back on the richness of some of the standard offerings is likely to ease editing complexity, but there can also be extra work if some components are named in a way that causes ambiguity or uncertainty in the circumstances of use. When this reaches frustration point among document users, there may be a rise in tag abuse or other inaccuracy, leading to calls for adaptation or writing a new grammar.

Given that the creation of a new document grammar and new document type or class is non-trivial, it would be useful to have some measure of how far off-piste you have to be to justify it.