Balisage Paper: Marking up and marking down

XML

If someone mentions markup, most of us think immediately of explicit markup. These days—for this audience—most likely XML. Given a text document, like this one, we embed XML start and end tags, entity references, etc. within it to impose a structure on it.

In fact, XML is a metalanguage: a language for creating languages. This paper is written using DocBook markup, because that’s what the proceedings require. At another conference it might have been marked up in JATS. Or TEI. If some future linguistic research is performed on the corpus of Balisage proceedings, perhaps it will be marked up in some entirely different way.

Consider the first paragraph of this section:

If someone mentions markup, most of us think immediately of explicit
markup. These days-for this audience-most likely XML. Given a text
document, like this one, we embed XML start and end tags, entity
references, etc. within it to impose a structure on it.

For these proceedings, it could be marked it up thus:

<para>If someone mentions markup, most of us think immediately of explicit
markup. These days&amp;mdash;for this audience&amp;mdash;most likely
XML. Given a text document, like <emphasis>this one</emphasis>, we
embed XML start and end tags, entity references, etc. within it to
impose a structure on it.</para>

In another context, it might be marked up like this:

<g13><s>If someone mentions markup<p>,</p> most of us think
immediately of explicit markup<p>.</p></s><s>These
days<p>&amp;mdash;</p>for this audience<p>&amp;mdash;</p>most likely
XML<p>.</p></s><s>Given a text document<p>,</p> like this one<p>,</p>
we embed XML start and end tags<p>,</p> entity references<p>,</p>
<q>etc</q><p>.</p> within it to impose a structure on
it<p>.</p></s></g13>

In fact, the number of ways that it might be marked up is essentially unbounded. Amongst the strengths of explicit markup is the fact that different structures can be imposed on the same text. (The weakness that different structures cannot, in general, be imposed at the same time with inline markup, is a subject oft discussed at this conference.)

At one level, this makes XML very extensible: new markup elements can be invented at will and used immediately. Namespaces provide a mechanism to ensure that the names one group invents are unlikely to be the same as the names another group invents.

At another level, however, XML is very inflexible and non-extensible. The specification leaves effectively no room for exensibility at the level of parsing:

This level of specificity (or brittleness) is part of a trade off between simplicity of parsing and intepretation on one hand and flexibility on the other.

Consider this random document with some markup in it:

<f29>"I believe so," <r>Alice</r> replied thoughtfully. "They have
their tails in their mouths—and they're all over crumbs."</f29>

While the semantics of the tags may be unknown, the structural interpretation of the document is wholly defined by the XML specification: it’s a document with an f29 document element that has three children: a text node, an r element, and another text node. The r element in turn has a single text node child.

SGML

SGML, like XML, is a grammar for creating new grammars. I mention it here for two reasons:

The SGML Declaration, DTD syntax, and other mechanisms such as notations and SDATA entities made SGML documents much more flexibile than XML documents. It also made the parsers much more difficult to write and the interpretation of documents much more difficult.

In an SGML context, it’s not possible to make definitive assertions about the intepretation of a random document like Figure 4. SGML (before the WebSGML corrigendum anyway) doesn’t attempt to define the intepretation of an SGML document absent its SGML Declaration and DTD.

We shall return to this topic later in section “The road not taken”.

JSON

If the sweet spot for XML and SGML is marking up “prose documents”, the sweet spot for JSON is collections of atomic values. If you are willing to squint and say that the collection of atomic values is analagous to the text document we mark up with XML, then JSON qualifies as explicit markup.

A typical JSON document looks something like this:

{
    "p": 3,
    "q": [
        false
    ],
    "r": [
        null,
        true,
        "true"
    ]
}

The two organizational structures in JSON are “objects” (unordered collections of name/value pairs) and “arrays” (strictly ordered collections of arbitrary types). Beyond these two structures, JSON defines only four data types: numbers, strings, booleans, and the literal value “null”. JSON is highly composable. The values in an object can be any JSON type as can the members of an array.

It’s interesting to note that XML has both of the organizational structures as well: attributes are unordered collections of name/value pairs and element content is an ordered collection of nodes. But XML is far less composable.

It’s not difficult to argue this XML document:

<p>This is <em>important</em>.</p>

And this JSON object:

{
    "p": [
        "This is ",
        {
            "em": [
                "important"
            ]
        },
        "."
    ]
}

are equivalent representations of the same underlying data: a “p” that has three “children” including one that is an “em” with one “child”.

As a matter of practicality, JSON documents are not at all constrained to represent data that has an equivalent form in XML. There isn’t any XML document that’s a lossless representation of Figure 5, for example. Consequently, JSON tools don’t lend themselves to efficient processing of the equivalent subset.

JSON “in the wild” is subject to considerable variation. The current official definition of the “application/json” media type is RFC 7159 but the format is a subset of the data structures allowed in Javascript. In as much as JSON data structures are often constructed by Javascript, it is not uncommon to find examples of objects and arrays that are putatively JSON but in fact stray outside the boundaries of RFC 7159. (For example, Javascript doesn’t require that property names be quoted.)

SGML and XML are defined with a concept of validity: their metalanguages define new elements (and attributes) in the context of a grammar that divides the universe of documents into valid ones that conform and invalid ones that do not. The definition of JSON includes no notion of validity at the structural level, but several proposals, such as JSON Schema, exist to fill this gap. If widely adopted, they would give JSON a degree of metamodel extensibility not unlike XML.

Syntactically, JSON is as tightly constrained as XML. Attempts to relax the syntactic constraints exist, for example HJSON, but they do so by defining wholly new syntaxes not be extending JSON in any standard way.

CSV

CSV files contain lines of text that are interpreted as “comma separated values” (or sometimes tab separated values, but it comes to the same thing).

In the interest of completeness, it can probably be argued that CSV also qualifies as a kind of explicit markup. In practice, there is tremendous variation in how applications interpret CSV documents.

This variation probably traces back to the notion that it’s “so simple it doesn’t need a specification” combined with implementors’ attempts to be “bugward compatible” with the tools with which they wanted to interoperate.

There is now a specification for the “Model for Tabular Data and Metadata on the Web” (W3C, 17 Dec 2015).

HTML

HTML, despite it’s surface similarity to XML and SGML, is a particular language, a particular set of tags, not a metalanguage. It was originally described using SGML, but never implemented that way. An XML definition was specified, but it was rejected by the browser vendors. Today, when we speak of HTML, we mean HTML5.

HTML uses angle brackets and entity references, and the realization of a document is ultimately a tree, but HTML is maximally Postelian. Where XML imposes both well-formedness and validity constraints, HTML imposes none. Every sequence of characters is an HTML document.

One of the principle goals of the HTML5 effort was to specify the language in such a way that every sequence of characters has exactly one interpretation as an HTML document. In other words, the goal is complete interoperability at the level of constructing a document tree. There is no formal, grammatical definition of HTML; it’s defined in prose in the specification and by running code and a test suite.

In principle, HTML leaves no room for extensibility. Every sequence of characters already has an interpretation. In practice, it provides several mechanisms: class attributes, “data” attributes (that is to say, attributes with names that begin data-), meta elements, rel attributes on link elements, script and embed elements, and Javascript.

HTML can obviously be used in a great many contexts, but for the overwhelming majority of users, the web browser is the only context that matters. On the modern web, Javascript and CSS are effectively required components of that context.

The current HTML maintainers have also asserted that no standard HTML element name will ever contain a hyphen. Various extensibility mechanisms have been described that rely on this fact. Cleverly packaging a hyphenated tag name (such as db-funcsynopsis) with appropriate Javascript and CSS offers the potential for considerable extensibility in the future.

Unlike namespaces in XML, the hyphenated names described by HTML provide no mechanism for establishing global uniqueness. This is obviously a problem in theory as your definition of ex-image and my version of ex-image are likely to collide in Very Bad™ ways if we attempt to use them together in the same document. Whether this turns out to be a significant problem in practice remains to be seen.

Markdown / CommonMark

Markdown was explicitly designed to enable authors “to write using an easy-to-read, easy-to-write, plain text format” that could be converted to structurally meaningful HTML. It was somewhat casually specified on John Gruber’s blog.

There are a lot of flavors of Markdown. CommonMark is a recent effort to write a more formal specification for Markdown. That’s the flavor used in this example.

Because transformation to HTML is explicitly the design goal of Markdown, there is a straightfoward extension mechanism: just insert HTML markup. We see that in Figure 12 for footnotes, description lists, tables, and figures.

# Document Title
<!-- There are proposed extensions for document metadata -->

This is a paragraph. See also [CommonMark](http://commonmark.org)

This is _another_ paragraph<sup><a id="fn1m" href="#fn1">1</a></sup>.

+ The first item of a bulleted list.
+ The second item.

  The second part of the second item.
+ The third item.

This is a paragraph.

1. The first item of a numbered list.
2. The second item.

   The second part of the second item.
3. The third item.

This paragraph contains a `code` word.

<!-- There are proposed extensions for DLs -->
<dl>
<dt>Mercury</dt>
<dd>Burn, baby, burn.</dd>
<dt>Venus</dt>
<dd>Where global warming ran amok.</dd>
<dt>Earth</dt>
<dd>Where global warming is running amok.</dd>
<dt>Mars</dt>
<dd>Future home of Elon Musk.</dd>
</dl>

This is a paragraph.

<table id="table">
<caption>A powerful table</caption>
<tr>
  <th>x</th>
  <th>x<sup>2</sup></th>
  <th>x<sup>3</sup></th>
</tr>
<tr>
  <td>1</td>
  <td>1</td>
  <td>1</td>
</tr>
<tr>
  <td>2</td>
  <td>4</td>
  <td>8</td>
</tr>
<tr>
  <td>3</td>
  <td>9</td>
  <td>27</td>
</tr>
</table>

<figure>
<figcaption>How long since then?</figcaption>

```xquery
xquery version "1.0-ml";
declare variable $startDate external;

let $date   := $startDate cast as xs:dateTime
let $diff   := current-dateTime() - $date
return
  current-dateTime() - $date
```

</figure>

Consider the function synopsis for the “max” function:

    #include <varargs.h>

    int max(int n, ...);

Finally, this is the *last* paragraph.

<sup><a id="fn1" href="#fn1m">1</a></sup> The examples in this document are awful, I know.

The focus of the CommonMark specification is the core language. Several extensions have been proposed: extensions for document metadata and a simpler, text syntax for description lists, for example.

The CommonMark example, converted to HTML with cmark is shown in Figure 13.

Figure 13: CommonMark rendered

Figure 14: CommonMark in Emacs

AsciiDoc

AsciiDoc was designed to be an alternate syntax for DocBook. It supports a fairly large, though not wholly complete, set of DocBook constructs. It can also be transformed into HTML and other XML vocabularies.

The common tools for transforming AsciiDoc into XML or HTML provide language-specific extension mechanisms. The AsciiDoc markup includes facilities for attribute annotations on blocks.

AsciiDoc also supports advanced features such as variable substitution (a mechanism for doing text substutition not unlike general entities in XML).

=  Document Title
:SUBTITLE: Subtitle
:AUTHOR: Jane Smith
:ABSTRACT: The document, abstractly.
:KEYWORDS: alpha, beta
:PUBLISHER: Yoyodyne Propulsion Systems
:JOHN: bigboote

This is a paragraph. See also link:http://asciidoctor.org/[Asciidoctor].

This is _another_ paragraphfootnoteref:[fn1,The examples in this document are awful, I know.].

* The first item of a bulleted list.
* The second item.
+
The second part of the second item.
* The third item.

This is a paragraph.

1. The first item of a numbered list.
2. The second item.
+
The second part of the second item.
3. The third item.

This paragraph contains a `code` word.

Mercury :: Burn, baby, burn.
Venus :: Where global warming ran amok.
Earth :: Where global warming is running amok.
Mars :: Future home of Elon Musk.

This is a paragraph.

[[table]]
.A powerful table
|======================
| x  | x^2^ | x^3^
| 1  | 1   | 1
| 2  | 4   | 8
| 3  | 9   | 27
|======================

.How long since then?
[source,xquery]
----
xquery version "1.0-ml";
declare variable $startDate external;

let $date   := $startDate cast as xs:dateTime
let $diff   := current-dateTime() - $date
return
  current-dateTime() - $date
----

Consider the function synopsis for the “max” function:

[source,c]
----
#include <varargs.h>

int max(int n, ...);
----

Finally, this is the *last* paragraph.

The AsciiDoc example, converted to HTML5 with asciidoctor is shown in Figure 16.

Figure 16: AsciiDoc rendered

Unlike the other conversions, the “default” conversion for AsciiDoc includes CSS to improve the appearance.

Figure 17: AsciiDoc in Emacs

Org Mode

Org Mode grew organically out of an earlier outline mode on Emacs. It boasts a very rich integration with Emacs including features like inline execution of code samples and a literate programming facility. Like AsciiDoc, it supports many different back ends.

#+TITLE: Document Title
:PROPERTIES:
:subtitle: Subtitle
:author: Jane Smith
:abstract: The document, abstractly.
:keywords: alpha, beta
:publisher: Yoyodyne Propulsion Systems
:john: bigboote
:END:

This is a paragraph. See also [[http://orgmode.org/][Org-Mode]].

This is /another/ paragraph[fn:1].

+ The first item of a bulleted list.
+ The second item.

  The second part of the second item.
+ The third item.

This is a paragraph.

1. The first item of a numbered list.
2. The second item.

   The second part of the second item.
3. The third item.

This paragraph contains a ~code~ word.

+ Mercury :: Burn, baby, burn.
+ Venus :: Where global warming ran amok.
+ Earth :: Where global warming is running amok.
+ Mars :: Future home of Elon Musk.

This is a paragraph.

#+CAPTION: A powerful table
#+NAME: table
|---+-----+-----|
| x | x^2 | x^3 |
|---+-----+-----|
| 1 |   1 |   1 |
| 2 |   4 |   8 |
| 3 |   9 |  27 |
|---+-----+-----|

#+CAPTION: How long since then?
#+BEGIN_EXAMPLE
#+BEGIN_SRC ml-xquery :var startDate="1967-06-16T18:24:00-07:00"
xquery version "1.0-ml";
declare variable $startDate external;

let $date   := $startDate cast as xs:dateTime
let $diff   := current-dateTime() - $date
return
  current-dateTime() - $date
#+END_SRC
#+END_EXAMPLE

Consider the function synopsis for the “max” function:

#+BEGIN_SRC c
#include <varargs.h>

int max(int n, ...);
#+END_SRC

Finally, this is the *last* paragraph.

[fn:1] The examples in this document are awful, I know.

The Org Mode example, converted to HTML within Emacs is shown in Figure 19.

Figure 19: Org Mode rendered

Figure 20: Org Mode in Emacs