How to cite this paper

Quin, Liam R. E. “Markup Formats In Context: A comparison of the strengths of some widely-used markup systems.” Presented at Balisage: The Markup Conference 2014, Washington, DC, August 5 - 8, 2014. In Proceedings of Balisage: The Markup Conference 2014. Balisage Series on Markup Technologies, vol. 13 (2014). https://doi.org/10.4242/BalisageVol13.Quin01.

Balisage: The Markup Conference 2014
August 5 - 8, 2014

Balisage Paper: Markup Formats In Context

A comparison of the strengths of some widely-used markup systems

Liam R. E. Quin

Copyright © Liam Quin 2014

Abstract

There are a number of popular text markup formats in use today. Some of these, such as JSON and Markdown, have risen in popularity recently; others, such as SGML or troff, have waned. Whenever a format becomes more popular it gains proponents who seem to want to see it used everywhere, for everything, for ever, right away. A fairly simple (and possibly over-simplistic) analysis of the rhetorical nature of some of these various formats is presented in this paper. The results of this analysis suggest areas of use for the different formats and demonstrate that, rather than being in competition with one another, the formats complement one another.

Table of Contents

Documents on Paper
Electronic Paper
When Robots Watch
Documents that Last
On Delivering XHTML or HTML
When Documents are not Documents
Programmer to Programmer, Machine to Machine: program-specific data formats
Programs and Humans: program-specific text formats
Factors for Evaluation
Information Life Cycle
Audience Language and Culture
Universal Access
Relationships between Documents
Default Formatting
Validation
Data Typing
Program Compatibility
Information Modelling
HTML and Web Browsers
Multiple Consumers: Transformations
Comparison of Formats
Plain Text (Unstructured)
Markdown
JSON
HTML
XHTML
RDF and Linked Data
XML
Some Use Cases
An Object Dump
A Technical Dictionary
Extended Journal Bibliography
Web-based Authoring Interface
Hybrid Approaches
RDF and JSON
RDF and XML
HTML and XML
Conclusions

Documents on Paper

Figure 1

Figure 1: person-to-person communication using physical transportation of pre-digital media objects.

When a human wishes to communicate extended ideas with another human not physically present, paper and a pencil can be used. Historically, this mechanism was extended using people trained to copy documents onto new, additional sheets of paper, but this was slow and expensive, and, after only a few thousand years, replaced by the automated printing press.

Paper documents are difficult to revise and cannot easily be searched.

Paper documents are independent of software and with care can be archived indefinitely.

Electronic Paper

Figure 2

Figure 2: person-to-person communication using electronic transportation of digital media objects.

Today if someone wants to write something for the benefit of another reader, they can use a word processor and either send computer-printed paper or send electronic files that the recipient can print and read. Those files are sent sometimes as plain text (electronic mail), or for longer documents or documents with more complex formatting, as PDF page images or as word processing files.

Word processing files represent a document complete with formatting but in an editable form, so that text can re-flow as needed. Such files generally make use of system resources such as fonts, so that a document may be differently paginated or, in the case of specialty symbol or language-specific fonts, may be partly or entirely unreadable. For some languages (Northern Cree and some of the scripts or writing systems used in India and in Africa come to mind) it’s customary to write documents using a font with a custom encoding, as Unicode coverage is (or is perceived to be) incomplete or insufficient; this means that if the recipient does not have the right font installed, the document may appear correct but will have some characters in the document silently substituted for others. To be fair this problem exists for all of the document formats discussed in this paper, but some of the formats alleviate the difficulties, or at least let documents be explicit about what was done, more than others.

Word processor documents today use complex and proprietary formats (although increasingly these are represented in XML). This means that they can be difficult to search, although they are usually easy to revise. Later versions of a word processor may interpret older files differently, with or without warning, so that the documents become tied to specific versions of software running in specific operating environments. Because word processor formats are (implicitly or explicitly) tied to specific versions of specific software, as well as to system resources such as fonts, they are not suitable for archival use.

Portable Document Format (PDF), a document format produced and maintained by Adobe Systems Inc. of California, USA, has a corresponding ISO archival standard, although in practice PDF documents can and do make use of extensions that are not archival. However, PDF files do contain all needed resources such as fonts and images, and are in most cases considered to be of archival quality. Software that creates PDF may have options to create PDF/A, the archival variant.

PDF documents do not generally reflow text if printed or viewed on a differently sized device than that for which they were created. The page dimensions are part of a PDF document and cannot easily be altered; hyphenation has been performed, footnotes have been numbered and so on. PDF documents can be extremely difficult to read on smaller devices, as the user may need to scroll horizontally back and forth to read each line of text.

The formatting of both word processor documents and PDF files is explicit (except as noted already) and it is therefore possible for a search engine to process and index the text in them and then to display formatted results and previews. However, because the formats are not intended for this purpose, there are some difficulties. For example, PDF does not require that the creating software explicitly mark word boundaries, since each “glyph” can be positioned independently from any other. If word and phrase boundaries are not clearly marked, indexing has to use heuristics: sometimes one will run across search engine results in which characters have been joined between paragraph breaks, or where words have been incorrectly split, or where hyphenated words result in two smaller sub-words being indexed separately.

Documents as essentially pictures of documents, whether because of proprietary or poorly documented file formats, because of insufficient information, or because the document actually contains bitmap (raster) images of text rather than the actual text, can pose difficult or insurmountable problems not only to search engines but also to people who cannot easily read from the pictures, for example because the lines of text do not reflow (creating a need for difficult sideways scrolling) or because the user is relying on a text reader to speak the text out loud and the document does not actually contain any text. Any system for mediating communication between humans must be useable by all humans.

When Robots Watch

Figure 3

Figure 3: person-to-person communication designed also for automated observers and mediators.

When people share documents and also expect their documents to be processed by automatic robotic services such as search engine indexers they must use formats that can be read by an unknown audience. HTML can be a suitable format because it has well-defined behaviour: the robots know where paragraphs start and end, which markup breaks up words or phrases and which does not, and how relationships to other resources such as images or linked documents are represented.

Although HTML 5 has added new structural elements such as article, it is common today for Web sites to use div elements with CSS-based styling for such things; this can increase the difficulty of determining the intended formatting: the search engines can determine word and phrase breaks only by applying the CSS styles. With the increased use of JavaScript-based styling this becomes harder, but fortunately there are strong financial incentives for commercial producers of HTML to use clear markup as otherwise their Web sites do not appear in user’s search results.

HTML is a moving, changing format and is not necessarily safe for archival purposes. PDF can be used with mediation, but PDF documents are not necessarily sufficiently accessible; it is possible to create PDF documents that consist of scanned bitmap page images rather than text.

Documents that Last

Figure 4

Figure 4: person-to-person communication designed also for automated observers and mediators.

When people share documents and need them to be archived for several years or longer, a combination of formats may be best.

XML is a suitable basis for archival formatting because the syntax of XML is not evolving significantly (unlike HTML). Since there are no behavioural semantics within XML there is nothing to change: it is a framework for carrying meaning. However, precisely because XML does not have universal behavioural semantics a robot, or a future human, cannot necessarily determine word, phrase and paragraph boundaries, nor relationships to other resources, by inspection. HyTime Architectural Forms (for use with the older SGML standard document format) might have provided a way for robots to do this, but they have not been adopted for XML.

Since XML documents cannot reliably be presented to humans or to robots it is necessary to augment them, either with transformations or with alternate additional document formats.

Whenever information is provided in multiple formats there is a possibility of errors and contradictions between the various versions of the documents. Providing one or more automated transformations, using standardized and non-proprietary transformation languages such as XSLT or XQuery, and clearly marking the XML version of the document as authoritative, may be sufficient to minimize the impact of the lack of default formatting for arbitrary XML vocabularies.

Suitably augmented XML, then, is suitable for archiving, can be transmitted across networks, and can be formatted to reflow on different devices or pages. The cost of attaining this goal can be high: it is the cost of anticipating the needs of others (including later, older versions of ourselves) as opposed to the cost of reacting only to our own present-moment needs. To motivate the expenditure we must realize short-term benefits. The ability to produce documents in multiple formats is part of this; other benefits will be discussed later in this document.[1]

On Delivering XHTML or HTML

Many people have written to say that XHTML has no advantages over HTML, or even has disadvantages. However, those writers all seem to be writing from a perspective in which HTML is itself considered a good thing, and in which the primary purpose of creating a document is to display it in a Web browser.

When a single document is to be consumed by many processes within a single organization the ability to use XML tools on it can make XHTML very useful. In addition, ebook readers are currently using XHTML and XML rather than unrestricted HTML 5.

Even if XHTML documents are served on the Web as text/html and not as XML, the design of “polyglot” XHTML is such that the result is predictable, and yet the document can still be processed with XML tools. The value, then, is to the producer. Any value to the consumer is coincidental, but there is also no significant detriment.

When Documents are not Documents

Figure 3

Figure 5: Writing in the Clouds.

When information is divorced from any context and becomes a set of facts it can be tempting to switch to RDF, the underlying knowledge representation used in the Linked Data Initiative. That context, however, may still be needed over time, so in larger projects RDF is most commonly used when it is automatically generated and known to be context-free, or kept in named RDF graphs and regenerated as needed, for example to support repudiation of facts or restrictions on sharing.

RDF cannot in general be represented in document format except through visualizations of graphs, and thus is even harder to format in search results or accessibility tools than XML (although since RDF can be interchanged in XML there is clearly and necessarily some overlap).[2]

Programmer to Programmer, Machine to Machine: program-specific data formats

When a computer program needs to communicate complex information to another program different considerations apply from human-readable documents.

  • Whatever format is used must map directly to data structures used within the programs at both ends, as otherwise the primary goal of communication between programs will not be achieved.

  • Programmers often consider efficiency to be an important goal, as measured by number of lines of code for parsing, amount of memory consumed, amount of processing used, and amount of data transmitted for a given result. For this reason terse formats are often preferred.

  • Flexibility of representation is not a benefit when one is marshalling data, saving/restoring/transmitting objects, or exchanging application-specific data. Instead, a very specific format may be easier to parse.

  • Standardization is not usually considered important by developers except insofar as widely-deployed code libraries might reduce work. One therefore often sees one-off formats in use.

One widely-used program-to-program data format is JavaScript Object Notation (JSON). Although, as the name suggests, this was originally a serialized form of data structures such as are found in Web browsers, the popularity of the World Wide Web and the desire to create and devour Web browser data structures on Web servers has meant that most programming languages today have libraries or native support available for handling JSON.

Since object serializations are by nature tied to specific versions of specific programs, and since JSON is not in general self-labelling with regard to version or conformance, JSON cannot be said to be suitable for archiving. None the less the syntax is compact and familiar to programmers working with most of the widely-used languages today, languages whose design was influenced by the C programming language.

Another widely used format is the “comma-separated values” (CSV) file. There are dozens of different syntax variations and software that reads CSV files often has to ask users to identify particular aspects of the variant in use, showing that the format is not very suitable for interchange or archiving. Recent work at W3C in supplying metadata for CSV files may help in this area in the future.

Programs and Humans: program-specific text formats

A variant on machine-to-machine communication is the set of markup formats designed by programmers for use in specific programs but intended to be authored and edited by humans using text editors.

This list includes languages such as Markdown (used for formatting wiki entries and for describing programs on github), Microsoft-style “ini” files, but perhaps also TeX and troff macros.

Over the years there have been many such formats, and long experience suggests several difficulties with the use of such formats and several strengths.

  • Ease of parsing can be so great there may not even be an identifiable piece of code that’s a parser. This can be both a strength (rapid prototyping and development) and a drawback (higher cost of maintenance).

  • Ad-hoc formats tend not to have any explicit document format version indication and yet be specific to specific versions of the software for which they were written.

  • If there is only one interpreter for a language it’s common to find that undocumented features become used, hindering future attempts at a second implementation and frustrating attempts to interpret data in the absence of the software for which it was created.

  • Errors in a file created in an ad-hoc format might go undetected, and, without other implementations to compare, or without a concept of validation, can become difficult, expensive or even impossible to correct after the fact.

Ameliorating some of the concerns is the fact that many human/programmer text formats are widely implemented.

One such widely-implemented format, Markdown, is used in multiple programs. Markdown is a text-based format designed for use in Web forms such as Wiki pages, with a syntax such as using equals-signs to underline a heading. It has the advantage that the text looks similar to the result of formatting, although the markup for that same reason tends to be presentational and not aimed at representing information which can be re-purposed. Unfortunately, there are many incompatible variations of Markdown and the format is not self-labeling, so that one can't be certain which variation one is seeing.

A strength claimed for Markdown is that people unaccustomed to HTML or other markup languages can work with it. Direct content-editing in Web browsers removes much of that appeal, since a word-processor style of input editing is presumably even more appealing to the same people who don't like HTML. In fairness one should also mention programmers who want a text-based document format but feel that XML and HTML are too verbose for their needs.

Factors for Evaluation

This section describes some of the factors that determine which format to use in a given situation. There is no complete list because situational and contextual factors are always the most significant in practice. Note that evaluation here is not in the sense of deciding one format to be in some way superior to another, but to suggest applications for which each is the most suited.

Information Life Cycle

Information that will be archived for future research purposes must be clear when taken out of context. This might be achieved through careful documentation and avoiding relying on application-specific or opaque formats.

Information that will be used once and discarded, such as an API message in a Web service or notification that a user moved a pointing device could reasonably be in an application-specific format, but if multiple programs might make use of the same message then there is greater value in a more generic format.

Information that will be stored and processed and perhaps queried will need to be in a format that supports that processing. This is the most common case for documents today and the least common for data (since the data is more easily queried in a data store than interchanged en masse).

Self-describing or clearly documented information will generally make querying easier and will facilitate recovery from an archive in the future, but that follows for all possible data formats. However, not all data formats are such that documents can easily, and routinely do, identify the format used and version of that format. For example, neither CSV files nor Markdown documents can in any standard manner identify the specification or language to which they might conform, and HTML 5 documents do not identity the dated version of the "living standard" to which they conform.

Audience Language and Culture

Information that contains mixed languages, scripts or dialects will need a mechanism to indicate this, such as xml:lang in XML or lang in HTML.

Where human-readable content is included and could be in any language (now or in the future), rich text (mixed content) will almost certainly be needed, at a minimum for supporting Japanese or Chinese ruby annotations.

Where text may be translated, in part or whole, a text replacement mechanism may be needed to make a translated version of a document. It may also be necessary to mark which parts are to be left untranslated (push the button labeled sokken: the label on the physical vending machine on the platform doesn't change just because you have an English guide book).

Universal Access

Any information presented to people will need to be accessible to them. This means that accessibility must be built in at all levels. Some of the formats described in this paper are accessibility-agnostic, but others can include or encourage user interface elements that can be harmful or exclusionary; in such cases extra vigilance may be needed on the part of document authors and system developers.

Relationships between Documents

Sometimes a document or piece of data might stand alone, but that surely is rare. A document might form part of a sequence, might contain links, might be contained in, or be a database, so that joins between sets of values might be performed.

Link discovery requires a standard vocabulary such as HTML or XLink or a standard discovery mechanism such as HyTime's architectural forms for SGML years earlier.

Implicit links, such as might be found by joins, are thus format-dependent; a dictionary site might make a link out of every word or phrase in a paragraph to a corresponding definition, but might do so programmatically (often with poor results in the face of homonyms). This ability is independent of format, but explicit linking requires syntax as does marking terms not intended to participate in such links.

Although simple querying can be performed on any of the formats, since they are text based, structure-aware querying is currently defined only for some formats, including RDF (SPARQL), XML (XQuery) and (although not a standard) JSON (JSONIQ).

Structure-based querying often has difficulty when one syntax is embedded within another: which HTML documents contain a definition for a particular JavaScript function with a given type signature, or which JSON documents contain a string with embedded HTML having a div element with a particular class attribute. Such hybrid queries can involve complex textual escaping conventions; XQuery systems supporting SPARQL queries of RDF embedded in XML provide a promising counterexample.

Default Formatting

Documents on the open Web need to be findable, and that generally means that search engines will need to parse them and then in response to user searches generate result snippets, short extracts that users can use to decide whether to read the longer document. Phrase and word breaks and basic formatting is necessary for the snippets.

Default formatting is also needed for operations such as copy and paste.

Validation

Although validation is a dirty word in some HTML circles, in other circles it's an essential part of doing business: context determines function.

Validation can be at the syntax checking level, or at the business logic level (every invoice must have a date, a customer number and an amount), or can be at the application level (the file is OK if the program reads it). Of these, the application level validation is the most powerful (arbitrary code) and the least portable. A standard way to express business or grammar rules means that documents can be tested against multiple programs and can also serve as documentation over time.

Data Typing

A document may contain components with identifiable data types such as "numbers" or "sequence of characters, string" or "truth value". This is essential for data binding and object dumping (as in JSON), but for some other systems it's also important to support user-defined types such as sock-colour or MailingAddress.

Program Compatibility

The constructs that a data format can represent should match the objects that a program needs in order to manipulate that data. If a format is too difficult to process it will not be popular with developers.

This must be balanced by the fact that programmers may not be the only, or even the most critical, stakeholders in a project.

In some cases (and some contexts) a compromise can be reached using scripting languages such as JavaScript, but then security implications must be considered.

The need to process data is intrinsic to computing with data; having standard data processing and transformation languages can help with staffing needs as well as system portability and longevity at the expense of using languages that are not necessarily optimized for the particular task at hand.

Information Modelling

One of the decisive factors for many projects in the past has been whether the goal of using markup is to model information (which may exist outside of the marked-up document, for example in a physical book or manuscript being transcribed or quoted, or an existing business process) or whether it is to guide presentation.

Markup as part of information modelling can be contrasted with markup as a syntax for conveying data, such as node-and-arc graphs or objects, which themselves may represent (or be) models.

HTML and Web Browsers

The markup in HTML is primarily driven today by the goals of Web browser vendors.

Although lip-service is paid to so-called “semantic tagging” what is meant is markup divorced from presentation specifics and yet tailored to a specific type of software application, the Web browser. An HTML document represents part of a Web Application, together with other resources such as Cascading Style Sheets (CSS), images, JavaScript programs, and perhaps input data in JSON or other formats.

So-called semantic tags (actually elements) added to HTML 5 have mostly included markup for blogging. Transcribing a play, writing a poem, even sharing song lyrics, these are not on the HTML agenda.[3]

Recent work on user-defined elements in HTML concentrates on their “behaviour” rather than on what (if anything) is being represented.

Since Cascading Style Sheets have built-in support for HTML features rather than being a general-purpose styling language for marked-up documents it is more convenient to use HTML rather than some other XML markup language when using CSS, whether for Web browser use or otherwise.

Since the HTML language is intended for use with CSS and JavaScript, primarily within a Web browser, and not for document modeling, it makes sense to use XML for authoring, transcriptions, and archival purposes, and to transform to HTML when needed.

Multiple Consumers: Transformations

The need for document creators to produce EPUB documents for electronic readers alongside other formats has led to an increase in the usage of XML, as opposed to (or as well as) proprietary page design or word processing formats. There is nothing about XML that makes it inherently more amenable to transformation than JSON, or than any format that can be parsed reliably and in an interoperable manner. In practice, however, the existence of XSLT, of XQuery and XPath, and the widespread availability of tools implementing those languages, means that XML is a particularly convenient choice. The use of XML schema languages to check that documents meet specified constraints can also help to control the scope of transformation programs.

It should be noted that a strength of XSLT is that it can be written, read and maintained by people who do not see themselves as programmers, but as document people. The declarative nature of XSLT, and the limited control flow possibilities, help to make the XSLT transformations easy to understand. As a result, organizations with people working on predominantly textual documents are very likely to have staff who can comfortably use XSLT, making XML in turn an excellent choice as a basis for transformations.

HTML and JSON, by contrast, do not have such transformation languages; JavaScript is much closer to “regular programming” than XSLT and may be seen as inappropriate for technical writers to use.

Comparison of Formats

So far this paper has introduced some use cases and (indirectly) markup formats. This section summarizes the strengths and weaknesses of each format using the factors for evaluation described above, after a brief introduction to make clear what is meant in this paper by each format.

it should be stressed that this is not a complete list of markup formats; the goal of this paper is to help the reader choose among several of the most likely formats to be used today, and to provide a starting-point for discussion.

Plain Text (Unstructured)

Mentioned here only for completeness, plain text files with no claim to using any particular markup strategy can be read by humans and if there is some regular ad-hoc syntax then a program can read the file, but there is no Network Effect: if the syntax were widely enough used to have multiple implementations and a user community it would no longer be considered a plain text file, but would have identifiable structure.

Since plain unstructured text does not by itself constitute a markup language, it will not be compared further.

Markdown

Although there are a number of mostly-compatible variants of Markdown, in this paper we will imagine a world in which a single variant dominates. The stated intent of Markdown is as a text to HTML conversion tool for Web writers.

Life Cycle: because Markdown is not a standard, variations between versions may mean Markdown is not ideal for archiving. This is exacerbated because Markdown files are not self-describing: they do not label themselves as Markdown and do not identify the version of Markdown to which they conform.

Audience, Language and Culture: Markdown is not internationalized. Lack of support of mixed language paragraphs, indications of language in use, explicit right-to-left markup, Ruby annotations and script selection may make it unsuitable for mixed language content. Lack of named identifiers for sections and paragraphs may make it difficult to keep translations in sync.

Universal Access: Markdown has limited support for HTML accessibility from a reader perspective; on the other hand Markdown has found a use for people writing blogs, because it can fairly easily be created in a text editor and uploaded, avoiding the user interface for the blogging system.

Situations: Markdown is suitable for simple computer-mediated human-to-human communication, since Markdown files can easily be read in their text form as well as when converted to HTML. Markdown cannot represent complex documents such as mathematical research papers.

Relationships: Markdown supports explicit URL-based links.

Default formatting: Markdown files can be seen as text files or as HTML, and it is reasonable to say that, although not as powerful r widely supported as HTML in this regard, Markdown documents are transparent with respect to the author's formatting intentions.

Data Typing and Validation: not provided except for basic syntax checking.

Program Compatibility: Markdown is not significantly easier to process in programs than HTML, and a common way to process it is in fact to convert it to HTML first.

Use case: Markdown is primarily used where a text-based "rich text" is needed for people uncomfortable dealing with HTML or XML directly, and where no tools are available.

Information Modelling: not attempted.

JSON

JSON (JavaScript Object Notation) is a mechanism for transmitting data that can easily be instantiated as programming-language-level objects by the receiver. The format was originally defined for JavaScript but JSON is now supported by most of the major programming languages. JSON is included in this paper because, even though it is not perhaps a markup language, and does not attempt to be particularly suited for textual documents, it is widely seen as a replacement for XML in Web services and interactive Web usage (AJAX), where JSON strings contain escaped fragments of HTML.

Situations: JSON is intended for program-to-program communication.

Life Cycle: JSON is primarily aimed at information that will be used once and discarded, such as search results communicated from a Web server to a Web browser. However, today there are databases for storing and querying "JSON documents".

Audience, Language and Culture: JSON documents do not have standard ways (at the time of writing) to mark the natural language used for text strings; even if it did, JavaScript objects are the wrong level of abstraction for this. It is, however, possible to embed escaped HTML string in JSON, and this can contain language tags. JSON is not intended as an authoring format for textual documents.

Universal Access: since JSON is intended for program-to-program communication this is not an issue. It is up to the creator of any HTML embedded inside JSON to ensure accessibility, however.

Relations between Documents: JSON documents represent objects with simple names; if it's known through some external source that the same name in multiple documents represents the same information then database query languages can associate the information. Additionally, JSON strings might include escaped HTML markup with links, but there is no meaningful way to point into a JSON file with a link, nor is there a standard meaning. JSON Schema defines a mechanism to point to JSON objects using a reserved name, "id".

The JSONIQ query language gives an extended XPath-like syntax, and there are other ways to refer to the inside of a JSON document, but pointing into an object in a computer program isn't the same as linking to part of a document.

There are no widely used ways to transform JSON objects outside of a programming language, although there is (or will be) JSON support in XQuery 3.1, XSLT 3 and JSONIQ.

Default Formatting: There is no default presentation for JSON objects beyond the "source code view" of the actual document.

Validation and Data Typing: The IETF JSON Schema language is still a draft, and does not have large traction yet, but is gaining maturity. It was influenced by XML Schema but does not support user-defined data types. it is intended for use at a programmer and API level, not at a business level.

Program Compatibility: This is the greatest strength of JSON: JSON documents are also JavaScript fragments. They can be embedded in the source code of programs, they can be read with "eval" (although security implications suggest this should be preceded with validation) and they can be generated directly from any object in a JavaScript program. Although usage in other programming languages typically requires a library, JSON's data structures usually map exactly onto data structures in popular programming languages, unlike (for example) HTML or XML, where attributes and mixed content must be modeled in terms of such data structures.

Information Modelling: JSON is all about program modelling and not information modelling. It's just syntax: one can map from SGML or HTML or XML into JSON, but the primary strength of JSON is its convenience for developers, not its easy (or otherwise) at modeling information. Another indication of the JSON culture is that JSON Schema does not provide for user-defined types, just number, string, boolean, array, object and null. Schema authors can restrict the value space to say that a field called socks_owned must be a whole number not less than zero, but cannot say that socks_owned is of type socks_count; this reflects the type system of JavaScript but is not for example a good match for the way people think about documents or objects outside the computer.

HTML

The HyperText Markup Language, standardized first at the IETF and the ISO and later at W3C, is a fixed markup language aimed at delivering documents to the World Wide Web. It is a vocabulary largely controlled by Web browser makers.

A recent variant, HTML 5, adds support for "Web Components", essentially user-defined HTML elements with content templates and JavaScript and CSS styles to supply any required browser-side behaviour. Unfortunately, HTML 5 is a "living standard" and features come and go from time to time. This is balanced by excellent support from Web browsers and clear documentation (in almost all cases) on exactly how a Web browser should recover from errors.

Situations: HTML is primarily intended for computer-mediated human to human communication of documents, but it is also increasingly used today for computer-to-human interactions with "Web Applications."

HTML is also used for computer-to-computer messages, but in this case the error recovery rules employed by Web browsers and by conforming HTML 5 implementations may not always be appropriate. Silent correction or acceptance of errors has in other languages and systems famously led to deaths in space missions and other engineering problems.

Information Life Cycle: HTML is implemented in perhaps a dozen or more Web browsers, with a very large deployment. As a result it is difficult for HTML to change in incompatible ways. None the less attempts to change HTML in that way are often attempted, and, as a result, archived HTML documents need to be explicit about the version of HTML they used.

The culture of HTML tends to be very much aimed at Web browser use. As such, behavioural and presentation semantics are emphasized, with "semantic" elements such as section and article being hailed as an advance over equally generic names such as div. Again, the challenge here for archiving is that the actual meanings of markup constructs will and do change over time, and also that JavaScript code may or may not continue working over a period of decades and may or may not sufficiently describe behaviour and intent.

A large number of content management systems and databases for storing HTML exist; some of them prefer XHTML, which can be parsed more reliably; see the next section for more details.

Audience, Language and Culture: HTML has strong internationalization and localization support, especially when used in conjunction with the Internationalization Tag Set (ITS). Individual elements down to the word or sub-word level can be marked for language, region and script, and can be marked as not to be translated. Ongoing work, for example in supporting all forms of Chinese and Japanese ruby annotations, is improving the situation still further, but, overall, HTML offers one of the best formats for international and multilingual documents today.

Early versions of HTML, unfortunately, put human-readable content such as alternate replacement text for when an image is not available, in attributes, precluding markup for mathematics, for Ruby annotations, for emphasis; this defect is slowly being corrected, for example with the picture element.

Universal Access: Extensive and very helpful information is available for document and application authors working with HTML. There are plenty of challenges since not all HTML documents are automatically accessible, but that is also true of other rich formats, especially when they are scriptable. A complex system of fallbacks makes it possible to write Web applications that will work on a wide range of devices and with assistive technologies such as text readers, alternate pointing devices and even Braille terminals.

Relationships between Documents: HTML has a rich vocabulary for representing relationships from one document to another, including explicit hypertext links and link relations as well as implicit links (for example with URI Templates) and links between information and remote descriptions with microdata and RDFa annotations.

There is no automated mechanism today for link discovery when links are implicit.

There is no widely-deployed standard HTML querying language, and there is no standard way in HTML to represent relationships between documents outside of any document.

Default Formatting: HTML today is used for the representation and formatting of best-selling printed books; it is not as sophisticated as other publishing platforms but it growing rapidly in that area. HTML documents have default associated formatting, although an increase in the use of cascading style sheets to redefine the formatting and purpose of elements can weaken that, and should be avoided.

Validation: There are widely-used syntax checkers for HTML, such as that at validator.nu and the W3C HTML validator. Validation at the business level, for example to say a heading must be followed by a paragraph, must be handled with other mechanisms, such as by using XHTML and XML Schema.

Data Typing: HTML did not define any specific data model until HTML 5; before that, although the HTML DOM was widely used, it was not mandated by HTML. Like JavaScript objects, however, the HTML DOM is not strongly typed.

Program Compatibility: Unlike JSON, HTML documents cannot easily be processed by programs in most traditional languages, even JavaScript. Attempts to alleviate this, such as the popular jQuery library, have been largely successful where they are available. HTML is not a strong choice for object serialization and deserialization, which is why JSON exists.

Information Modelling: HTML documents are closely (and increasingly) tied to Web browser design. HTML is adequate in many cases for modelling a blog, although it does not have standard support for song lyrics, poems, footnotes, or a host of other basic rhetorical forms and devices.

XHTML

There are two main versions of XHTML in use today, and two meanings of the term; XHTML 1 was designed to be an XML-based version of HTML 4 which can be served to Web browsers as either XML or HTML. XHTML 5 is an XML serialization of HTML 5 with the same goal: that when a Web browser reads an XHTML 5 document it creates the same internal representation (DOM) regardless of whether the HTML or the XML syntax was used. XHTML 5 is not, however, a successor to earlier versions of XHTML.

All of the considerations for HTML apply to the XML syntax for HTML, except that parsing of XHTML as XML means firstly that errors may be treated as fatal and second that XML tools can be used with XHTML documents.

RDF and Linked Data

The Resource Description Framework, RDF, is a standard for representing metadata as sets of decontextualized triples of atomic values that form a (possibly disconnected) graph. RDF is most often exchanged in three formats: RDF/XML; Turtle (a text-based syntax); and SPARQL Results in XML, a format intended to be transformed (often with XSLT) into a user-visible format such as HTML or SVG.

Linked Data (LD) is a name for the practice of publishing and combining RDF-based graphs; it is mentioned here in the context of making abstract RDF graphs available from documents.

Situation: RDF is primarily used in computer-to-computer communication, although many RDF data sets are hand-authored.

Information Life Cycle: RDF documents are frequently stored in databases, whether hybrid or RDF-only (RDF-only databases are often called triple stores). Although RDF can be used for one-off communication it is more often stored and queried. RDF is also commonly embedded in other formats, especially HTML. The most common standard querying language for RDF is SPARQL.

Since RDF uses URIs, and URIs are defined to be opaque and meaningless to an outside observer, RDF is strictly speaking not self describing. In practice, though, URIs are normally made from natural language words and represent what those words name. Most RDF serializations do identify the file as conforming to a specific version of RDF.

Audience, Language and Culture: RDF nodes have opaque identifiers that are not in any natural language. it is possible to create "labelFor" nodes in the RDF graph and give them language tags, although it should be noted that RDF does not handle XML or HTML style mixed content well.

The Linked Data culture wants all information about everything and everyone to be public. Privacy and security remain challenges for the various RDF communities. A talk at XML Prague suggested storing RDF graphs in XML databases and using XQuery to construct a set of triples for SPARQL queries based on security, but this should probably be seen as an outlier; in the long term one can expect SPARQL itself to learn about security. A technical challenge is that there is nowhere in a triple to store sharing or security information.

Universal Access: RDF, like JSON, does not have any inherent user interaction. Graphical visualizations, however, can be a challenge for people who are not able to see them, and alternatives therefore need to be considered.

Relationships between Documents: RDF is all about relationships, but, oddly, cannot easily refer from one graph to another. RDF named graphs (new in RDF 1.1) may provide a mechanism there, but it is too soon to measure deployment.

Default Formatting: RDF documents do not have textual representations other than (like JSON) as source. However, they are conventionally represented as node and arc graphs. This visual representation conveys the overall structure of an RDF graph but not necessarily the actual content.

Validation: There has been recent work on RDF Shape Expressions for constraining the shape of RDF graphs; this is not yet deployed.

Data Typing: RDF does support associating data types with values, and these can be user defined.

Program Compatibility: The RDF model is graph based, not object based, and does not correspond to the native data structures and type systems of modern programming languages. However, those same current languages are easily able to represent RDF graphs, and there is no mixed content to complicate things.

Information Modelling: RDF is about modelling knowledge, not information. it is a knowledge representation system used primarily for first-order logic and inferencing.

XML

The Extensible Markup Language, defined at W3C as a subset or profile of SGML (and originally known as Web SGML), is not really a single markup language like HTML, but instead a framework for defining one's own markup languages, all of which have a common syntax.

This paper distinguishes where appropriate between arbitrary XML documents and documents in some specific XML-based markup language such as XHTML 5 or DocBook.

Situation: XML is used in all areas of communication: person to person, person to computer, and computer to computer, and can to some extent also be used without computer mediation (that is, text-oriented XML documents can be moderately readable, although not as much as Markdown documents).

Information Life Cycle: XML documents have a life cycle that depends on how they are used more than on the fact they are XML. For example, a message from an automobile engine to a garage mechanic's diagnostic system, a message from one operating system component to another when a user double-clicks on a desktop icon, a transcription of an Anglo-Saxon poem, a health-care provider's record of treatment for a patient, all are likely to be in XML, and each have different longevity and processing characteristics.

Trees based on parsing XML documents can be stored in relational, XML-native or hybrid data stores, and the XQuery language can be used to access them efficiently.

Audience Language and Culture: XML documents can support all of the internationalization features of HTML and XHTML, but it depends on the specific XML vocabulary. If you are designing an XML representation for text you should consider adopting the HTML model where possible because of widespread understanding and adoption.

The W3C Internationalization Tag Set (ITS) can be used directly in XML to help with translation and localization.

Universal Access: Again, this depends on the ways in which the XML documents are used. Awareness of the W3C Web Content Accessibility Guidelines can help document designers to create accessible systems using XML.

Relationships between Documents: The XLink specification has not gained much traction, and today people are more likely to use an ad-hoc attribute called href, or possibly to use the HTML "a" element by means of an XML namespace. it is also possible to embed RDF in XML documents.

Default Formatting: This is one of the two biggest weaknesses of XML: since there are no default presentational semantics, search engines cannot generate reliable snippets for results. Using XML on the World Wide Web can therefore be a problem.

Validation: XML has a wide variety of validation mechanisms, from simple and widely-supported DTDs, through to the baroque complexities of W3C XML Schema. A part-way compromise is RELAXNG, but this does not perform the data binding role of XML Schema, as described in the next paragraph. User-defined data types and compound types are available.

Data Typing: XML Schema validation can assign type annotations to elements in the parsed XML tree; type labels can be user-defined type names as well as built-in types. Note that RELAXNG does not support assignment of type annotations in a deterministic way, so that XML Schema is generally used where data binding (object loading and dumping) is required.

Program Compatibility: This is the second of the two main weaknesses of XML: the concept of an annotated tree of nodes is not a native data structure in most programming languages. As with HTML, mixed content such as paragraphs with embedded elements considerably complicates processing.

The situation is mitigated by the popularity of XSLT and XQuery, XML-specific languages for querying and manipulating trees.

Information Modelling: This is the greatest strength of XML: that it can be used, and culturally is used, to model documents or other information outside of any particular application or process. This strength comes at a cost: because XML documents are usually independent of any one program they are also not optimized for processing by any one program, and this can make XML unpopular with application developers.

Some Use Cases

This section gives examples chosen to illustrate a typical use case for each of the main formats discussed, together with indication of how to represent the example in the other formats.

An Object Dump

Consider a JavaScript program running in node.js on a Web server, communicating with a database to provide persistent storage of objects. Objects will have JavaScript types and values; the obvious choice is JSON, which was designed for this purpose.

One could use RDF instead; direct mappings from UML to RDF exist. But then a library would be needed, and the various transfer syntaxes of RDF are not as convenient for JavaScript programmers. In languages where JSON also needs a library, or where JSON does not map well to objects, RDF may be a stronger contender.

XML is also commonly used for object dumps. A library is needed, both for serialization and for loading, but such libraries exist for most languages. Since object dumps tend to be specific to a particular state of a particular program at a particular time, they are not easily reused by other programs; JSON may be more suited in that case. The strongest use cases for XML are when documents will be used in multiple ways.

The lack of standard transformation tools for JSON (compared to XML for example) is likely to be short-lived; there are several contenders as well as native-JSON NoSQL databases in widespread use.

A Technical Dictionary

In this example an organization edits a complex dictionary and produces editions in print, in PDF, in HTML on a subscriber-only Web site and in EPUB for ebook readers. Subsidiary products are also produced and might include a dictionary defining only terms needed for specific high-school (K12) or undergraduate courses, or subsets containing, say, only entries that mention a specific compound.

Dictionaries are examples of documents that often feature mixed content very heavily: superscript and subscripts, mathematics, terms that are to link to definitions, multiple languages, symbols and small diagrams may all occur in running text. Even a simple English dictionary may contain relatively mixed content, as in the example in Figure 6

Figure 6

Figure 6: A definition from a 1730s dictionary showing mixed English and Greek used in an etymology.

Since EPUB 3 used for electronic books is essentially a "Web site on a stick" there is considerable pressure to use HTML. However, custom markup can support business-level validation (for example, every major definition must have at least three examples, and can help with research and querying.

A compromise is to use (X)HTML augmented using ARIA attributes to provide so-called so-called structural semantics, with microdata, or even with custom XML elements; since HTML 5 Web Components provide a standard way to add elements this approach is likely to become popular. However, enforcing appropriate markup on authors may be necessary to preserve the value of the work, and that may suggest a custom XML-based markup with transformations to HTML as needed. Multilingual mixed content is today the home turf of the XML team.

RDF metadata can be embedded in dictionary entries, or, more likely, generated on the fly, perhaps with XQuery or XSLT, from the higher-level XML notations that are more convenient for authors to work with. Representing mixed content in RDF would typically involve explicit and tedious representation of sequences of anonymous nodes.

Markdown quickly runs out of power to express complex texts, whether multilingual like the English dictionary or containing chemical formulae and mathematics as in the technical dictionary. Variants that are sufficiently powerful start to stretch what is feasible with ad-hoc text-based syntax and the extra difficulty of using HTML or XML for the simpler parts probably pays off with consistent markup for the harder parts.

Extended Journal Bibliography

In this example entries for different authors are to be connected; any text formatting is minimal and formulaic. RDF is a strong candidate here. JSON could also be used.

A common need with bibliographical data is powerful full text searching, including similarity, starts-with, lexical containment, proximity within a field or element, and more. The XPath and XQuery Full Text extension was created with this in mind, suggesting that in some environments an XML-compatible representation may be worth investigating. Note that XQuery and XSLT 2 and later are defined to operate on trees which, although commonly created from XML, could come from any source that meets the necessary constraints.

Although Markdown again is not a likely choice, it should be noted that the text-based format pioneered by Mike Lesk for the refer program in the 1970s, and later taken up by BibTeX, is widely used and widely supported in technical and academic communities.

Web-based Authoring Interface

This example considers a Wiki-like situation, with a large and diverse group of authors for most of whom interaction with the Web site is not a major part of their lives, so that they will have little interest in learning about “syntax.”

This is a typical use case for Markdown today. The Markdown markup is embedded in an HTML form, and the user interacts with the Web browser's built-in text editor.

More recently, the content-editable property of HTML elements can be used to support word-processing style editing of parts of documents in place, which may reduce the desire to use Markdown.

Hybrid Approaches

Just as it would be wrong to suggest that the various formats all compete in the same space, so it would be wrong to insist that they stand alone. Some obvious combinations are given in this section, but it is necessarily not an exhaustive list.

RDF and JSON

People are already exchanging linked data using JSON instead of XML or N3 to transmit RDF graphs. This is to be expected since RDF is primarily a format for machine-to-machine communication and programmers like the strong match between JSON and internal data structures.

There are a number of competing formats, including JSON-LD, RDF/JSON, JSN3, JROn and more, although JSON-LD may at the time of writing be winning out.

RDF and XML

There are three main approaches to adding RDF to XML: storing RDF triples explicitly within XML documents alongside other XML information; storing RDF separately from XML, perhaps in a triple store; generating RDF from XML documents. Each has its place as circumstances dictate,and combinations of these methods are also in use.

Converting from RDF to XML (other than serializing RDF as RDF/XML or some other XML representation of RDF graphs) is not useful in general, but the results of querying an RDF graph with SPARQL are often processed with XML tools such as XSLT or XQuery for presentation in human-readable form.

Visualizations of RDF graphs as SVG and also using the XML-based GraphML should also be mentioned here.

HTML and XML

Mixing two document formats, rather than a data format and a document format, rarely seems to be productive. The combination of HTML and XML is HTML represented in XML (XHTML). Another combination is found commonly in RSS feeds and Atom, and is escaped HTML inside XML. This is done because HTML (not XHTML) has different syntax rules that conflict with XML, so that one cannot simply embed HTML inside XML.

Conclusions

It is not possible to give universal recommendations for when to use a particular format because many unforeseeable considerations may apply. For example, local knowledge of particular programming languages or ways of working may dictate consideration of a subset of the formats, or may even mandate the use of a particular format regardless of suitability to task.

The formats discussed here do not compete with one another. They complement one another, and are often used in conjunction with each other.



[1] This is not to say that the benefits are not discussed in this document but will be so discussed in some future version of this document, but rather, that a reader proceeding in a linear fashion forwards through this document, as published, will encounter the discussion of those benefits later than encountering the reference to this footnote. Those readers reading the document backwards will already understand why this is an important point.

[2] It should be noted that at least one mapping exists from the XML Information Set into RDF, so that in theory at least one can represent any XML document as RDF; in practice the mapping creates a separate RDF graph node for every text character in the document, and is unlikely to have much practical application. However, such RDF graphs could, at least in principle, be returned to their XML form and hence at least some of them could be represented as documents.

[3] See. however, http://www.w3.org/html/wg/wiki/PoeticSemantics for a discussion of ways to add poetry to HTML; a discussion that appears not to have gained much traction.