Balisage Paper: PreTeXt

Content versus Presentation

Every attempt is made to rigorously separate content from presentation. This can be a hard argument to make to authors from a WYSIWYG background, and can even be hard to make for authors experienced with LaTeX, since they think LaTeX already does this (it could, but there are few facilities to enforce it). The obvious payoff is the ability to produce many (dissimilar) output formats from one source, with no editing. And it has been a long time since anybody asked for a bold tag.

Multiple Output Formats

PreTeXt has excellent support, via XSLT stylesheets, for conversion to the following output formats based on other markup languages: online HTML, PDF for viewing on a screen, and PDF for print-on-demand physical versions. There are good, but not complete, conversions to EPUB and Jupyter notebooks. The EPUB conversion renders well in iBooks, and work is in-progress to convert further to Kindle format. Jupyter notebooks are online computational notebooks [Jupyter], allowing editing in a web browser with embedded code executed by connections to modular computational kernels. Their underlying format is JSON [JSON] holding a mix of Markdown [Markdown] and HTML.

Cross-Referencing

The ID/IDREF system provides the basis for a robust and unified system for cross-referencing. Many elements in PreTeXt can be the target of a cross-reference, and the cross-reference produced can take many forms (display number, display number plus type-prefix, title, etc.). By contrast LaTeX uses \label to mark objects, and then \ref produces display numbers for cross-references, but there is also \pageref to generate page numbers, \cite for citations, and \eqref for equations. Further, certain natural parts of a document are not easy to cross-reference in LaTeX, such as a proof, or a case within a proof, since they do not earn a display number. In HTML output a cross-reference can be implemented with a knowl [Knowl], which is a form of transclusion [Wikipedia]. So when a reader clicks on a cross-reference to definition, the page splits and a box opens containing the text of the definition. Clicking again on the knowl puts the box away. Readers rapidly come to expect a citation to open up with the bibliographic information, rather than spiriting them away to a division of the back matter. Rather than having page numbers in the index, a knowl works much better. Every knowl contains a traditional in-context link if the reader wishes to migrate away to another location.

Extensive and Flexible Numbering

A textbook or monograph is not a novel. Readers may sample, authors will cross-reference previous material, and a textbook may later serve as reference material. So items that need to be identified unambiguously are automatically assigned numbers (which are, of course, the same in each output format). There are six numbering schemes in use (divisions, blocks/environments, equations, exercises, citations, and footnotes) and the form of the number can be configured to follow the document hierarchy to a specified level. So the same theorem in the source could be Theorem 7.3.4 or Theorem 7.17 or Theorem 105, depending on an author's document-wide choice. Almost everything that is not unique (e.g. the Acknowledgments division) is automatically numbered, with no provision for turning it off, either document-wide nor one-off. Equations are the one notable exception, allowing numbers optionally.

Online Output (HTML)

Our HTML output is the most capable. We can embed interactive components such as video, and activities expecting more reader interaction through sliders, checkboxes and text fields. Anything that can live on a standard web page becomes a candidate for a component of a book, including current support for HTML5 canvas element with Javascript libraries. Extensive work by David Farmer and others has led to CSS designed to meet the needs of serious readers. Rich navigational devices, knowls (information hiding and cross-referencing), integrated search, and other features further aid the reader.

We aim to find the best mix of the old and new. Centuries of book design have given us a hierarchical organization, an optimal number of characters per line, and navigational devices like an index. Web browsers allow extensive hyper-linking, a copious margin for tangential material, and integrated search. We are careful to not throw out the baby with the bathwater.

Accessibility

The accessibility of online materials for students with physical limitations is a huge concern at many universities, and enhancing the opportunities for those with disabilities should be part of the promise of electronic formats. Because we do not presume electronic outputs are housed in proprietary systems, we can employ open standards and best practices wherever possible. For example, the MathJax library [MathJax] gives every mathematical expression a textual representation that a screen reader can read in sync with the expression tree. Presently, the HTML markup is being fine-tuned to validate completely and roles from the Accessible Rich Internet Applications [ARIA] standard will then be added.

Diagrams Authored in Source

Graphic images, created with other tools, often get disassociated from their final resting place in a finished document, and then reside only in uneditable formats. We encourage authors to use graphics languages, such as LaTeX's tikz, or Asymptote, to include the source code of graphics within their PreTeXt source. An XSLT stylesheet and a Python script, with other executables, then manage a conversion from source to final image format. SVG is the ideal for online output, since it scales uniformly along with surrounding text.

Documentation

As mentioned earlier, we have an Author's Guide, which explains the elements and attributes, and partially serves as an implementer's guide. The Publisher's Guide is more concerned with aspects of particular conversions, and topics like choosing an open license.

We have formatted a style guide for source, so that contributors will feel comfortable working with another author's material, and so material can be shared and remixed across different projects. This should remind Python programmers of the infamous PEP 8 [PEP8].

Book Design

A printed book delivers information in a format that has evolved over hundreds of years. Some aspects of a book fail miserably in an electronic medium. For example, page numbers. We believe others are still important, like a hierarchical organization expressed through chapters, sections, subsections, and so on. Not unlike this article. So a decision to use XML syntax is natural.

There are many more examples of our fourth Principle: PreTeXt respects the good design practices which have been developed over the past centuries. Such as, we limit the width of text in HTML output to fall within the optimal number of characters for the mechanical and mental aspects of reading.

LaTeX and MathJax

Mathematics is complicated. And complicated typographically. The simplest fraction becomes a two-dimensional exercise rather than a linear exercise (which is not to really imply that regular typography is not two-dimensional). A sequence of complicated expressions (aligned on an equal sign) or a matrix with three rows and five columns, where entries contain symbols, quickly becomes challenging.

MathML would seem an obvious choice. However we want authors to be able to write and edit their source directly at this point. So requiring MathML would suggest another tool to compose expressions. Instead we rely on MathJax [MathJax], a Javascript library to interpret and render mathematical expressions expressed in LaTeX syntax. It could be said that MathJax really is the enabling technology for this entire project. TeX is great between the dollar signs (its math delimiter), and MathJax supports a broad enough subset of symbols and constructions to satisfy almost any technical author.

So an author constructs their mathematical expressions with tools and techniques they may already know. A conversion to HTML puts the right Javascript on the page along with the raw LaTeX in the right wrapping. Features, such as screen-reading of mathematics are simply a consequence of employing the MathJax library. For a conversion to LaTeX/PDF/print the raw LaTeX is similarly dropped into the right place, and the resulting PDF output leverages the thirty years of development of TeX.

Lists and Display Mathematics

The logical placement of lists and display mathematics (runs of centered equations) was a difficult decision. Rightly or wrongly, we decided that these items are parts of paragraphs (rather than siblings of paragraphs). A fortunate consequence is that the conversion of deeply-nested lists became less difficult. An unfortunate consequence is that this decision is contrary to the HTML specification, so we must frequently explode a PreTeXt paragraph into a sequence of mixed HTML paragraphs, lists, and display mathematics.

Print Representation of Interactive Content

Including interactive components in the online version of a text begs the question of what to do for a print representation. Our solution is to include in print a screenshot and a link, accessible through a matrix barcode, specifically a Quick Response (QR) code.

Horizontal Flow

Normal text is linear across the page as a sequence of characters, but is then stacked vertically as a sequence of lines, paragraphs, figures, lists, etc. as you move down the page or screen. But sometimes you want to go horizontal. Maybe you have three images from a science experiment, and a comparison will work best if you line them up side-by-side. Or, because width always seems to be at a premium, you do not want to leave large amounts of whitespace, and so want to place a table and some text next to each other.

We accomplish this with a container named sidebyside, which suggests that it is specifying layout. Carrying crude controls for widths, margins, and vertical alignment, it probably is a layout device. But it is incredibly useful, and important enough to get an annual refactor every summer.

Short Names, Long Names, Attributes, IDs

XML does not have to look like it was machine-generated. We have tried very hard to make the vocabulary easy to remember, easy to use, intuitive to read, and not overly verbose. To this end, we use full names (not abbreviations) for elements used infrequently, such as introduction, proposition, and subsection. For frequently used elements, we have kept the names short, often borrowing from HTML, such as p, em, ol, and li. And we have added our own, such as c (code), q (quotes), and sq (single quotes).

We have tried to keep attributes to a minimum, especially by providing sensible default behavior in their absence. Boolean attributes are named to work well with only the values yes or no (not true/false, nor 0/1). The xml:id attribute is used as more of an author-provided short-identifier. It is the basis of the cross-referencing system, and migrates to other identifiers, such as URLs in the HTML output.

We try to recycle element names as much as possible, when resulting behavior will be predictable. The best example is title, which is ubiquitous. Sometimes this backfires and makes constructing XSL templates difficult. Some of the places an introduction can appear are in a chapter, prior to a list of objectives, at the start of a project, and at the start of an exercisegroup. This works well for authors, but requires some care in processing.

Escaping Escape-Character Hell

With many different markup languages in play there can be considerable confusion moving from one to another. Thankfully, XML has very few special characters. LaTeX has at least ten, JSON has several, Markdown has a multitude, including combinations. In technical documents there is often a need to express every single character literally. PreTeXt aims to insulate an author from all this. An interesting exercise is to give examples of a markup language, authored using that same language. But of course, for us, the author needs to write in XML!

Consider the ampersand, &. (How was that just authored?) In LaTeX this symbol is used to separate cells of a table, or separate entries of a matrix, or provide alignment points in a sequence of displayed equations. And in XML is the the escape character. So PreTeXt provides an ampersand element for normal text use, which can be output properly into a given markup language. For use in mathematics, PreTeXt makes available the \amp TeX macro (which is actually built-in as part of MathJax, and a simple addition for LaTeX output itself). For verbatim text (such as a logical or bit-wise and-operator in a programming listing) we provide education and advise authoring & (how was that just authored?). The conversion then handles literal characters according to the expectations of the target language. We very assiduously avoid instructing authors in the use of a CDATA section.

Coordinating LaTeX and XSLT Numbering

LaTeX provides automatic hierarchical numbering of divisions (chapters, etc.) and their contents (theorems, figures, etc.) through a system of counters which increment as used and reset as boundaries are crossed. The depth of the numbering, the number of different counters, and the boundaries where they reset, can all be configured by a knowledgeable user of LaTeX.

In PreTeXt we implement a significant and flexible subset of this behavior through the use of the xsl:number instruction. Remarkably, we are able to provide authors with a wide range of options for customizing numbering, write hard-coded numbers into HTML output, create preamble instructions for LaTeX, and have the automatic numbers produced by LaTeX match the numbers created by counting nodes in the XML tree.

Whitespace Handling

LaTeX is fairly permissive about whitespace, though not as much so as HTML. For example, two consecutive newlines indicates the start of a new paragraph. And there are some instances where a blank line will cause a fatal error (e.g. the align environment for multi-line mathematics). Sometimes excess space at the end of a line will cause subtle additions of unwanted vertical space.

Authors have been conditioned to expect to a lot of freedom in using whitespace liberally. Further, authors accustomed to programming and revision control like to think of text in 80-character chunks, so want to be very liberal with newlines. Fortunately, xsl:strip-space solves the problem of whitespace interspersed with the elements of structured content, but mixed content proved more delicate. At least we know where that is, and where it should go. So we can effectively sanitize normal text and LaTeX mathematics without doing any harm and meeting the expectations of the output format.

A happy consequence of sanitizing the mixed content is that we have a good handle on what that version looks like. We can then provide services like the following. Recall that we decided that display mathematics belongs in a paragraph. When several equations in a row are displayed, and end a sentence, then the period should be rendered in the display, at the end of the last equation (which could have a number pushed to the right margin). The author places the period after the closing tag of the element containing the mathematics, where it logically belongs, and the stylesheet wraps it properly for its automatic insertion into the display at the right place.

Conversion of Legacy LaTeX

There is a wealth of valuable material authored in LaTeX. There is no converter that reliably does a good job converting it to a structured form, since the extensibility of LaTeX and the numerous consequent add-on packages, give an author freedom to accomplish many variations. (Remember that Tex has a Turing-complete language.) However, David Farmer maintains a Python script, based on the use of regular expressions, to break apart a LaTeX document, throw away formatting information, deduce intent and purpose based on previous experience, and then reassemble the content in a structured way, using PreTeXt as the language. His script is not perfect, and never will be, but with each conversion it improves. An author needs to do a careful review, and a little bit of tedious editing, but quickly gains new output formats beyond a PDF.

Duplicated Content

Some content migrates to new places, like titles going to a Table of Contents. Other content gets duplicated, like the statement of an exercise being reprised in the back of a textbook. We implement cross-references to all but the largest chunks of a book via knowls, which duplicate content.

So it is important to recognize when content is being born, and so is the original version, and should have a unique identifier, such as an HTML id attribute, or when it is a duplicate version and none of its components should have a repeated version of an identifier. This is simply a matter of careful programming, and passing a flag (original versus duplicate) to templates.

Verbatim Text

When writing about computer code, either for a text in computer science, or in a discipline that is enhanced by computation, there is the need to use verbatim text that can display any character (including reserved, special, or escape characters). And a typical presentation is a fixed-width font of slightly heavier weight, meant to remind one of typewriters, teletypes, and primitive consoles.

This simply becomes a tedious and tricky exercise to let authors write without concern for the various output formats and let the stylesheets and templates perform the necessary manipulations. We have empty elements for many such characters. Some are necessary, such as ampersand described above, others are a convenience, such as copyright for the circled-c symbol.

Scaling Images

We do not believe the size of images should be given in explicit units by an author, nor do we think it is the job of PreTeXt to manipulate the aspect ratio. But the natural size of an image is not a reliable indicator. So we have authors specify desired width as a percentage of available width. This works well, especially since width is a scarce resource. However, it lacks precise control, and some authors desire to have the font-size of their diagrams exactly match the font-size of the surrounding text, in all possible formats. We do need to make this approach more robust, but do not yet have a concrete plan.

Styling and Themes

The CSS used in our HTML relies on extensive use of semantic class names, with sensible HTML elements for fallback styling when the CSS is not available. We have begun to improve the CSS by refactoring to provide greater modularity and extensibility. So providing alternative styling through additional CSS should see excellent support soon.

A strong suit of LaTeX is its customizability, though that is also one of its weaknesses. We have some good ideas on how to mimic the versatility of CSS to style LaTeX output via a system of hooks in existing templates and simple imported stylesheets.