Balisage Paper: Encoding Transparency: Literate Programming and Test Generation for Scientific Function Libraries

Introduction

We outline a (nearly) language-neutral approach to embedding in ordinary source code semantic information that we use to generate documentation for multiple audiences, unit tests, and code translation templates. Our approach is within the literate programming tradition for enhancing the usability of software documentation, specifically the variants that focus on inline documentation. Accordingly, we embed the structured documentation within the ordinary comment syntax available in all common programming languages.^[1]

Knuth (1984, 1992) introduced literate programming to negotiate the fundamental tension between the needs of two readerships for computer programs: humans and compilers. Knuth’s original recommendation and most implementations work from a hybrid literate document containing the information needed to generate both machine-readable code and human-readable documentation.^[2]For example, Knuth’s (1984) first implementation, called WEB, provided two toolchains to export information from the literate source file: (a) TANGLE extracted executable source code for delivery to a Pascal compiler; and (b) WEAVE extracted documentation for delivery to a TeX interpreter.

Despite its impeccable pedigree and a devoted community of supporters, literate programming as originally envisioned has not gained widespread acceptance. Wilson (2011) even calls it a beautiful idea that failed. The psychological order requirement that literate source documents be arranged to best serve the human reader implicitly places a heavy burden on the programmer, whose human-facing literate document must simultaneously satisfy an error-intolerant compiler. The programmer must have fluency in both the literate syntax and the compiler syntax. Even for coders with both skills, there is a cognitive cost to switching repeatedly between the two commingled dialects-- one unwoven and the other untangled. As a result, the original literate programming paradigm is perhaps best suited to applications where the emphasis is inverted: pretty printing is indispensable and the code itself is in a sense secondary, as in Leisch’s (2002) Sweave, which allows statistical code to be embedded in scientific source documents to enable reproducible research.

On the other hand, less orthodox technologies in the literate tradition (i.e., combining executable source code with structured documentation) have been quite successful. Javadoc and its generalized peer Doxygen, which rely on structured comments embedded in source code to generate documentation, are among the most prominent examples. Similarly, tools like Python’s pydoc generate documentation from docstrings, string literals that are retained and accessible at runtime as special properties of objects. Under both approaches, the comparison to orthodox literate programming is instructive: structured documentation is still embedded with executable source code, but the requirement of psychological ordering is dropped. The compiler, by being stubbornly inflexible, has won the battle over how to order the statements in the source file.

Orthogonal to the literate programming paradigm, but still important for our project, is the realization by Peters (1999) that the usefulness of docstrings for documentation purposes could be harnessed to provide users with a simple, inline method for generating unit tests. This realization led to his creation of doctest.

Our approach, outlined in the following sections, follows in the tradition of Javadoc-style inline documentation, but we use XML intermediates with XSLT not only to allow for the generation of documentation for multiple audiences, but additionally to define and generate unit tests, as in doctest, and to generate templates for code translation. Section 2 provides further context for our project, section 3 describes the targeted use cases, and section 4 concludes.

Context and objectives

Function libraries play a prominent role in scientific computing.^[3]A function library is a collection of well documented, callable routines of the form y = f(x), where each function takes some argument list, x, executes one or more statements depending only on x, and returns a list of values, y. Scientific functions tend to involve specialized, technical logic that can be obscure to non-experts. As a result, there is a greater-than-usual need for good documentation and extensive testing. At the same time, scientific routines typically adhere to the functional-programming convention of exhibiting no side effects. Routines without side effects are especially amenable to unit testing, and we restrict attention to this class of functions to contain the scope of this exercise. Lastly, scientific libraries are more likely than most to address technical issues that are abstract to the implementation context. For example, an invocation of a quadratic programming routine will have essentially the same meaning (execution logic) regardless of the language in which it is written; the same is not true for a resize_frame routine in a GUI windowing toolkit. Because of this, scientific routines are more amenable to porting from one programming language to another.

Our technique is at once more and less ambitious than orthodox literate programming. We similarly target multiple simultaneous readerships, both human (e.g., coders, testers, analysts, etc.) and compilers/interpreters (e.g., C++, Python, Fortran, etc.). However, like Javadoc, and in keeping with Dijkstra’s (1972) admonition that, brainpower is by far our scarcest resource, we concede the impracticality of psychological ordering. In particular, there is no tangling to extract executable statements; instead we work with valid source files that can be delivered to the compiler as is. We also hesitate to propose our method as broadly applicable to any programming paradigm. For example, our source-code portability proposal would likely be more complicated in an object-oriented environment, due to the possible presence of state-altering side effects and the (typically) fragmented sequence of control.

Since we use a form of inline documentation that, like Javadoc and others, exploits the existing commenting syntax available in some form in all common programming dialects, ours is highly versatile with respect to the choice of source language, to the point of being nearly language-neutral. This provides significant benefits when developing parallel implementations of scientific function libraries in a number of languages.

On the ambitious side of the ledger, we are interested in more than simply weaving handsome end-user documentation from embedded, structured documentation, although pretty printing is very much in scope. We also extend the basic methodology to encompass basic unit testing of functional logic. In the absence of side effects, it becomes straightforward to state the most common unit-test assertions declaratively and embed these declarations within comments in the source code. Due to the desire to be language-neutral, ours is not as straightforward or elegant as doctest, but it does provide many of the same benefits. Lastly, we use the template processor required by our documentation generation toolchain to offer a method for creating function prototypes-- including pseudocode-- for ports of an existing program to essentially an arbitrary target programming language.

While none of the individual components of this approach is by itself novel, we find that at least within the restricted scope of scientific function libraries without side effects, significant benefits are provided by the flexibility of our approach and the synergistic effects of using all of the individual elements of inline documentation, unit test generation, and code translation in concert.

Specifically, we note that the problem of code translation consists of three sub-problems: ensuring that the logic of a particular function is consistent across languages, performing the syntactic translation between languages, and optimizing the implementation within each language. Of these, the implementation is where a good programmer is needed most, both to write code that makes use of the appropriate idioms provided by a language, and to optimize the implementation of a particular set of logic. Thus, by providing a programmer with a template for a function that both defines the function signature and provides ready-made documentation from another language, we reduce unnecessary burdens on programmers porting the original source code. Additionally, by providing unit test routines, we enable easier verification of ported code. Finally, by examining discrepancies between implementations, we reduce the model risk inherent in any single implementation.

We consider our approach to be a simplified variation of the literate programming paradigm, targeting multiple simultaneous readerships, both human (e.g., coders, testers, analysts, etc.) and compilers/interpreters (e.g., C++, Python, Fortran, etc.). We offer a prototype implementation in XSLT and DocBook.

Usage scenarios

In this section, we describe the process in somewhat greater detail. As indicated above, the techniques cover three important usage scenarios for managing source code in function libraries.

End-user documentation

Knuth’s (1984, 1992) original vision for literate programming emphasized the need for source code to communicate with human programmers as well as with compilers or interpreters. A well documented program should speak clearly to the coder about what the code is doing, and the programmer should speak back by enhancing and refining the documentation. Knuth quotes a well known passage from Hoare (1973, 3),

documentation must be regarded as an integral part of the process of design and coding. A good programming language will encourage and assist the programmer to write clear self-documenting code, and even perhaps to develop and display a pleasant style of writing. The readability of programs is immeasurably more important than their writeablility.

Knuth’s (1984) initial implementation, called WEB, mixed discussion and logic in a single source file, which is then pulled apart to create both executable code and typographic source for onward processing by Pascal and TeX, respectively. Figure 1 depicts the basic structure of the workflow in WEB (adapted from Figure 1 in Pieterse, Kourie and Boake, 2004, 113):

Figure 1: Figure 1

Literate programming workflow in WEB

A key design decision here is to mix documentation and logic as co-equals in the source. Because the source document adheres to psychological ordering rather than executable ordering, a burden falls on the programmer to think bilingually (see Wilson, 2011) to understand how the source document will be simultaneously tangled and woven to the executable and typographic dialects.

We adopt a variation on this basic workflow, similarly mixing documentation and logic in a single source file, depicted in Figure 2 (a familiar example of this same workflow is Javadoc):

Figure 2: Figure 2

Documentation and logic in Javadoc

An important difference in this architecture relative to orthodox literate programming is the use of a compiler-valid source document. Because compilers impose rigid validation rules on executable code, this relationship can be and is typically managed by programming IDEs with facilities such as real-time validation, syntax highlighting, debuggers, profilers, etc., sharply reducing the bilingual burden.

As described above, we embed documentation steganographically in the ordinary comment lines within the source code. A sample appears in Figure 3, using standard XML angle brackets as documentation markup within a Matlab or Octave source file. Comments containing content intended for delivery to the final end-user documents get a special syntax, %# instead of simply %. Given a source document in this form, the comment processor performs four straightforward pre-processing steps, resulting in valid XML that encapsulates all of the content required for end-user documentation:

1. Wrap any executable statements in <code> tags

2. Delete ordinary comment lines

3. Convert documentation comments by removing the leading %# markers

4. Wrap the entire document in <codefile> tags

Figure 3: Figure 3

A working example of steganographic documentation

An obvious (and planned) enhancement to this is to replace the angle brackets with a more felicitous markup scheme, such as Markdown, Textile or YAML.^[4] Note that we have prototyped this process on Matlab/Octave source files, but it should work with any programming language that supports inline comments containing arbitrary text. The pre-processing routine is therefore customized to each source language, but the subsequent processing steps would be the same for any source language. Given a valid XML representation output by the preprocessing step, the next processing phase is an XSLT transformation to a standard publication format such as DocBook or DITA. Transformation from this intermediate form to final print or web format is then straightforward via standard tools.^[5]

Conclusions

We have outlined a general approach to automated documentation, unit-testing and code portability for scientific function libraries, using the ordinary comment syntax as a vehicle to embed declarative logic steganographically in the source code. This addresses immediate and practical needs in our workplace, and may be useful to others similarly situated. We have a working prototype of significant portions of this tool chain, and are working to build it out further.