Balisage Paper: Interchange vs. Interoperability

Teaser

Figure 1

pussycat in front of Thermes de Cluny, I think. Taken 2006-07-05.

Interchange, interchange, where have you been? / I’ve asked “interoperability — is it a sin?”

Interchange, interchange, where have you been? / I’ve asked interoperability: “is it a sin?”

The two captions above ably demonstrate that markup (in this case typographical, not XML) has meaning. In both cases, the narrator asks the personified interchange where she has been. In the first case, interchange responds by stating that she has asked the question is interoperability a sin. Whereas in the second case, she replies by stating that she has asked the personified interoperability whether or not it is a sin (whatever it is).

Introduction

In the digital humanities, we encode texts for three primary reasons:

The first goal (longevity) can be met by using a very common, well-defined, open standard character encoding (i.e. Unicode), and by sticking to human-readable, well-defined, open standard text formats (e.g. TeX or XML). That is, by avoiding any proprietary or binary formats.^[2]

The second goal (analysis) can be met by sticking to what we used to call “descriptive markup”MarkFut but is now better described as “indicative logical markup”Flaw, and developing a sufficiently detailed description of your data for the purposes envisioned. That is, to develop a markup system that separates out the parts of the document you find interesting and describes that which is interesting about them in sufficient detail.

The third goal (sharing) can only be met if the others you are sharing with understand that which you are sharing. It does you no good whatsoever for me to send you a manuscript transcription in a proprietary binary format which your computer cannot read, and not much good to send it in an open text format you do not understand. In order for you to make use of the transcription, we need to have a shared understanding of both the operating details and semantics of the markup language used.

However, because the details of humanities texts that scholars wish to engage with and the assertions scholars wish to make about these texts are (almost definitionally, I daresay) so widely varied, no single markup language could possibly represent them all. That is, no markup language can meet the second goal for all scholars simultaneously. A given markup language can represent some of the analytic needs of all of the scholars, and all of the analytic needs of some of the scholars, but it can not represent all the analytic needs of all of the scholars. This holds whether the markup language is a widely used standard language like TEI, or a home-grown language specific to the purpose.

Even if the third goal were about sharing data with a particular individual, a shared language would be required — in order for a language to be shared, both the sender and the receiver, regardless of discipline, whether they know each other or not, or whether they can even talk to each other or not, must agree on it, even if only implicitly. But in fact the third goal is about sharing data in a collaborative fashion with almost arbitrary interested parties. In this case, an agreed upon and widely known markup language is required.

The TEI Guidelines try to be that language. The full title of the impressive and ubiquitous guidelines is Guidelines for Electronic Text Encoding and Interchange (emphasis provided). But with the advent of TEI P5, the focus, I claim, has been more on interoperability than on interchange. This paper will explore the meaning of the two terms, particularly with respect to the aforementioned goals of text encoding, and the impact of trying to achieve interchangeability or interoperability on encoding languages (TEI in particular) and thus the communities served.

Definitions

Standard dictionary definitions of these terms do not provide a particularly useful perch. E.g. the online edition of the Oxford English Dictionary definition of interchange (as a noun) includes the concept of reciprocity: The act of exchanging reciprocally; giving and receiving with reciprocity; reciprocal exchange (of commodities, courtesies, ideas, etc.) between two persons or parties. oeinch. Merriam-Webster’s definition of the verb interchange is more succinct, but still has at its core the idea of reciprocity, i.e. of swapping places: to put each of (two things) in the place of the other mwinch.

The Guidelines do not formally define the term, but the meaning can be gleaned from the operational definition provided: The TEI format may simply be used as an interchange format, permitting projects to share resources even when their local encoding schemes differ. http://www.tei-c.org/release/doc/tei-p5-doc/en/html/AB.html#ABAPP2.

As for interoperable, the OED is quite succinct, but somewhat unhelpful: Able to operate in conjunction. oeinop Merriam-Webster is mildly helpful, although it is not clear whether their systems include humanities encoding systems: ability of a system (as a weapons system) to work with or use the parts or equipment of another system mwinop.

The Guidelines occasionally use the word interoperability (4 times), but never define it. I would like to interpret the use of interchange and interoperability in the Guidelines as implying that they are different: As different sorts of customization have different implications for the interchange and interoperability of TEI documents, … http://www.tei-c.org/release/doc/tei-p5-doc/en/html/USE.html#CFOD. However, it is possible that the author of that sentence thinks of interchange and interoperability as synonyms, and is just trying to be emphatic or completely clear.

But moreover, these are not the definitions the designers of TEI (and perhaps other data interchange systems) had in mind when they originally discussed interchange, whether negotiated or blind. E.g., back when TEI P1 was released (1990-07), many of the concerns about interchange were concerns about just getting the data from one machine to another with the same characters. Only then could you worry about the syntax of the encoding system and the semantics of the particular encoding language.

It is no small accomplishment that today character fidelitous interchange is easy. It has taken the advent of more standardized character encoding (i.e., ISO 10646 and Unicode), more widely and uniformly supported transport protocols (e.g., FTP and HTTP), standard encoding schemes to work-around 7-bit mail gateways (e.g., base64 encoding), and standard ways to indicate content type (e.g. MIME), for us to be able to transport files from one place to another with such ease that we barely think about it. These great strides in computer technology allow us to take faithful transmission of a series of characters for granted; thus I will not consider it further in this paper.^[3] Rather here I consider these terms in light of the syntax and semantics of the documents in question, rather than in the operational details of how to move the documents from here to there.

So, for purposes of this discussion of the transfer of encoded information from one party to another, I’d like to present operational definitions of these terms as follows.^[4]

So an interoperable text is one that does not require any direct human intervention in order to prepare it to be used by a computer process other than the one(s) for which it was created. There are two caveats worth mentioning, here.

Variants of Interoperability

Using these definitions of interoperability, of course, means that whether or not a document can be considered interoperational depends very much on the process with which we would like to interoperate. For simple enough processes, every XML file is interoperational. That is, any XML file will operate with the process. Consider, e.g., the process that counts elements in a file (count(//*)): no matter what non-conformant, arbitrarily invalid, or silly TEI documents you send this process, no human intervention will ever be needed.^[5]

But there are cases toward the other end of the spectrum, as well. It is hard to imagine a process that could read in an arbitrary conformant TEI file and typeset a book matching the publisher’s house style from it. Take, for example, just the case of getting headings into the proper case. If house style is that headings should be title-cased with acronyms in small caps, some of the following perfectly reasonable, valid, conformant TEI <head> elements will be very hard to process properly.

The point here is that being interoperable is not a static, independent feature of a document. Being written in French, being a standalone XML document, these are static, independent features. But being interoperational depends entirely on which application you are trying to interoperate with. That is, interoperationality is very context dependent.

Interoperability is Difficult

Many XML user communities will at least bristle at my assertion that interoperability is difficult, and some people will even object, using their own projects as counter-examples. For example, it may be quite easy to take documents in SVG or Triple-S and use them in an application other than the one for which they were originally written. But I hold that interoperationality is not just about use of XML data in current predicted applications (I will want to use this file in an SVG viewer), but rather is also concerned with the use of data with both unintended applications (e.g., linguistic analysis of survey questions) and future applications (e.g., submitting a set of vector graphics images to software that makes a mosaic of them).

Notice that I am being (deliberately) self-contradictory, here. I am suggesting that interoperability is always contextual, that we need to know what application we are trying to operate with in order to measure our success at interoperationality; and simultaneously that in order to be considered interoperational, a document should work with applications that are not only unforeseen, but haven’t even been written yet. If the application has not been written yet, it is obviously impossible to know whether or not our data will work with it. It is this very contradiction that drives me to say that interoperationality is hard. We cannot know the various contexts in which we would like our files achieve interoperationality.

Figure 2: Pushmi-Pullyu

One take on the Pushmi-Pullyu. A somewhat more authentic take (IMHO) as available as a video on YouTube

This contradiction does not necessarily make interoperationality difficult for a language like SVG, which has both very tight operational (as opposed to declarative) semantics and a sharp focus on particular kinds of processing. But TEI is a declarative language that is intended to record, for various disparate purposes, an enormously wide array of texts (to wit, those of interest to humanities scholars). Thus its semantics are necessarily somewhat looser, its ambitions are sky-high, and its field of view is very wide.

For example, consider an application (present or future) that for some purpose wishes to process the authorial words in the main content of a TEI-encoded document; that is, we need to parse out the words, but ignore things like metadata, running heads, editorial notes, chapter titles, etc. Many TEI projects have had reason to extract the words in this way, and have found it not particularly difficult to do. But doing so for a generic TEI file turns out to be impossible, and many people find this surprising. (Remember that TEI, like many languages, is really a framework for building a particular markup language with many, but not all, features in common with other TEI languages.)

Before demonstrating that this is impossible, I will first elaborate on one of the reasons it is hard: end-of-line hyphenation. In order to parse words out from the input document, we need to know whether a hyphen at the end-of-line is hard or soft. In many cases this is obvious to a human (pro-tection vs. pro-choice), but in some it is not at all clear (e.g., Sea-markCeHx), and it is generally harder for computers than for humans. The TEI currently (P5 v. 1.9.1) provides at least five different methods for encoding soft hyphens at end-of-line, three of which might use different characters for the representation of the hyphen itself (which is displayed as a hyphen (U+002D) in the listing below), and some previous releases of TEI allowed even more. This may strike some as silly or even excessive, but in most cases these choices make sense. For example, the TEI cannot prescribe to scholars who are not remotely interested in original lineation or soft hyphens that they record these features.

      pro-
      tection

      protection

      pro<lb break="no"/>tection

      pro-<lb/>tection

      pro-<lb break="no"/>tection
    

The multitude of encoding methods for soft hyphens at end-of-line would not be a problem if an application could easily query the document and ascertain which encoding was used. The good news is that TEI does actually provide a specific construct^[6] for encoding how end-of-line hyphens are handled. The bad news is that the vocabulary available to describe the different encodings is insufficient to the task; furthermore, it is optional, and many, if not most, projects do not use it.

End-of-line hyphenation is but one of many problems that make extracting just the words from generic TEI files difficult. Some, however, actually make it impossible. For example, in TEI annotations or notes are recorded in a <note> element, whether the note is an aside, an annotation, a footnote, marginalia, a textual note, a comment to the encoder’s manager, or whathaveyou. The TEI provides a @type attribute for differentiating these different kinds of a note; but since there is no pre-determined controlled vocabulary for this attribute (rather, projects are expected to come up with their own vocabularies), and furthermore the attribute is not required, generic TEI processing software cannot really do much with it (at least not without human intervention). Thus while it is easy for any given project (which knows how annotation vs. textual notes were encoded) to differentiate, it is impossible for general-purpose TEI software to do so.

Interchange is (not) Difficult

Some might think that, on the surface, interchange is harder than interoperation. After all, it requires human intervention. And certainly dropping a file into a system and presto it works is far easier than doing work to get your system to process the file as desired. But we’re talking here about the effort needed to make a file that is interoperable vs. interchangeable. As I’ve just described, for many purposes it will be impossible to process a random TEI file with a given system without intervention. Thus however difficult it is to make interchangeable files, it is easier than making interoperable ones.

The bad news is that it can take quite a bit of work to make a file easily interchangeable with generic future processes. The good news is that the effort put into making it interchangeable does double-duty, and simultaneously increases the data’s longevity by making curation of the file easier. (After all, porting a file forward to a different platform of some sort is a process, and we can think of the file as being interchangeable with that process.)

The reason it takes work to make a file interchangeable is because (in general) it requires that the semantics, not just the syntax, of the encoding be explained to the humans involved with consuming the file. In order to know which <note> elements should be ignored, our prospective word-counter needs to know what the various possible values of the @type attribute mean. Yes, one can imagine a system in which the values are inherently obvious to one who speaks the natural language which the values (which are arbitrary NCNames) are designed to mimic ("authorial", "transcriptional", etc.), but this is not the general case, and moreover there is no guarantee that the future consumer of the file speaks that natural language or uses those terms the same way.

Thus contemporaneously generated detailed prose documentation of the indicative logical markup constructs used to mark up a text can significantly facilitate the use of the text in future systems and processes, both those we can predict and those we cannot, both by those that generated the text and by unforeseen recipients of it. Put more succinctly, providing descriptions of the descriptive markup can make future interchange of a document possible where it might have been impossible, or easy where it might have been hard.

Interoperability vs. Expression

It is an old chestnut of political theory that liberty and equality are at least at odds with one another, if not completely incompatible societal goals.^[7] Any economic system, social engineering, regulation, or legislation that enforces equality will limit liberty, and any that encourage liberty will make equality less likely to obtain.

Liberty in general, and freedom of thought, speech, and the press in particular, drive both individuals and (more importantly) society to improve.

In society at large, the free expression of ideas — be they true or false, good or bad — improves the society as a whole and the condition of mankind in general, for in the marketplace of ideas eventually the true and good win out.

That said, there are societal advantages to equality as well. A pure communist society eliminates not only relative poverty and hunger, but also alienation and class oppression.

Likewise (I claim), interoperability and expressiveness are competing goals constantly in tension with each other. While the analogy is far from exact, interoperationality is akin to equality, and expressiveness is akin to liberty. And the very advantages ascribed to political liberty in the social sphere can be ascribed to the liberty of expressiveness in the encoding of humanistic texts. The free expression of scholarly ideas embodied in the encoding of a text puts our encoding systems to the test, and helps to improve them.

Those areas of an encoding system that are designed to make the resulting documents more equal to all the other documents that will be used in the target process and thus more interoperational are going to curtail the encoder’s liberty to be expressive. Those areas of an encoding system that are designed to permit the liberty of expressiveness (e.g., the @type attribute of <tei:note>) are going to permit documents that are less like (i.e., less equal to) the other documents intended to be consumed by the target process, and thus less interoperational.

Said another way, to make her document (more) interoperational, an encoder either needs to know the application semantics of interest ahead of time, or needs to stick rigidly to prescriptions (sometimes, but not always, expressed as schemas) that she believes the target application (which may not be known) will be able to understand. But rigidly sticking to the encoding that (she thinks that) the target application needs, or encoding to the stylesheet robs the encoder of the ability to be expressive in ways that (she believes) might not be correctly handled by the target application. Common sense says this could result in, and my experience says this often does result in, encoder’s deliberately using encodings that are substandard, less faithful to the document, or outright incorrect per the encoding language, in order to achieve the desired results.

In the general case, interoperationality (at least of random TEI-encoded files) is impossible. But that does not stop encoders from striving for it, and in so doing curtailing their own ability to express the features of interest to them in individual ways. While this is not always a bad thing, differing expressions of content are part of the force driving scholarship forward and cannot be entirely forsaken for the short-term goal of avoiding the work needed for interchange as opposed to interoperation.

Expression and Interchange

For humanistic scholarship variation of expression is a necessity. Ask a room full of scholarly editors to examine a document in their field, and you will have as many interpretations (i.e., encodings) of the document as there are editors. And no matter how expressive your underlying language is, if you ask enough scholars you will come across cases where the interpretation of the document cannot be straightforwardly expressed in the standard language. Additional expressiveness, a deviation from the standard, will be required.

Just making such deviations silently is a recipe for disaster. Consumers of the document will presume that it adheres to the standard and will find the more expressive bits either uninterpretable or, worse, will interpret them incorrectly.

But carefully documenting such deviations has multiple benefits:

Conclusion

In summary, since interoperationality (equality) is often bought at the expense of expressivity (liberty), interoperability is the wrong goal for scholarly humanities text encoding. Practitioners of this activity should aim for blind interchange. This still requires a lot of adherence to standards (e.g., TEI), but eschews mindless adherence that curtails expression of our ideas about our texts.

Acknowledgments

I would like to thank Paul Caton, C. Michael Sperberg-McQueen, and Kathryn Tomasek for their assistance.