Balisage Paper: The structure of content

Introduction

The basic purpose of markup is to make the implicit structure of texts explicit; the same is true of "text analytics" (or "TA"), a fast-growing application area involving the automated extraction of (some) meaning from documents. In this paper I will try to place markup and analytics in a larger frame, and discuss their relationship to each other and to the notion of explicit vs. implicit information.

To begin, note that 'implicit' and 'explicit' are matters of degree. Language itself is an abstract system for thought, entirely implicit in the relevant sense here. The words and structures of natural language make thought more explicit, fixing it in the tangible medium of sound (of course, some people cannot use that medium, but still use language). Writing systems take us another step: particularly in their less familiar, non-phonetic features: sentence-initial capitals, quotation marks, ellipses, etc. all make linguistic phenomena explicit, and it has long been noted that punctuation (not to mention layout) are kinds of "markup"[Coom87], albeit often ambiguous. In a yet broader sense, the "s" on the end of a plural noun can be considered "markup" for a semantic feature we call "plural", and a space is a convenient abbreviation for <word>. XML markup thus fills a "metalinguistic" role analogous to punctuation, though (ideally) richer and less ambiguous.

'Documents' is also an imprecise term. Here I am focusing only on the more traditional sense: human-readable discourses, in a (human) language, with typically near-total ordering. This doesn't preclude including pictures, data tables, graphs, video, models, etc.; but I am for now excluding documents that are mainly for machine consumption, or are in a specialized "language" such as for mathematics, vector graphics, database tables, etc.

What, then, can text analytics do for XML? What are some phenomena in texts that we might make explicit, or check, using text analytics methods? How do these mesh with how our linguistic and markup sytems are organized? An anonymous reviewer raised the valuable question of where text analytics can be effective for auto-tagging material; which in turn poses an old problem in a new guise: If you can use a program to find instances of some textual feature, why mark it up?

Tedious and brief: Language vs. Markup?

To explore these questions we may arrange reality into successive levels of abstraction and explicitness. This is similar to the typical organization of layers in natual language processing systems.

In large-scale text projects, the creation of high-quality markup is one of the largest expenses, whether measured by time, money, or perhaps even frustration. How do we choose what to mark up? Can text analytics shift the cost/benefit calculus, so that we can (a) mark up useful features we couldn't before, and (b) assure the quality of the features we have marked up?

Text Analytics to the rescue?

Text analytics tries to extract specific features of meaning from natural language text (i.e., documents). It mixes linguistics, statistical and machine learning, AI, and the like, to do a task very much like the task of marking up text: searching through documents and trying to categorize meaningful pieces. Like markup, this can be done by humans (whose consensus, when achievable, is deemed the "gold standard" against which algorithms are measured), or by algorithms of many types.

Text analytics today seeks very specific features; largely ones that are saleable; it does not typically aim at a complete analysis of linguistic or pragmatic meaning. Among the features TA systems typically seek are:

Text analytics is related to many more traditional techniques, such as text summarization, topical search, and the like. However, it tends to be more focused on detecting very specific features, and operating over very large collections such as Twitter feeds, FaceBook posts, and the like. The OpenAmplify[oa] service, for example, regularly analyzes several million documents per day.

A natural thing for a text analysis system to do, is to mark up parts of documents with what features they were found to express: this "he" and that "our renowned Duke" are references to the "Theseus" mentioned at the beginning; this paragraph is focused on fuel shortages and the author appears quite angry about them; and so on. Of course, some features only emerge from the text as a whole: the text may be about space exploration, even though that topic is never mentioned by name.

Users of text analytics commonly want to process large collections of short social media texts, searching for particular kinds of mentions. For example, Acme Mustardseed Company may want to know whenever someone posts about them, and what attitude is expressed. Beyond mere positive vs. negative, knowing that some posters are advocates or detractors (advising others to buy or avoid), can facilitate better responses. Other companies may want to route emails to tech support vs. sales, or measure the response to a marketing campaign, or find out what specific features of their product are popular.

Text analytics algorithms are varied and sometimes compex, but there are two overall approaches:

One method is almost purely statistical: gather a collection of texts that have been human-rated for some new feature, and then search for the best combination of features for picking out the cases you want. This "Machine Learning" approach allows for fast bootstrapping, even for multiple languages, and is sometimes very accurate. However, since it does not take much account of language structure, complex syntax, negation, and almost any kind of subtlety may trip it up. It's also very hard to fix manually -- if you go tweaking the statistically-derived parameters, unintended consequences show up.

The other general method is heuristic or intuitive: linguists analyze the feature in question, and find vocabulary, syntactic patterns, and so on that express it. Then you run a pattern-matching engine over documents. The plusses and minuses of this approach are opposite those of machine learning: It is much more time- and expertise-intensive; if the analysts are good the results can be amazing. But it's hard to do this for 100 languages. When problems crop up, the linguists can add new and better patterns, add information to the lexicon, etc.

As Klavans and Resnik point out[Kla96], it can be very effective to combine these approaches. One way to do that is to use statistal methods as a discovery tool, facilitating and checking experts' intuitions.

With either approach or a combination, you end up with an algorithm, or more abstractly a function, that takes a text and returns some rating of how likely the text is to exhibit the phenomenon you're looking for. For the trivial

    
    <s>
       <nps>
          <at base="the">The</at>
          <nn base="course">course</nn>
       </nps>
       <pp>
          <in base="of">of</in>
          <np>
             <jj base="true">true</jj>
             <nn base="love">love</nn>
          </np>
       </pp>
       <pred>
          <rb base="never">never</rb>
          <vb base="do">did</vb>
          <vb base="run">run</vb>
       </pred>
       <jj base="smooth">smooth</jj>
          <dl base=".">.</dl>
    </s>
    
    

Text Analytics' Relation to Markup

Markup has multiple purposes; among them are

Markup makes aspects of document structure explicit. In principle, any phenomenon that text analytics can identify, can then be marked up, to a corresponding level of accuracy. Exactly the same analytics can be used in checking: If a text is already marked up for feature X, we need only run an auto-tagger for X and compare. This simultaneously gives feedback on the text-analytic output's accuracy, and the prior markup's.

When two sources of data are available like this, they can be used to check each other. In addition, the degree of overlap in what is "found" by each source, enables estimating the number of cases not found be either. A simple statistic called the "Lincoln Index", originating in species population surveys, provides this estimate. In the same way, text analytics can be used to do XML markup de novo, or as a direct quality check on existing markup.

Such comparative analysis may be one of the most useful applications of TA to XML evaluation. In a text project where markup is not straightforward, how can one evaluate how well the taggers are doing? Say a literary project is marking up subjective features such as allusions, or sometimes-unclear features such as who the speaker is for each beat in dialog. TA methods can learn how to characterize such features, and then be run against the human-tagged texts. Disagreement can trigger a re-check, thus saving time versus checking everything.

There seems to be an implicit "sweet spot" for markup use. We don't mark up sufficiently obvious phenomena, such as word boundaries (except in special cases). Given that almost every kind of processing needs to know where they are, why not? Probably because finding word boundaries seems trivial in English.^[2] Yet word boundaries are also unlikely to be marked up in Asian languages, where identifying them is far from trivial. Thus, simplicity can't be the whole story. Perhaps it is that consciously or not we assume that most any downstream software will do this by itself, so there would be no return on even a small investment in explicit markup.

Language-use has better ROI, for example enabling language-specific spell-checking or indexing. Downstream software is perhaps less likely to "just handle it." Nevertheless, it is not very common to see xml:lang more than once in a document except in special cases such as bilingual dictionaries, diglot literature editions, and the like.

TA systems can certainly add (or check) word-boundary and language-use markup, and the most common related attributes, such as part of speech and lemma. Such markup is perhaps of limited value except in special applications, such as text projects in Classical languages or that contend with paleographic issues.

Marking up small-scope, very specific semantics such as emphasis and the role of particular nouns is a traditionally awkward matter in markup. Some schemas merely provide tags for italic, bold, and the like; using less font-oriented tags such as <emph> is considered a step up, but often accomplished little more than moving "italic" from element to attribute. If more meaningful small-scale elements are not available, conventions such as RDF[RDF] and microformats[micro] make it feasible to represent the results of text analytics or even linguistic parsing in ways accessible to XML systems.

DocBook[docb] provides many more meaningful items at this level: guibutton, keycap, menuchoice, etc. As with other fairly concrete features already described and as an anonymous referee pointed out, text analytics could be used to tag (or check) many such distinctions automatically: "ENTER" is going to be a keycap, not a guiItem; in other cases nearby text such as "press the ___ key" can help, as can context such as being in a section titled "List of commands". This seems entirely tractable for analytics, and could have significant value because such markup is valuable for downstream processings (particularly search), but tedious for humans to create or check, and therefore error-prone.

However, even this level of markup can get subtle. In Making Hypermedia Work[DeRo94] David Durand and I decided to distinguish SGML element names from HyTime architectural form names in the text, because the distinction is crucial but, at least to the new user, subtle. We also decided, in many examples, to name elements the same as the architectural form they represented. In most cases there was only one element of a given form under discussion; and because elements are concrete while forms are abstract, one cannot easily reify the latter without the former. In most cases this was trivial; but in a few cases the decision seemed impossible. Examing those cases via TA methods would likely reveal much about the distinction we were trying to achieve, as well as no doubt reveal marup errors.

Bibliography entries are notoriously troublesome, whether in XML or any other representation. They have many sub-parts, with complex rules for which parts can co-occur; the order (not to mention formatting) of the parts varies considerably from one publisher to another; and there are special cases that are difficult given the usual rules. PubMedCentral receives an extraordinary number of XML articles, often in a standard XML Schema[NCBI]; but usage varies significantly even in valid data, and the code to manage bibliographic entries in the face of such variability is substantial. Many publishers opt for the "free" style, in which most schema constraints are relaxed, and recovering the meaningful parts of entries is a task worthy of AI.

At a higher or at least larger level, many schemas are heavy on tags for idiosyncratic components with much linguistic content, which also have distinctive textual features: Bibliography, Abstract, Preface, etc. For example, a Preface will likely use much future tense, while a Prior Work section will use past. Text analytics can find and quantify such patterns, and then find and report outliers, which might show up due to tagging errors, or to writing which, whether through error or wisdom, does not fit the usual patterns.

This provides a particularly promising area for applying text analytics. Although the titles of such sections differ, and there may or not be distinct tags for each kind, a text analytics system could learn the distinctive form of such components, and then evaluate how consistent the tagging and/or content of corresponding sections in other documents are.

We usually mark up things that are necessary for layout; the ROI is often obvious and quick. But it takes a lot of dedication, sophistication, and money to, say, disambiguate the many distinct uses of italics, bold, and other typography in the Oxford English Dictionary[Tom91], or to characterize the implicitly-expected style for major components such as those in front and back matter.di. Many of the implicit data described earlier can be detected using text analytic methods, but using this to assist the markup process has been little explored.

How shall we find the concord of this discord?

If you can find it reliably via some algorithm, why mark it up? In a sense, creating markup via algorithms is kind of like the old saw about Artificial Intelligence: "as soon as you know how to do it, it's not AI anymore." If text analytics (or any other technology) could completely reliably detect some feature we used to mark up, we might stop marking it up. But in reality, neither humans nor algorithms are entirely reliable for marking up items of interest. The probability that the two will err in quite different ways, means there is synergy to be had.

Anyone who has tried to write a program to turn punctuated quotes into marked-up quote elements, has discovered that there are many difficult cases, at a variety of levels. Choice of character set and encoding, international differences in style, nested and/or continued quotations in dialog, alternate uses of apostrophe and double quote, and even quotations whose scope crosses tree boundaries[see DeRo04]. Would we bother marking quotations if the punctuation were unambiguous, or if we had widespread text-analytics solutions that could always identify quotations for formatting search, and other common needs?

Typical XML schemas define some very common specific items: Bibliography, Table of Contents, Preface; and some common generic items: Chapter, Appendix, etc. But (perhaps pragmatically) we don't enumerate the many score front and back matter sections listed in the Manual of Style, or the additional ones that show up in countless special cases -- at some point we just say "front-matter section, type=NAME" and quit. Worse, we sometimes cannot choose the "correct" markup: whether we are the original author or a later redactor, we may simply be unable to say whether to mark up "Elizabeth" as <paramour> or <tourist> in "Elizabeth went to Essex. She had always liked Essex."[TEI P3]

The short response to these issues, I think, is that markup is always a tradeoff; there are levels we make explicit, and levels we don't. Perhaps it cannot be otherwise. Intuitively, it seems that at least for authors many of the choices should always be clear; and to that extent text analytics can also find many of these phenomena. So why does the principle not work, that an author knows when component X is needed, and so should have an easier time just naming X, than carrying out commands to achieve a look that others will (hopefully) process back to X?[Coom87]

I think it is because people's interaction with language (not via language) is largely unconscious. We rarely think "the next preposition is important, so I'm going to say it louder and slower"; we don't think "I've said everything that relates to that topic sentence, so it's time for a new paragraph"; nor even "'Dr' is an abbreviation, so I need a period". Expertise has been defined as the reduction of more and more actions to unconsciousness -- that's how we (almost always) walk and/or chew gum. Our understanding of language is often similarly tacit. As Dreyfuss and Dreyfuss put it[Drey05, p. 788), If one asks an expert for the rules he or she is using, one will, in effect, force the expert to regress to the level of a beginner and state the rules learned in school. Thus, instead of using rules he or she no longer remembers, as the knowledge engineers suppose, the expert is forced to remember rules he or she no longer uses.

The act of markup, whether automated or manual, seems similar: we know a paragraph (or emphasis, or lacuna) when we see it, just as we know an obstacle on the sidewalk when we see it; but neither often makes it to consciousness.

Text analytics and markup are very similar tasks, though they tend to identify different things; it is rare for (say) a literary text project to mark up sentiment in novels, while it is equally rare for text anaytics to identify emphasis (although emphasis mught contribute to other features, such as topic weight).

Perhaps the most obvious place to start, beyond simple things like language-identification, is checking whether existing markup "makes sense", at a higher level of abstraction that XML schema languages -- a level closely involving the language rather than the text. The usual XML schema languages do little or nothing with text-content; with DTDs one can strip out *all* text content and the validity status of a document cannot change. With a text analytics system in place, however, it is possible to run tests related to the actual meaning of content. For example:

Text analytics is strongest at identifying abstract/conceptual features, when those features are not easily characterized by specific words or phrases, but emerge from larger linguistic context. The most blatant example is the perennial problem with non-language-aware search engines: negation. There are many ways to invert the sense (or sentiment) of a text, some simple buy many extremely subtle or complex. Tools that do not analyze syntax, clause roles, and the like can't distinguish texts that mean nearly the opposite of each other. Thus, at all levels from initial composition and markup, through validation and production, to search and retrieval, text analytics can enable new kinds of processes. Perhaps as such technology becomes widespread and is integrated into our daily workflows, it may help us to reach more deeply into the content of our texts.

Little has been published on the use of text analytics in direct relation to markup, although text analytics tools often use XML extensively, particularly for the representation of their results. However, TA has the potential to contribute significantly to our ability to validate exactly those aspects of documents, that markup does not help with: namely, what's going on down in the leaves.