Balisage Paper: Auto-Markup BenchMark: towards an industry-standard benchmark for evaluating automatic document markup

Overview

One of the barriers to entry for markup systems is the cost of getting documents from unstructured formats into structured formats, usually with a mix of programming and human authoring. AI in general, and Large Language Models in particular, may allow us to build systems that shift the balance further towards full automation.

Long term, the growth of large language models raises the question of whether marked up documentation will actually decline in relevance. The main argument for that position is that markup exists to help machines understand complex human languages. If the machines understand human documents as well as humans do, perhaps markup will no longer be necessary, just as it was not necessary when human scribes would transfer documents between contexts before markup was invented. The notations humans developed for their own use (italics, superscript, subscript) will presumably be sufficient for “truly intelligent” machines, as well.

In the medium term, however, machine inference is still fairly expensive, slow and not entirely reliable. Programmatic and deterministic-declarative systems will still need structured inputs for many years or decades. Automated markup might play the same role in making data available to structured transformation systems and stylesheets that OCR or TTS does in making text available to digital computers. Automated markup can also generate a “first draft” for human authors to perfect.

People have implemented various forms of automatic markup for several decades, but it seems that there is no standardization of how to evaluate them, and therefore no clear sense of what represents State Of The Art, and whether we are actually making progress. This paper describes an early version of such a benchmark created by a team of researchers collaborating as part of the RoundtableML community and releasing tools and datasets through Github.

Prior Work

Researchers have applied machine learning techniques to the Automatic Markup problem since at least 2002 [Shazia et al., 2021]. More recently, some organizations, such as Docugami [Paoli, 2019] and Innodata have turned this into a commercial enterprise. So far, however, there has been no standardized way to compare these offerings to each other or demonstrate progress in the State Of The Art.

The Role of AI Benchmarks

“Are we there yet”? Are we even making progress? An AI sub-field lacking metrics is like a vehicle without a speedometer or odometer.

As described in Liao et al., 2021:

These stories illustrate the influential role of benchmarks in the AI research community, pushing the boundaries of what is possible and fostering innovation across various domains.

Creating an AI benchmark for the understanding and generation of structured documents and markup languages, would have several benefits:

The ideal situation would be if OpenAI, Google, Anthropic and other industry leaders would adopt our benchmarks as goals in the training of their Foundation Models, which would mean that GPT-5, Claude 2, etc. might be trained with a deep understanding of markup and schemas.

The Project

An international group of industry engineers and academics from the RoundtableML community have come together and initiated this research direction. Our work is open source and we look forward to collaboration with others.

We have undertaken the following steps, and this document represents a checkpoint in our progress.

This paper discusses our progress on Steps 1 through 5. As indicated by the emojis, much of this project plan was formulated in collaboration with ChatGPT!

What is Auto-Markup?

We define Auto-Markup as an automated process that can consume documents and produce structured XML, or equivalent, markup in an autonomous or mostly autonomous fashion.

Given:

Generate:

We call these tools Auto-Markup engines and we anticipate a few categories of them:

We anticipate that just as with tools that translate between human languages, different Auto-Markup tools will vary in their performance based on the input/output pair. For example, a tool that is specialized in HTML to DITA might do a poor job converting Plain Text to Docbook.

Building specific benchmarks for every variant described above would be a monumental task, so we assume that the model has already been trained, prompted or otherwise taught how to deal with specific input and output formats and we do not include the efficiency of training or prompting as part of the benchmark. This is similar to how human language translation metrics are used.

Use Cases

We identified several user stories and use cases to help us ground our thinking. Here are a couple of examples.

    Meeting Notes - 9 June 2023
    New Feature Discussion
    People: 
    Jane - Project Manager
    Tom - Developer
    Sarah - Designer
    Agenda:
    - Talk about new features
    - Finalize the design
    Discussion:
    1. Jane started the meeting.
    2. Tom suggested a new search bar - link: www.example.com/searchbar
    3. Sarah showed new design mockup - image at: C:/images/design.png
    4. Team agreed to work on search bar and new design.
    Next steps:
    - Tom to create a prototype.
    - Sarah to finalize designs.
    Meeting concluded at 3:00 PM.

<!DOCTYPE html>
<html>
  <head>
    <title>Meeting Notes - 9 June 2023</title>
  </head>
  <body>
    <h1>Meeting Notes - 9 June 2023</h1>
    <h2>New Feature Discussion</h2>
    <h3>People:</h3>
    <ul>
      <li>Jane - Project Manager</li>
      <li>Tom - Developer</li>
      <li>Sarah - Designer</li>
    </ul>
    <h3>Agenda:</h3>
    <ul>
      <li>Talk about new features</li>
      <li>Finalize the design</li>
    </ul>
    <h3>Discussion:</h3>
    <ol>
      <li>Jane started the meeting.</li>
      <li>Tom suggested a new search bar - <a href="http://www.example.com/searchbar">link</a></li>
      <li>Sarah showed new design mockup - image at: C:/images/design.png</li>
      <li>Team agreed to work on search bar and new design.</li>
    </ol>
    <h3>Next steps:</h3>
    <ul>
      <li>Tom to create a prototype.</li>
      <li>Sarah to finalize designs.</li>
    </ul>
    <p>Meeting concluded at 3:00 PM.</p>
  </body>
</html>

Title: Using the Heart Rate Monitor
Introduction:
The heart rate monitor helps doctors to accurately measure patients’ heart rates in real-time. 
Prerequisite: Ensure the device is sanitized before use.
Steps:
1. Turn on the device by pressing the power button.
2. Place the monitor on the patient’s finger.
3. Wait for the device to calibrate.
4. Read the heart rate on the device screen.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE task PUBLIC "-//OASIS//DTD DITA Task//EN" "task.dtd">
<task id="heart_rate_monitor">
  <title>Using the Heart Rate Monitor</title>
  <shortdesc>The heart rate monitor helps doctors to accurately measure patients’ heart rates in real-time.</shortdesc> 
  <taskbody>
    <prereq>Ensure the device is sanitized before use.</prereq>
    <steps>
      <step><cmd>Turn on the device by pressing the power button.</cmd></step>
      <step><cmd>Place the monitor on the patient’s finger.</cmd></step>
      <step><cmd>Wait for the device to calibrate.</cmd></step>
      <step><cmd>Read the heart rate on the device screen.</cmd></step>
    </steps>
  </taskbody>
</task>

    Auto-Markup BenchMark: towards an industry-standard benchmark for Evaluating Automatic Document Markup

    Abstract:

    Large language models (LLMs) have greatly advanced in comprehending and generating human language. Markup remains crucial for structured inputs in publishing systems due to the high costs and unreliability of machine inference. Automated markup could bridge AI systems and unstructured text systems with structured publishing software by making it possible to move documents into XML at much lower cost. Currently, there is no standard way to evaluate these automatic markup systems. Benchmarks are always vital in AI development as they provide standardized datasets and evaluation metrics. Establishing a benchmark for the understanding and generation of structured documents and markup languages can drive innovation, standardize evaluations, identify algorithm strengths and weaknesses, clarify the State of the Art and foster interdisciplinary collaborations. This paper introduces an early version of a benchmark for this purpose.

    Overview

    In recent years, large language models (LLMs) and generative text technologies have made significant strides in natural language processing, yielding impressive results in understanding and generating human language.

    In the very long run, …

Conceptual Challenges

There are many challenges that arise in defining a metric for such a fuzzy task. If “correctness” were well-defined and clear, we probably would not need AI to help with it: we could simply use normal imperative code.

Here are some of the subjective issues we must address:

Ambiguity

A key challenge is the question of ambiguous labeling. What if two different tags are equally valid for a text run, or if one tool decides to insert an optional attribute that another might ignore? Options we explored are:

Human evaluation (rating) is another option, although it is too expensive to scale to the largest systems.

To a certain extent, this challenge can be addressed through normalization. For example, whitespace normalization is an obvious way to reduce the space of potential answers. One could also decide that certain tags are similar enough that they can be treated as equivalent in this context, although this requires schema-specific tweaking of the metrics, which introduces another point of subjectivity and perhaps disagreement.

Supplying multiple “correct” reference documents is another standard way of dealing with this issue and is implemented in our tooling.

Scale Challenges

Running metrics on large documents presents two main challenges:

The simpler one is that many metrics depend on quadratic algorithms which scale poorly with document size.

The more subtle one is another kind of exponential explosion. If there are M legitimate different ways to mark up the first section of a document and N ways for the second, as well as O, P and Q ways for the third, fourth and fifth, then the number of possible representations for the document is M*N*O*P*Q. Shorter documents are less prone to this explosion of alternatives.

For now, we have focused our attention on shorter documents.

Project Deliverables

Our project consisted of building a suite of open-source tools that can be used in the evaluation and training of markup engines:

In order to make the project tractable, we have scoped it to a few kinds of inputs and outputs.

For our input we use plain text, because it is the most universal format that can be extracted from almost any other format. By “plain text” we mean the kinds of texts that humans might write in a text editor for the consumption of other humans rather than tools. We call this variant of Plain Text “Messy Markdown” because it is similar to Markdown (in fact it was the inspiration for Markdown). Messy Markdown is necessarily much more fuzzy, broad and mutable than “real” Markdown. After all, the process of converting real, strict, Markdown into HTML is well-understood. It’s the messy output of copy and paste to Notepad, or direct typing into notepad, which requires AI.

For our output we use HTML and DITA to take advantage of the varying knowledge sets of project participants.

Adding HTML as an input format is a logical next step in our project:

Project Scope Restrictions

In order to keep the project tractable, and in recognition that Auto-Markup Engines need to walk before they run, there are certain aspects of the problem which we do not intend to address in the current project:

XML-centric AI Datasets

Our open source team has started the process of constructing a massive dataset of open source XML documents suitable for both training and evaluating XML AIs. It is frequently easier to generate an input/output pair for evaluation by starting with the output (e.g. DITA) because it is simple to generate plain text from DITA whereas the opposite is hard (until Auto-Markup is mainstream and reliable).

Our dataset consists of a mix of newly-written documents and documents repurposed from other projects.

The repurposed portion consists of open source documents from Github via The Stack:

We have filtered out non-document-oriented XML such as configuration files and Ant build files.

The code for assembling this data is at https://github.com/prescod/the-xml-document-stack

Some of the properties we will use to manage the dataset as it grows are:

The project has not yet organized the dataset meaningfully. When it does, the dataset should be segmented into three parts:

Metrics: Analogs from other fields

There are two fields of Artificial Intelligence that seem similar to the Auto-Markup problem. Studying these fields allows us to transfer decades of learning to our own problem.

The first is Named Entity Recognition, the field of Natural Language Processing that categorizes phrases in text. In our case, the entities would be identified so that they can be marked up.

The other is Machine Translation, wherein text is translated from one human language to another. In our case, the languages are “Unstructured Text” (or some more specific variant) and “Structured XML” (in some specific vocabulary).

Rather than think of XML markup as an entity recognition task, we could instead think of it as a generative text task, akin to translating between languages or summarizing text: generating a new document from the old one.

Viewed this way, as a content generation or transformation problem, reference metrics could be repurposed from quantitative scores used to assess machine translations and summarization:

BLEU (Bilingual Evaluation Understudy):

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)):

Translation Edit Rate:

In code:

    ter = number_of_edits / length_of_reference_text_in_tokens

Levenshtein distance:

Trade-Offs to Consider

None of these metrics is completely perfect even within their own domains but especially when imported into the domain of Auto-Markup.

When modeled as a classification problem, Auto-Markup for most commonly used languages is heavily class imbalanced. Certain tags happen much more often than others (those in the “long tail”).

Therefore, metrics such as accuracy and Macro-F1 can lead to unrealistically optimistic views of model performance. Weighted F1 scores adapt to this limitation by biasing by class frequency, but this approach can be unrealistically pessimistic if long-tail tags are less important when evaluating the overall performance of the system.

Another challenge we encounter when modeling auto-markup as a classification problem is the existence of open-set attributes such as the href attribute of the a tag and the src attribute of the img tag in HTML. These attributes cannot be modeled using closed-set classification in a straightforward way (although classification error could be augmented with use of some secondary metric, such as character-wise Levenshtein distance).

The limitations of TER, BLEU and ROUGE depend on what we intend them to measure.

TER, BLEU and ROUGE are designed to work at the level of n-grams or complete entities; since most tags are singular entities, if you sum up the discrete measurements for each tag, these metrics will behave much the same as accuracy when applied to single words.

For example: If you are evaluating the BLEU score of a one-word sentence against a reference one-word sentence using unigrams (single words), then the BLEU score would be either one if the words match (i.e., the generated word is the same as the reference word) or zero if they do not match.

If the system output and the reference are the same word, no edits are required, so the TER would be zero (indicating a perfect match). If the words are different, one substitution is required, so the TER would be one (one edit / one word = 1).

If we use these metrics naively at the level of the document, then the metric’s mass will concentrate in the text (which is supposed to be copied from one document to another) rather than the markup (which is what we aim to measure). And yet, we cannot simply ignore the text, because there is no guarantee that the engine (especially an LLM) will not modify it in the course of generating the markup.

A prosaic concern is implementation efficiency. Many of these algorithms are designed for comparing sentences to other sentences and their implementations are quadratic or even cubic in time complexity. When applied to big documents, they can grind to a halt. This is unfortunate, because models themselves may have poor scaling behaviors which we would like to probe with our tests.

Finally, the efficacy of our auto-markup metric will depend upon the tokenization strategy we use. Because markup tags are not formatted like words, typical tokenization strategies like byte-pair encoding will break them up into small blocks of characters. But if we use TER on partial-tag tokens, we will wind up more heavily penalizing some tags than others, depending on how they are tokenized. For instance, if the tag <img> is tokenized as (<) (im) (g) (>), and GPT produces instead the tag <a>, which is then tokenized (<) (a) (>), this mistake will be penalized more heavily than if GPT produces a non-existent tag such as <ims>, tokenized (<) (im) (s) (>).

One Proposal: XATER = The XML Translation Edit Rate

At the time of publishing, the Metric that we have used most often is a custom variant of Translation Edit Rate. This is a form of Translation Edit Rate that processes the output of a custom tokenizer. The XATER tokenizer normalizes unambiguously meaningless or likely-meaningless changes in markup.

In particular, our tokenizer recognizes the following types of tokens:

The handling of words is configurable: if you split text nodes on words, then markup engines that leave words alone get high marks and ones that mangle the words get low marks. But given that the default situation is that most auto-markup systems do NOT mangle words, this causes a kind of “grade inflation”.

XATER uses a different scale than TER. TER scores are normally in the range of 0 to 1 with 0 being better (no edits) and 1 being worse (virtually everything had to change). If the reference and hypothesis documents are extremely different in length then TER can even be greater than 1.

XATER flips the scale and shifts it to a percentage so that the best score is 100 and in extreme cases of mismatched length it can be a negative number.

The XATER calculation is merely:

    return 100 - pyter.ter(input.hypothesis_tokens, input.reference_tokens) * 100

The Tokenizer is a SAX ContentHandler containing about 30 lines of code.

    ** How to Start the Calculator App **
    Here you will find information on how to start the Calculator app on your mobile phone.
    Prerequisite: To use the calculator app, you need to unlock your phone first.
    Steps:
    1.  Navigate to the Calculator application icon.
    2.  Tap the icon.
    The application starts. You are now ready to calculate with it.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE task PUBLIC "-//OASIS//DTD DITA Task//EN" "task.dtd">
<task id="start-the-calculator">
    <title>How to Start the Calculator App</title>
    <shortdesc>Here you will find information on how to start the Calculator app on your mobile phone.</shortdesc>
    <taskbody>
        <prereq>
            <p>To use the calculator app, you need to unlock your phone first.</p>
        </prereq>
        <steps>
            <step>
                <cmd>Navigate to the Calculator application icon.</cmd>
            </step>
            <step>
                <cmd>Tap the icon.</cmd>
            </step>
        </steps>
        <result>
            <p>The application starts. You are now ready to calculate with it.</p>
        </result>
    </taskbody>
</task>

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE task PUBLIC "-//OASIS//DTD DITA Task//EN" "task.dtd">
<task id="start-calculator-app">
    <title>How to Start the Calculator App</title>
    <shortdesc>Here you will find information on how to start the Calculator app on your mobile phone.</shortdesc>
    <taskbody>
        <prereq>To use the calculator app, you need to unlock your phone first.</prereq>
        <steps>
            <step><cmd>Navigate to the Calculator application icon.</cmd></step>
            <step><cmd>Tap the icon.</cmd></step>
        </steps>
        <result>The application starts. You are now ready to calculate with it.</result>
    </taskbody>
</task>

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE topic PUBLIC "-//OASIS//DTD DITA Topic//EN" "topic.dtd">
<topic id="how-to-start-calculator-app">
    <title>How to Start the Calculator App</title>
    <shortdesc>Here you will find information on how to start the Calculator app on your mobile phone.</shortdesc>
    <body>
        <section>
            <title>Prerequisite</title>
            <p>To use the calculator app, you need to unlock your phone first.</p>
        </section>
        <section>
            <title>Steps</title>
            <ol>
                <li>
                    <p>Navigate to the Calculator application icon.</p>
                </li>
                <li>
                    <p>Tap the icon.</p>
                </li>
            </ol>
        </section>
        <section>
            <title>Result</title>
            <p>The application starts. You are now ready to calculate with it.</p>
        </section>
    </body>
</topic>

Companion Metric: Validation Error Metric

The Validation Error Metric is based on a normalized count of well-formedness and validation errors. It uses the popular open source LXML engine to do the counting in a royalty-free and portable manner.

The scoring is as such:

    # Calculate the score based on the percent of correct tags
    total_errors = num_wf_errors + num_dtd_errors
    good_elements = total_elements - total_errors
    # in case any elements generated multiple errors, 
    # we might have a negative number
    good_elements = max(0, good_elements)
    good_tags_ratio = good_elements / total_elements
    # flip it back to a zero to 100 score with better scores being better
    score = good_tags_ratio * 100

Roughly speaking it is the ratio of “correct” elements to “total” elements, but if each element generates more than one error then the algorithm could theoretically generate a score of less than zero. The code above limits the minimum score to zero, however.

Experimental Trial Run

We have implemented three Prototype Engines so far.

The dummy_automarkup engine serves to test the lower bound of auto-markup engines that properly preserve text but get all markup wrong.

The gpt3.5_am1_automarkup uses the gpt-3.5-turbo model through the OpenAI API and simplistic prompting to auto-markup simple DITA tasks. The gpt4_am1_automarkup uses the gpt-4 model.

Additionally, we assembled a test suite using sample Tasks from the DITA Toolkit. Each task was transformed into a variety of plain text formats, such as Markdown, Pandoc Plain Text, Emacs OrgMode and several variants of Messy Markdown, as described in section “Project Scope Restrictions”. A total of 24 DITA tasks were represented in 288 plain-textish input files. The goal of each Auto-Markup engine was to re-create the DITA task given only a single plain text file.

Our three simplistic prototypes are all less than 60 lines of code each and do not represent the State of the Art at all. Nevertheless, we can hypothesize that if they behave as we expect them to, and the metrics do as well, then gpt4_am1_automarkup should get a better score than gpt3.5_am1_automarkup which should in turn beat dummy_automarkup.

This is in fact what we see in the scores:

As a reminder, xater_metric tests whether the engine matches a reference text, and validation_error_metric tests whether it generates well-formed and valid XML.

The GPT 4-based engine did a reasonable job, with a reference-matching score of 70.56% and 99.61% for DTD-validity. As an example of its flaws, it did not know how to mark images up, because its extremely limited training data did not have images.

Another failure mode is unique to Generative Large Language Models. When presented with the following ambiguous Task, the LLMs may sometimes 'hallucinate' or generate the 'missing' content (the steps)!

Workbench.txt:

    # Organizing the workbench and tools
    Keep your garage neat by organizing the things you keep in the garage.

In one case, the engine garnered a negative score, because it generated this text for the above:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE task PUBLIC "-//OASIS//DTD DITA Task//EN" "task.dtd">
<task id="workbench-organization">
    <title>Organizing the workbench and tools</title>
    <shortdesc>Keep your garage neat by organizing the things you keep in the garage.</shortdesc>
    <taskbody>
        <context>Keeping your garage organized not only makes it easier to find what you need, but it also creates a safer work environment.</context>
        <steps>
            <step><cmd>Sort all items in your garage into categories.</cmd></step>
            <step><cmd>Decide on a place for each category on your workbench or in your tool cabinet.</cmd></step>
            <step><cmd>Place each item in its designated spot.</cmd></step>
        </steps>
        <result>Your workbench and tools are now organized!</result>
        <postreq>Make sure to return each item to its designated spot after use to maintain organization.</postreq>
    </taskbody>
</task>

This demonstrates that a mere 60 lines of code are far from sufficient to fully solve the auto-markup problem. With investment, we can overcome these problems by training (“fine-tuning”) specialized language models. This training was outside of the scope of this project, but we laid the groundwork for it by curating large, complex datasets. A future paper will show that we can improve these scores using those techniques.

Future Project Governance

Although the project initially began as a volunteer-run open-source initiative, its significance might warrant institutional support from organizations like OASIS, TEI, or a similar new establishment.

Funding may be needed over the long term. In addition to future research, the benchmarks should evolve over time:

Other Applications of Markup AI

We consider Auto-Markup as only a single application of a broader field of Markup AI. Other examples include:

We imagine new benchmarks or extensions to this benchmark to cover all of these use-cases in the future.

Conclusion

Auto-Markup is an exciting new direction for structured publishing workflows. We believe that empirical study of such Engines can accelerate progress in the development of them. Standardized metrics will allow us to compare tools and analyze their progress.

So far, our team has implemented some simple AutoMarkup engine prototypes to test our Metric and come to the conclusion that it does produce intuitive and useful evaluations.

We invite anyone interested in contributing to either join us at http://bit.ly/automarkup-metrics or to contact the lead researcher.