On beyond Invisible XML: parsing with implicit (and explicit) next-generation markupWe are grateful to Bethan Tovey-Walsh for her generous responses to some of the ideas we
        discuss here, for providing some of the IXML grammars that we used as we developed those
        ideas, and for her careful and constructively critical reading of an earlier version of this
        paper.

Ronald Haentjens Dekker; David J. Birnbaum; Bram Buitendijk

Abstract

In this paper we explore what Invisible-XML-like processing might look like when applied to what we refer to as next-generation features of markup, that is, structural properties of textual documents like overlap, discontinuity, and unordered content, which XML was not designed to represent at the level of the model. Like actual IXML, our Invisible-Next application is intended to recognize implicit, untagged markup and express it with explicit markup, but it differs from IXML by also supporting next-generation features that are not part of XML modeling and syntax.

We implement this Invisible-Next insertion of markup as a thin wrapper around the same data model and code base used for the Schema-Next validation that we described in Haentjens Dekker et al. 2025. Our exploration of Invisible-Next processing with next-generation features does not depend on that shared code base, but the appeal of coordinating the implementation of a validation tool with the implementation of a tool for inserting explicit markup foregrounds what the two operations have in common and suggests that each might contribute insights into efficient ways of implementing the other.

Research goals and questions

In the following paper we begin to explore the possibility of implementing Invisible-Next processing by using the same types of mildly context-sensitive grammars (MCSG) that we employed in Haentjens Dekker et al. 2025 for Schema-Next validation. We now attempt to answer three related questions:

Is it realistic to expect to use an MCSG to parse implicit markup that employs next-generation features and to validate those features during parsing?

As we illustrate below, we are able to parse implicit Markup-Next that includes features such as overlap, and add markup explicitly, as IXML does when it adds explicit XML markup to input that it ingests as a stream of characters. Our model is also able to recognize unordered (set) models, and although RELAX NG interleave is capable of modeling set content during validation, IXML, because it is context-free, can recognize sets only by pregenerating all possible combinations. This may be viable for attributes on an element, which typically are not unmanageably numerous, and where IXML does not require it because it already recognizes the nonrepeatability of attribute nodes, but it would fail with the potentially much larger models that correspond to, for example, unordered RDF statements or JSON key/value pairs.

The extent to which we are able to recognize implicit next-generation features requires further exploration and testing, but the examples below confirm that the goal is attainable at least for some types of documents and some types of implicit markup.
Is the incorporation of an MCSG into an Invisible-Next parsing and validation process likely to be a useful feature for end-users? More specifically, what useful next-generation structural features of documents can be supported once we employ an MCSG?

Our experiments below demonstrate that at least some types of implicit next-generation structuring can be identified and tagged with an Invisible-Next process. Insofar as adding explicit markup to documents motivated the development of IXML, we see no reason why the ability to add at least some types of next-generation markup should not prove equally useful.

We are mindful here of what Norm Tovey-Walsh writes about the utility of IXML:

Bear in mind as you go that the goal of Invisible XML is to produce visible XML. That’s not necessarily the final output form that you want. It’s text to XML, not text to DocBook 5.2 with embedded MathML and SVG. There are other, better tools to transform one flavor of XML into another, Invisible XML doesn’t have to do all the lifting.

— Norm Tovey-Walsh 2025
Can the use of a MCSG in Invisible-Next parsing reduce undesired grammatical ambiguity, where by undesired we mean ambiguity that is not included deliberately with the goal of capturing an ambiguity in the input that is intended to be modeled as ambiguity in the interpretation?^[2]

Our experiments below do not explore ambiguity in Invisible-Next grammars extensively. With that said, the experiments have to navigate whitespace-handling issues comparable to those that frequently arise with IXML, and the same strategies that help manage that type of potential ambiguity in IXML are available in our Invisible-Next application. Furthermore, as we show below, the new model successfully resolves the type of ambiguity in IXML that corresponds to what we regard as concurrency, although we also show that ambiguity is nonetheless possible even in the presence of concurrency. A thorough analysis of ambiguity and the mitigation thereof in Invisible-Next processing requires more attention that we are able to allocate in this initial exploration.

Introduction

Overview

A well-known limitation of XML markup is that it is not designed to represent at the model level structural properties of textual documents like overlap, discontinuity, unordered content, and others, which we refer to as next-generation features.^[3] In an XML environment these features can be represented with workarounds that must be managed during processing, and in previous Balisage presentations we explained why it might be better if inherent structural features of documents could, for reasons both philosophical and practical, be represented instead at the level of the model.^[4]

The current presentation explores what Invisible-XML-like processing might look like when applied to next-generation markup languages, such as LMNL (Tennison and Piez 2002), TexMECS (Huitfeldt and Sperberg-McQueen 2003), or TAGML (Haentjens Dekker et al. 2018).^[5] In the discussion below we use the following terms for next-generation markup and related technologies:

Markup-Next Markup languages that support such next-generation features as overlap (concurrence), discontinuity, unordered content, and partially ordered content. Examples include LMNL, TexMECS, and TAGML.
Schema-Next A RELAX-NG-like technology that can be used to support the validation of Markup-Next according to a user-supplied schema. One such example is Creole, which we discuss below.
Invisible-Next An IXML-like technology for adding Markup-Next to input according to a user-supplied grammar. Invisible-Next and its relationship to Schema-Next constitute the focus of this paper.

Three goals of invisible markup

Situations for Invisible-Next processing include the following:

An Invisible-Next grammar can be used to parse input without explicit next-generation markup and add that markup, much as IXML can be used to add explicit XML markup to character-stream input.^[6]
An Invisible-Next grammar can be used to parse input that has already been tagged according to Markup-Next without inserting additional markup or otherwise modifying the input. This operation uses Invisible-Next processing only as a recognizer, so that the only result that matters is whether the input document can or cannot be parsed successfully against a grammar.^[7] Much as IXML parsing has no built-in knowledge about XML markup that might already be present in an input document, and would therefore need to use grammar rules to distinguish XML markup in the input from content, Invisible-Next parsing of documents that already contain Markup-Next would need to read the input as a character stream and use grammar rules to recognize existing markup.
A processor might ingest an input document that is already (partially) tagged in Markup-Next and insert additional markup. Alternatively, a processor might ingest and parse a document that contains Markup-Next and emit a document that is marked up in other ways.^[8]

The third variant is a combination of the first two: ingesting and parsing a document that contains Markup-Next (scenario #2) and adding markup to it (scenario #1). Adding markup to an input document (the first and third examples above) assumes that the input can be parsed successfully against a supplied grammar. The second example above makes no such assumption and, instead, seeks to learn whether a parse can or cannot succeed.^[9]

Experiments: Invisible markup, ambiguity, and concurrency

Added operators in Invisible-Next: interleave, concur, partition, unordered

Next-generation processing requires operators, in addition to those that we borrow from IXML, to deal with interleave (where we build on the RELAX NG use of & to represent interleave during validation), concurrency (overlap), partitions (content present in only a single hierarchy in an overlapping context), and sets (unordered content). In this report and in our implementation we use the following operators:

For concurrency the Schema-Next implementation in Haentjens Dekker et al. 2025 used the tilde character (~), adopted from Tennison 2007. Because tilde in Invisible XML Specification is defined as an exclusion operator (e.g., ~[",."] matches any single character that is not a comma or a dot), in our Invisible-Next grammar notation we use a double tilde (~~) for concurrency.^[10]
For unordered (set) content in the RHS of a production we use parentheses preceded by an ampersand, e.g., the pattern &(A, B, C) matches exactly one instance each of the three nonterminals in any order in the input. In our Schema-Next application we followed Creole and RELAX NG and used an ampersand between terms for that purpose.^[11]
We also use an ampersand on the LHS of a production to indicate that the output should use markup for unordered content, such as the tags <e|| and ||e> (instead of the normal tags <e| and |e>) in TexMECS.^[12]

The principal container unit in Haentjens Dekker et al. 2025 is an element, and elements are subject to possible concurrency (overlap). We followed Tennison 2007 in recognizing also a need for a container unit that would suspend overlap, until the end of the group, in an otherwise overlapping context, which Creole called an element. The term element was not available for this purpose for us because it was already used for our primary container unit, so we represented a container unit that is not subject to concurrency with the keyword PARTITION, with no operator. In our Invisible-Next grammar a nonterminal that represents a partition is preceded by a double caret (^^).^[13]

We do not advocate for the specific notations we adopt here for these operators. Our options are constrained by established and sometimes competing interpretations of operator characters in RELAX NG, Creole, and IXML, which makes it difficult to rely on user familiarity with the notational conventions of those other specifications. We leave open the question of whether alternative operator characters might ultimately prove more intuitive or appropriate than those that we employ here. For what it’s worth, the operators in question appear only in the human-facing Schema-Next schema or Invisible-Next grammar, and not in the compiled grammar, where, as we write in Haentjens Dekker et al. 2025, they are replaced by words like INTERLEAVE and CONCUR.^[14]

Experiment 1: Hamlet

IXML is able to parse certain types of overlap, which it models as ambiguity. For example, assume we have the following excerpt from Shakespeare’s Hamlet:

GERTRUDE
Good Hamlet, cast thy nighted colour off,
And let thine eye look like a friend on Denmark.
Do not for ever with thy vailed lids
Seek for thy noble father in the dust:
Thou know'st 'tis common; all that lives must die,
Passing through nature to eternity.
HAMLET
Ay, madam, it is common.
GERTRUDE
If it be,
Why seems it so particular with thee?

These three speeches contain eight lines of iambic pentameter, and the detail that interests us is that the seventh metrical line is split over two speakers, where Hamlet speaks seven syllables and Gertrude speaks three.^[15] This excerpt illustrates both overlap and discontinuity. Specifically, we might model this text as follows:

A fully tessellated speech hierarchy is expressed a sequence of speeches that include every newline-delimited input line. Each speech begins with a speaker, followed by one or more newline-delimited input lines that contain the spoken words.^[16]
There is a metrical hierarchy represented by a sequence of metrical lines, each of which is connected to one or more newline-delimited input lines. Input lines that identify speakers are not part of this hierarchy because the speaker names are not part of the metrical structure of the text. In this excerpt, each metrical line contains exactly one newline-delimited input line except for the metrical line that is split across two speeches, described above.

We explore ways of visualizing this relationship in detail below, but one visualization of the structure described above is:

Figure 1: Face-to-face dual-tree layout

Our analysis of this text, and the data structure we construct on the basis of that analysis, differs from some traditional perspectives on overlap in the following related ways:

The overlapping hierarchies are not required to be fully tessellated. In a tessellated structure, a) the beginning of the first node on a tier coincides with the first node under it, b) the end of one node on that tier is followed immediately by the beginning of the next, and c) the end of the last node on the tier coincides with the last node to which it connects. This means, in this example, that every leaf node (newline-separated input line), in order, would be part of a node on the tier above. From this perspective, the speech hierarchy in our model is fully tessellated. The metrical line hierarchy is not fully tessellated because the newline-delimited input lines that correspond to speaker names are not part of that hierarchy. Nodes that are part of one hierarchy and not part of the other correspond to partitions, which are distinct, in our adaptation of Creole, from elements, which are part of both hierarchies.^[17]
Partitions are fundamental to modeling discontinuity of the type illustrated by this example, where speakers play no role in the metrical structure of the text. Similarly, in the classic thought Alice example (Sperberg-McQueen and Huitfeldt 2008a) a quotation is interrupted by plain text, that is, by a text node. The text reads:

Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do: once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, and what is the use of a book, thought Alice without pictures or conversation?

The analysis in Sperberg-McQueen and Huitfeldt 2008a communicates that the narrative interpolation is present, but is not part of the quoted text. Insofar as the speaker name in our Hamlet example can be understood, on the metrical-line level, as metadata about the lines, it might reasonably be said not to be part of the content of the metrical hierarchy at all. The idea that concurrence (that is, overlap) requires that all text nodes be part of all hierarchies may, then, be natural for some overlap examples, such as page-and-lines vs chapters-and-paragraphs, but we find it less intuitive in cases involving discontinuity.^[18]
We have not found any discussion of partitions in the scholarly literature after the concept was introduced in Tennison 2007, but it is fundamental to modeling the type of overlap that occurs in this example. At the same time we might regard a situation where two speakers interrupt each other as representing self-discontinuity, by which we mean that an element of a particular type (e.g., <speech>) overlaps with another element of the same type within the same hierarchy, with the result that pieces of one speech interrupt pieces of a different speech. This invites us to ask whether the concept of partitions, introduced originally in the context of overlap, should be recognized also in the context of interleave.^[19]

Hamlet in separate IXML grammars

As we ingest and parse the plain text input, we identify the transition between speeches by recognizing that each speech begins with a speaker name and each speech ends before the next speaker name or at the end of the document. We identify the transition between metrical lines by recognizing that each metrical line begins at the beginning of the document or after another metrical line and ends as soon as we have accumulated ten or eleven syllables with a (whitespace-separated) word before a newline character.^[20] Because English syllabification is not easily deduced from English orthography, we assume a pre-processing NLP operation that prepends a running syllable count to each word token (speech only, not counting speaker names), and resets after the total has reached 10 or 11.^[21] The output of that preprocessing step looks like:

GERTRUDE
1Good 3Hamlet, 4cast 5thy 7nighted 9colour 10off,
1And 2let 3thine 4eye 5look 6like 7a 8friend 9on 11Denmark.
1Do 2not 3for 5ever 6with 7thy 9vailed 10lids
1Seek 2for 3thy 5noble 7father 8in 9the 10dust:
1Thou 2know'st 3'tis 5common; 6all 7that 8lives 9must 10die,
2Passing 3through 5nature 6to 10eternity.
HAMLET
1Ay, 3madam, 4it 5is 7common.
GERTRUDE
8If 9it 10be,
1Why 2seems 3it 4so 8particular 9with 10thee?

Our goal is to tag both the speeches (there are three: by Gertrude, then Hamlet, and then Gertrude again) and the metrical lines (there are eight: six by Gertrude, then one shared by Hamlet and Gertrude, and then one more by Gertrude). Because IXML is intended to emit XML, our IXML approach to the task treats both of the overlapping hierarchies as fully tessellated (that is, without modeling the speaker names as an invisible partition in the metrical-line hierarchy). Under this approach, the document could be tagged in XML as a sequence of uninterrupted (except by insignificant whitespace) speeches, which must have, as a first child, a speaker-name element, or, alternatively, as an uninterrupted (except by insignificant whitespace) sequence of metrical lines, which may have speaker-name children. These can be modeled with ambiguous IXML as follows:

For the sequence of metrical lines, the IXML grammar:

hamlet: m_line++eol.
m_line: (token; speaker)+, f_token.
-f_token: ddigit, [L; P]+.
-token: digit, [L; P]+, (sp; eol).
speaker: ["A"-"Z"]+, eol.
-digit: -["0"-"9"].
-ddigit: digit, digit.
-eol: -#a.
-sp: " ".

produces the following unambiguous output:

<hamlet>
   <m_line>
      <speaker>GERTRUDE</speaker>Good Hamlet, cast thy nighted colour off,</m_line>
   <m_line>And let thine eye look like a friend on Denmark.</m_line>
   <m_line>Do not for ever with thy vailed lids</m_line>
   <m_line>Seek for thy noble father in the dust:</m_line>
   <m_line>Thou know'st 'tis common; all that lives must die,</m_line>
   <m_line>Passing through nature to eternity.</m_line>
   <m_line>
      <speaker>HAMLET</speaker>Ay, madam, it is common.<speaker>GERTRUDE</speaker>If it be,</m_line>
   <m_line>Why seems it so particular with thee?</m_line>
</hamlet>

For the sequence of speeches, the IXML grammar:

hamlet: speech++eol.
speech: speaker, eol, line++eol.
speaker: ["A"-"Z"]+.
line: word++sp.
-word: digits, [L; P]+.
-digits: -["0"-"9"]+.
-eol: -#a.
-sp: " ".

produces the following unambiguous output:

<hamlet>
   <speech>
      <speaker>GERTRUDE</speaker>
      <line>Good Hamlet, cast thy nighted colour off,</line>
      <line>And let thine eye look like a friend on Denmark.</line>
      <line>Do not for ever with thy vailed lids</line>
      <line>Seek for thy noble father in the dust:</line>
      <line>Thou know'st 'tis common; all that lives must die,</line>
      <line>Passing through nature to eternity.</line>
   </speech>
   <speech>
      <speaker>HAMLET</speaker>
      <line>Ay, madam, it is common.</line>
   </speech>
   <speech>
      <speaker>GERTRUDE</speaker>
      <line>If it be,</line>
      <line>Why seems it so particular with thee?</line>
   </speech>
</hamlet>

Hamlet in one IXML grammar using alternative operator

We can merge the two IXML grammars by changing the first rule (for hamlet) to allow either one or the other, that is, to separate the two with an or connector (;).^[22] Below is an IXML grammar that does that, which incorporates the following changes:

A shared section creates a top-level rule that matches either of the two hierarchies and unifies shared rules for end-of-line and space.
The models for lines were different in the two independent grammars, where the one for metrical lines is m_line: (token; m_speaker)+,f_token. and the one for speeches is line: word++sp. We use renaming to output <line> for both.
The models for speaker names were different in the two independent grammars, where the one for metrical lines is m_speaker: ["A"-"Z"]+,eol. and the one for speeches is speaker: ["A"-"Z"]+. We use renaming to output <speaker> for both.

ixml version "1.1".
{ shared }
hamlet: m_line++eol; speech++eol.
-eol: -#a.
-sp: " ".

{ metrical only }
m_line>line: (token; m_speaker)+, f_token.
-f_token: ddigit, [L; P]+.
-token: digit, [L; P]+, (sp; eol).
m_speaker>speaker: ["A"-"Z"]+, eol.
-digit: -["0"-"9"].
-ddigit: digit, digit.

{ speech only }
speech: speaker, eol, line++eol.
speaker: ["A"-"Z"]+.
line: word++sp.
-word: digits, [L; P]+.
-digits: -["0"-"9"]+.

This grammar is ambiguous, with two possible parses, which correspond to the outputs of the two individual grammars:

<ixml parses="2" totalParses="2">
  <hamlet xmlns:ixml='http://invisiblexml.org/NS' IXML:state='ambiguous'>
     <line>
        <speaker>GERTRUDE</speaker>Good Hamlet, cast thy nighted colour off,</line>
     <line>And let thine eye look like a friend on Denmark.</line>
     <line>Do not for ever with thy vailed lids</line>
     <line>Seek for thy noble father in the dust:</line>
     <line>Thou know'st 'tis common; all that lives must die,</line>
     <line>Passing through nature to eternity.</line>
     <line>
        <speaker>HAMLET</speaker>Ay, madam, it is common.
        <speaker>GERTRUDE</speaker>If it be,</line>
     <line>Why seems it so particular with thee?</line>
  </hamlet>
  <hamlet xmlns:ixml='http://invisiblexml.org/NS' IXML:state='ambiguous'>
     <speech>
        <speaker>GERTRUDE</speaker>
        <line>Good Hamlet, cast thy nighted colour off,</line>
        <line>And let thine eye look like a friend on Denmark.</line>
        <line>Do not for ever with thy vailed lids</line>
        <line>Seek for thy noble father in the dust:</line>
        <line>Thou know'st 'tis common; all that lives must die,</line>
        <line>Passing through nature to eternity.</line>
     </speech>
     <speech>
        <speaker>HAMLET</speaker>
        <line>Ay, madam, it is common.</line>
     </speech>
     <speech>
        <speaker>GERTRUDE</speaker>
        <line>If it be,</line>
        <line>Why seems it so particular with thee?</line>
     </speech>
  </hamlet>
</IXML>

This application of renaming is seductive because it is easy to implement and because the term line is commonly used with different meanings to refer to metrical lines in prosody and to orthographic lines in a printed script. At the same time, applying this type of renaming in an XML context associates the same element type with two different meanings, which impinges on the ability of the markup to make sameness and difference accessible in an intuitive way to automated processing. It is, to be sure, easy to preserve different names for the two types of lines, should that prove more useful down stream.

Hamlet in IXML with complete LMNL and TexMECS output

LMNL output

Our next task was to see whether IXML could create LMNL output that could contain overlapping markup in a single serialization. LMNL is, of course, able to represent overlap, but IXML is able to serialize only as XML. We work around that discrepancy in the following ways:

We create a root XML element around the LMNL markup so that the LMNL content will satisfy the requirements for well-formed XML. Any subsequent LMNL processing would need to ignore that XML wrapper element.
We suppress all XML tags and create LMNL tags using IXML insertions.

Here is an unambiguous IXML grammar that creates LMNL output:

hamlet: +#a, +"[hamlet}", +#a, first_speaker, body, +"{line]", +#a, +"{hamlet]", -#a?, +#a. 

{ First line precedes line traversal, open first line and first speech and process speaker 
  Transitions below depend on newline, and first line has no preceding newline
}
-first_speaker: +"[line}[speech}[speaker}", speaker, +"{speaker] ", no_eol. 
-body: soft_line; hard_line.

{ STATE MACHINE: each line transition signals one of four possible states based on end of line }
 
{ STATE 1: Line ending in a single-digit token continues metrical line with new speaker }
-soft_line: token_sp*, token_end, -#a, soft_speaker_change. 

{ STATES 2, 3, 4: Line ending in a double-digit f_token 
  STATE 2: new metrical line continues with same speaker 
  STATE 3: new metrical line changes speaker
  STATE 4: no new metrical line means end of file }
-hard_line: token_sp*, f_token_end, 
            (hard_continuation; hard_speaker_change; final_line). 

{ Inject a space explicitly between mid-line speech boundaries and continue }
-soft_speaker_change: +"{speech] [speech}[speaker}", speaker, +"{speaker] ", no_eol, body.

{ After a hard line ending insert line transition tags with newline between and continue }
-hard_continuation: +"{line]", #a, +"[line}", body. 

{ Remove space character between closing speech and line tags }
-hard_speaker_change: +"{speech]{line]", #a, +"[line}[speech}[speaker}", speaker, +"{speaker] ", no_eol, body. 

-final_line: +"{speech]". 

{ THREE TYPES OF METRICAL-LINE TOKENS, PLUS SPEAKER:
  token_sp: Mid-line tokens must be followed by a space 
  token_end: Line-final token that is not end of metrical line
  f_token_end: Line-final token that signals end of metrical line
  speaker: all upper case }
-token_sp: digit, [L; P]+, sp. 
-token_end: digit, [L; P]+. 
-f_token_end: ddigit, [L; P]+. 
-speaker: ["A"-"Z"]+. 

{ Misc }
-digit: -["0"-"9"]. 
-ddigit: digit, digit. 
-no_eol: -#a. 
-sp: " ".

The output is:

<hamlet>
[hamlet}
[line}[speech}[speaker}GERTRUDE{speaker] Good Hamlet, cast thy nighted colour off,{line]
[line}And let thine eye look like a friend on Denmark.{line]
[line}Do not for ever with thy vailed lids{line]
[line}Seek for thy noble father in the dust:{line]
[line}Thou know'st 'tis common; all that lives must die,{line]
[line}Passing through nature to eternity.{speech]{line]
[line}[speech}[speaker}HAMLET{speaker] Ay, madam, it is common.{speech] [speech}[speaker}GERTRUDE{speaker] If it be,{line]
[line}Why seems it so particular with thee?{speech]{line]
{hamlet]
</hamlet>

We note that this is not a general-purpose invisible LMNL processing model for at least the following reasons:

It works with MNML, a restricted subset of LMNL, and not with the full range of markup permitted by the original LMNL model.^[23]
It takes advantage of the simple and regular structure of the input, which has only five types of event (other than start and end): speaker, metrical-line-non-final token, metrical-line-final token, space, eol.^[24]
There is no self-overlap.
We have two independent hierarchies, which requires us to map out all possible interactions of the two: 1) same metrical line continues with new speaker, 2) new metrical line with same speaker, 3) new metrical line with different speaker, and 4) end of input. These combinations would increase rapidly if there were additional overlapping hierarchies.

A concurrency operator would allow us to describe the hierarchies separately and assert that they concur (that is, that they overlap), without our having to manage, in the grammar that we write, all possible states that result from the interactions of speech-related state transitions and metrical-line-related state transitions. Our IXML grammar with ambiguity, above, comes close to this model; the principal difference is that because IXML lacks a concurrency operator, we fall back on describing concurrency as ambiguity.^[25] As we discuss below, however, not all ambiguity can be modeled as concurrency, which means that when we build on IXML capabilities, below, by adding both a concurrency operator (and the ability to process it without generating ambiguity) and a non-hacky serialization capability, those enhancements do not preclude the construction of other types of ambiguous grammars that yield ambiguous parses.
Because IXML can emit only well-formed XML, our LMNL output is wrapped in an XML root element, so that from an XML perspective the root element has just a single child, which is a text node. This means that the output is not actually LMNL, and should we wish to embed this IXML operation within a pipeline, we would have to handle the XML root element.

Finally, the LMNL output does not represent discontinuity because we do not find a documented representation of discontinuity in xMNML.(xMNML grammar) The TexMECS output, below, represents discontinuity explicitly, using documented TexMECS markup features.

TexMECS output

The portions of our text-to-LMNL grammar that parse the input can be reused in a text-to-TexMECS grammar. The following grammar:

{ Run with: coffeepot -g:grammar-hamlet-texmecs.ixml -i:hamlet.txt | sed '1d;$d' | tr '{}' '<>' }
hamlet: +#a, +"{hamlet|", +#a, first_speaker, body, +#a, +"|speech}", +#a, +"|hamlet}", -#a?, +#a. 

{ First line precedes line traversal, open first line and first speech and process speaker 
  Transitions below depend on newline, and first line has no preceding newline
}
-first_speaker: +"{speech|", +#a, +"{speaker|", speaker, +"|speaker}", +#a, +"{line|",  no_eol. 
-body: soft_line; hard_line.

{ STATE MACHINE: each line transition signals one of four possible states based on end of line }
 
{ STATE 1: Line ending in a single-digit token continues metrical line with new speaker }
-soft_line: token_sp*, token_end,-#a, soft_speaker_change. 

{ STATES 2, 3, 4: Line ending in a double-digit f_token 
  STATE 2: new metrical line continues with same speaker 
  STATE 3: new metrical line changes speaker
  STATE 4: no new metrical line means end of file }
-hard_line: token_sp*, f_token_end, 
            (hard_continuation; hard_speaker_change; final_line). 

{ Inject a space explicitly between mid-line speech boundaries and continue }
-soft_speaker_change: +"|-line}", +#a, +"|speech}", +#a, +"{speech|", +#a, +"{speaker|", speaker, +"|speaker}", +#a, +"{+line|", no_eol, body.

{ After a hard line ending insert line transition tags with newline between and continue }
-hard_continuation: +"|line}",#a,+"{line|",body. 

{ Remove space character between closing speech and line tags }
-hard_speaker_change: +"|line}", #a, +"|speech}", +#a, +"{speech|", +#a, +"{speaker|", speaker, +"|speaker}", +#a, +"{line|", no_eol, body. 

-final_line: +"|line}". 

{ THREE TYPES OF METRICAL-LINE TOKENS, PLUS SPEAKER:
  token_sp: Mid-line tokens must be followed by a space 
  token_end: Line-final token that is not end of metrical line
  f_token_end: Line-final token that signals end of metrical line
  speaker: all upper case }
-token_sp: digit, [L;P]+, sp. 
-token_end: digit, [L;P]+. 
-f_token_end: ddigit, [L;P]+. 
-speaker: ["A"-"Z"]+. 

{ Misc }
-digit: -["0"-"9"]. 
-ddigit: digit, digit. 
-no_eol: -#a. 
-sp: " ".

when used according to the instruction in the IXML comment on the first line, emits:

<hamlet|
<speech|
<speaker|GERTRUDE|speaker>
<line|Good Hamlet, cast thy nighted colour off,|line>
<line|And let thine eye look like a friend on Denmark.|line>
<line|Do not for ever with thy vailed lids|line>
<line|Seek for thy noble father in the dust:|line>
<line|Thou know'st 'tis common; all that lives must die,|line>
<line|Passing through nature to eternity.|line>
|speech>
<speech|
<speaker|HAMLET|speaker>
<line|Ay, madam, it is common.|-line>
|speech>
<speech|
<speaker|GERTRUDE|speaker>
<+line|If it be,|line>
<line|Why seems it so particular with thee?|line>
|speech>
|hamlet>

Because IXML emits XML, we wrapped an XML <hamlet> root element around the entire TexMECS output, as we did with our LMNL output, above, and we used a sed operation to delete the first and last lines of the output of the IXML processor. TexMECS markup uses angle brackets, which IXML output would replace with character entities (< and >), and we found it simpler and more legible to use curly braces in place of angle brackets at the IXML stage and then replace them with a post-processing tr operation. The state logic for the IXML grammars that emit LMNL and TexMECS is the same; the differences pertain to output serialization, where we found it more natural to impose different wrapping and ordering priorities for tags in TexMECS (speech tags outside line tags, all tags on their own output lines except for line tags) and LMNL (metrical lines on a single output line, line tags outside speech tags). Tags that are associated with the same position in the input character stream are unordered according to the LMNL range algebra, and our output order in these cases prioritizes our understanding of human legibility and convenience. Because TexMECS employs distinct markup delimiters in cases of discontinuity, it seemed most legible to wrap speakers and lines (whether complete or partial) inside speech tags in our TexMECS output. Pretty-printing entails letting convenience and legibility, which are necessarily at least somewhat subjective, influence decisions involving informationally insignificant whitespace, and distinguishing significant vs insignificant whitespace ultimately merits careful attention, rigorous formulation, and explicit documentation. Our pretty-printing decisions in this experiment have not undergone that level of analysis, and we intend to return to the issue in future experiments.

Hamlet in Invisible-Next with `CONCUR` and TexMECS output

If we compare the complexity of the IXML grammar to emit TexMECS, above, with our Invisible-Next grammar with the same functionality, below, the differences are striking:

{ shared }
hamlet: speeches ~~ metrical.
-speeches: speech++eol.
-metrical: m_line++eol.
-eol: -#a.
-sp: #20.
-digit: -["0"-"9"].

{ speech only }
speech: speaker, line++eol.
^^speaker: ["A"-"Z"]+, eol.
-line: word++sp.
-word: digits, [L; P]+.
-digits: digit+.

{ metrical only }
m_line: token+, f_token.
-token: digit, [L; P]+, (sp; eol).
-f_token: ddigit, [L; P]+.
-ddigit: digit, digit.

We chose to emit TexMECS (rather than LMNL) for this part of the Hamlet experiment because TexMECS has well-documented support for representing discontinuity in markup (using the |-line> suspend tag and the <+line| resume tag). The Invisible-Next grammar uses new operators for concurrency/overlap (~~) and partitions (a ^^ mark at the beginning of the LHS), neither of which is part of IXML. The most striking differences between the IXML and Invisible-Next grammars, though, are that:

The IXML grammar needed to monitor the state of both hierarchies with the same rules, which led to complex productions based on those interactions. The Invisible-Next grammar looks much more like our earlier ambiguous IXML grammar, except that it is not ambiguous. The simplicity is possible because managing the interaction of the states has been moved out of the user-supplied grammar and into the MCSG-based parser.
Just as IXML grammars do not have to include serialization information other than marks and minus signs, since an IXML processor is expected to know how to create XML elements and attributes, Invisible-Next knows how to serialize next-generation markup (in this case, TexMECS). An Invisible-Next grammar does not need to contain the string-insertion work-arounds that we used to emit TexMECS markup from a IXML process.

Experiment 2: Ozymandias

We conducted a second experiment using Ozymandias, a poem by Percy Bysshe Shelley first published in 1818. We selected this example because Wendell Piez had tagged a version of it using LMNL markup,^[26] and we wanted to explore whether we could replicate his LMNL markup using Invisible-Next. Piez’s LMNL document reads:

[sonneteer [id}ozymandias{] [author}Percy Bysshe Shelley{] [title}Ozymandias{]}
[sonnet}[s}[octave}[quatrain}[line}[phr}I met a traveller from an antique land,{phr]{line]
[line}[phr}Who said —{phr] [phr}“Two vast and trunkless legs of stone{line]
[line}Stand in the desart....{phr][phr}Near them,{phr] [phr}on the sand,{phr]{line]
[line}[phr}Half sunk a shattered visage lies,{phr] [phr}whose frown,{phr]{line]{quatrain]
[quatrain}[line}[phr}And wrinkled lip,{phr] [phr}and sneer of cold command,{phr]{line]
[phr}[line}Tell that its sculptor well those passions read{line]
[line}Which yet survive,{phr] [phr}stamped on these lifeless things,{phr]{line]
[line}[phr}The hand that mocked them,{phr] [phr}and the heart that fed;{phr]{line]{quatrain]{octave]
[sestet}[tercet}[line}[phr}And on the pedestal,{phr] [phr}these words appear:{phr]{line]
[line}[phr}My name is Ozymandias,{phr] [phr}King of Kings,{phr]{line]
[line}[phr}Look on my Works,{phr] [phr}ye Mighty,{phr] [phr}and despair!{phr]{line]{tercet]{s]
[tercet}[line}[s}[phr}Nothing beside remains.{phr]{s] [s}[phr}Round the decay{line]
[line}Of that colossal Wreck,{phr] [phr}boundless and bare{line]
[line}The lone and level sands stretch far away.{phr]”{line]{s]{tercet]{sestet]{sonnet]{sonneteer]

To avoid being distracted by content not involved in creating overlapping markup we ignored the root element ([sonneteer} … {sonneteer]) and metadata (author and title information), and focused instead on the following two hierarchies:

Sonnet structure: sonnet (root), octave, quatrain, sestet, tercet, line
Linguistic structure: sonnet (root), s (sentence), phr (phrase), and q (quoted text).

To create plain-text input for our experiment we removed the first (metadata) line of Piez’s LMNL example and stripped out all LMNL markup. The result is:

I met a traveller from an antique land,
Who said — “Two vast and trunkless legs of stone
Stand in the desart....Near them, on the sand,
Half sunk a shattered visage lies, whose frown,
And wrinkled lip, and sneer of cold command,
Tell that its sculptor well those passions read
Which yet survive, stamped on these lifeless things,
The hand that mocked them, and the heart that fed;
And on the pedestal, these words appear:
My name is Ozymandias, King of Kings,
Look on my Works, ye Mighty, and despair!
Nothing beside remains. Round the decay
Of that colossal Wreck, boundless and bare
The lone and level sands stretch far away.”

Tagging quoted text programmatically is problematic with this input because there are two instances and they are represented differently in the plain text. One quotation opens on the second line of verse and closes at the end of the poem, and that quotation is demarcated by typographic (curly) quotation marks. There is also, though, an embedded quotation that consists of one sentence that occupies two full lines:

My name is Ozymandias, King of Kings,
Look on my Works, ye Mighty, and despair!

Piez does not tag either of the quotations in his LMNL text. Although our output in other respects seeks to replicate Piez’s LMNL tagging, we elected to replace the literal typographic quotation marks in the plain-text input with LMNL start- and end-tags. Like Piez, we do not tag the embedded quote; we regard this as a compromise because recognizing these two lines as quoted text requires analysis that might entail NLP processing that is not a realistic task for invisible-markup management.^[27]

Ozymandias in separate IXML grammars

The following IXML grammar adds markup for the sonnet structure (only) to the plain-text input:

{sonnet (root), octave, quatrain, sestet, tercet, line }
sonnet: octave, -#a, sestet.
octave: quatrain, -#a, quatrain.
quatrain: line, -#a, line, -#a, line, -#a, line.
sestet: tercet, -#a, tercet.
tercet: line, -#a, line, -#a, line.
line: ~[#a]+.

The unambiguous output is:

<sonnet>
   <octave>
      <quatrain>
         <line>I met a traveller from an antique land,</line>
         <line>Who said — “Two vast and trunkless legs of stone</line>
         <line>Stand in the desart....Near them, on the sand,</line>
         <line>Half sunk a shattered visage lies, whose frown,</line>
      </quatrain>
      <quatrain>
         <line>And wrinkled lip, and sneer of cold command,</line>
         <line>Tell that its sculptor well those passions read</line>
         <line>Which yet survive, stamped on these lifeless things,</line>
         <line>The hand that mocked them, and the heart that fed;</line>
      </quatrain>
   </octave>
   <sestet>
      <tercet>
         <line>And on the pedestal, these words appear:</line>
         <line>My name is Ozymandias, King of Kings,</line>
         <line>Look on my Works, ye Mighty, and despair!</line>
      </tercet>
      <tercet>
         <line>Nothing beside remains. Round the decay</line>
         <line>Of that colossal Wreck, boundless and bare</line>
         <line>The lone and level sands stretch far away.”</line>
      </tercet>
   </sestet>
</sonnet>

The following IXML grammar adds markup for the linguistic structure (only) to the plain-text input:^[28]

ixml version "1.1".
sonnet: (s; q)+.
s: (q; phrp)*, (q; phr, sPunc).
phr: (-" "; -#a)?, text.
phrp>phr: -" "?, text, pPunc, nows.
-sPunc: ["?!"]; "."+.
-pPunc: [",;:"]; " —".
-text: [L], ([L]; ws)+, [L].
q: qStart, s+, qEnd.
-qStart: -"“".
-qEnd: -"”".
-sp: " ".
-eol: #a.
-ws: sp; eol.
-noeol: sp; -#a.
-nosp: -" "; #a.
-nows: -" "; -#a.

The unambiguous output is:

<sonnet>
   <s>
      <phr>I met a traveller from an antique land,</phr>
      <phr>Who said —</phr>
      <q>
         <s>
            <phr>Two vast and trunkless legs of stone
Stand in the desart</phr>....</s>
         <s>
            <phr>Near them,</phr>
            <phr>on the sand,</phr>
            <phr>Half sunk a shattered visage lies,</phr>
            <phr>whose frown,</phr>
            <phr>And wrinkled lip,</phr>
            <phr>and sneer of cold command,</phr>
            <phr>Tell that its sculptor well those passions read
Which yet survive,</phr>
            <phr>stamped on these lifeless things,</phr>
            <phr>The hand that mocked them,</phr>
            <phr>and the heart that fed;</phr>
            <phr>And on the pedestal,</phr>
            <phr>these words appear:</phr>
            <phr>My name is Ozymandias,</phr>
            <phr>King of Kings,</phr>
            <phr>Look on my Works,</phr>
            <phr>ye Mighty,</phr>
            <phr>and despair</phr>!</s>
         <s>
            <phr>Nothing beside remains</phr>.</s>
         <s>
            <phr>Round the decay
Of that colossal Wreck,</phr>
            <phr>boundless and bare
The lone and level sands stretch far away</phr>.</s>
      </q>
   </s>
</sonnet>

Ozymandias in one IXML grammar using alternative operator

As was the case with our Hamlet experiment, above, we can merge the two IXML grammars for Ozymandias by changing the first rule (for sonnet) to allow either one pattern or the other, that is, to separate the two right-hand sides with an or connector (;). Below is an IXML grammar that does that:

ixml version "1.1".
{ Concurrent linguistic and poetic hierarchies }
{ shared }
sonnet: (s; q)+; octave, -#a, sestet.

{ linguistic structure: s, phr, q }
s: (q; phrp)*, (q; phr, sPunc).
phr: (-" "; -#a)?, text.
phrp>phr: -" "?, text, pPunc, nows.
-sPunc: ["?!"]; "."+.
-pPunc: [",;:"]; " —".
-text: [L], ([L]; ws)+, [L].
q: qStart, s+, qEnd.
-qStart: -"“".
-qEnd: -"”".
-sp: " ".
-eol: #a.
-ws: sp; eol.
-noeol: sp; -#a.
-nosp: -" "; #a.
-nows: -" "; -#a.

{ poetic structure: octave, quatrain, sestet, tercet, line }
octave: quatrain, -#a, quatrain.
quatrain: line, -#a, line, -#a, line, -#a, line.
sestet: tercet, -#a, tercet.
tercet: line, -#a, line, -#a, line.
line: ~[#a]+.

The unification was simpler with Ozymandias than with Hamlet because the only markup shared by the two hierarchies in Ozymandias is the root element. As expected, the output is ambiguous, with one parse for each of the hierarchies:

<?xml version="1.0"?>
<ixml parses="2" totalParses="2">
  <sonnet xmlns:ixml="http://invisiblexml.org/NS" ixml:state="ambiguous">
    <s>
      <phr>I met a traveller from an antique land,</phr>
      <phr>Who said &#x2014;</phr>
      <q>
        <s><phr>Two vast and trunkless legs of stone
Stand in the desart</phr>....</s>
        <s><phr>Near them,</phr><phr>on the sand,</phr><phr>Half sunk a shattered visage lies,</phr><phr>whose frown,</phr><phr>And wrinkled lip,</phr><phr>and sneer of cold command,</phr><phr>Tell that its sculptor well those passions read
Which yet survive,</phr><phr>stamped on these lifeless things,</phr><phr>The hand that mocked them,</phr><phr>and the heart that fed;</phr><phr>And on the pedestal,</phr><phr>these words appear:</phr><phr>My name is Ozymandias,</phr><phr>King of Kings,</phr><phr>Look on my Works,</phr><phr>ye Mighty,</phr><phr>and despair</phr>!</s>
        <s><phr>Nothing beside remains</phr>.</s>
        <s><phr>Round the decay
Of that colossal Wreck,</phr><phr>boundless and bare
The lone and level sands stretch far away</phr>.</s>
      </q>
    </s>
  </sonnet>
  <sonnet xmlns:ixml="http://invisiblexml.org/NS" ixml:state="ambiguous">
    <octave>
      <quatrain>
        <line>I met a traveller from an antique land,</line>
        <line>Who said &#x2014;Two vast and trunkless legs of stone</line>
        <line>Stand in the desart....Near them,on the sand,</line>
        <line>Half sunk a shattered visage lies,whose frown,</line>
      </quatrain>
      <quatrain>
        <line>And wrinkled lip,and sneer of cold command,</line>
        <line>Tell that its sculptor well those passions read</line>
        <line>Which yet survive,stamped on these lifeless things,</line>
        <line>The hand that mocked them,and the heart that fed;</line>
      </quatrain>
    </octave>
    <sestet>
      <tercet>
        <line>And on the pedestal,these words appear:</line>
        <line>My name is Ozymandias,King of Kings,</line>
        <line>Look on my Works,ye Mighty,and despair!</line>
      </tercet>
      <tercet>
        <line>Nothing beside remains.Round the decay</line>
        <line>Of that colossal Wreck,boundless and bare</line>
        <line>The lone and level sands stretch far away.</line>
      </tercet>
    </sestet>
  </sonnet>
</ixml>

Ozymandias (or surrogate) in IXML with complete LMNL and TexMECS output

Our next task was to see whether IXML could create LMNL and TexMECS output with overlapping markup in a single serialization. As with the Hamlet experiment, because IXML can serialize output only as XML:

We create a root XML element that contains the LMNL or TexMECS document as plain text. The tags for that root element would subsequently need to be removed to create a well-formed LMNL or TexMECS document.
We suppress all XML tags and create LMNL or TexMECS tags using IXML string insertions.
Because TexMECS uses angle brackets as tag delimiters and emitting plain-text angle brackets is not possible in IXML (IXML output must be serialized XML, which means that angle brackets [or, at least, opening angle brackets] must be escaped, typically as character entities), we emit curly braces instead of angle brackets and then transform them to angle brackets in post-processing.

The challenge to creating an IXML grammar for this task is that we need to manage the two hierarchies simultaneously. Because TexMECS is hierarchical, if we have a phrase transition at the end of a line and want to serialize as TexMECS, we would need to emit output in the following order: 1) end phrase, 2) end line, 3) original end-of-line character (if we wish to retain it), 4) start line, 5) start phrase. Because LMNL tags are not hierarchical, the order of tags at the same offset position in the plain text is not informational, but for human legibility we would prefer to output LMNL tags in the same order as TexMECS tags.

Managing two hierarchies simultaneously means handling all possible transitions from a combination of states in both to a different combination of states in both. The number of possible combinations with the two Ozymandias hierarchies was so great that it proved impractical for expository use, so we switched at this point to a four-line poem by Ogden Nash, entitled The Open Road:

I think that I shall never see
A billboard lovely as a tree.
Perhaps, unless the billboards fall,
I’ll never see a tree at all.

We identified two hierarchies to tag in this poem:

Structural: couplets and lines
Linguistic: sentences and phrases

A state consists of a combination of couplet, line, sentence, and phrase, which means that there are sixteen possible states, where each of the four features (couplet, line, sentence, phrase) can be qualified with a 1 or 2, e.g., c1-l1-s1-p1 (couplet #1, line #1, sentence #1, phrase #1), etc. The numbers do not mean first and second; their purpose is that when we move from one state to another, the same number means that that part of the state does not change and a different number means that it does. For example, a transition from c1-l1-s1-p1 to c1-l2-s1-p1 corresponds to a phrasal enjambment from the first line of a couplet to the second: the line changes, but the couplet, sentence, and phrase remain the same. Any state can transition to any other state except that 1) a state cannot transition to the same state, 2) a change in couplet implies a change in line, and 3) a change in sentence implies a change in phrase. Additionally, there is a start state (start must transition to c1-l1-s1-p1) and an end state (which is only reachable from states that would otherwise include a couplet transition and a sentence transition (because in our model a poem cannot end in the middle of a couplet or the middle of a sentence). The result is that except for the start, every state can transition to exactly five or six other states, where the sixth is a transition to the end state. For example, c1-l1-s1-p1 can transition to:

c1-l1-s1-p2 (change phrase in the middle of a line and sentence)
c1-l1-s2-p1 (change sentence in the middle of a line; the phrase number is not informational because a new sentence always implies a new phrase)
c1-l2-s1-p2 (change line in the middle of a couplet while also changing phrase in the middle of a sentence)
c1-l2-s2-p1 (change line in the middle of a couplet and start a new sentence on the new line)
c1-l2-s1-p1 (change line in the middle of a couplet and continue the same phrase, and therefore the same sentence; this is phrasal enjambment)

The total number of possible transitions is 89 (16 source states, half of which have 5 target states and half of which have 6, plus one transition from start). Below is an unambiguous IXML grammar that creates LMNL output from The Open Road—all 143 lines of it!:

{ Run with: coffeepot -g:open-road-ixml-all-transitions-ixml-syntax.ixml -i:open-road.txt | sed '/^</d' }
start : +#a, +"[poem}", +#a, +"[c}[l}[s}[p}", c1-l1-s1-p1.
-end : +#a, +"{poem]", +#a.
-text : -" "*, ~[" "; #a], ([L]; ["’ "])+.
-pPunc : ",".
-sPunc : ".".
-eol : #a.
-c1-l1-s1-p1 : text, 
  (
    pPunc, +"{p] [p}", c1-l1-s1-p2; 
    +"{p]", sPunc, +"{s] [s}[p}", c1-l1-s2-p1; 
    +"{p]{l]", pPunc, eol, +"[l}[p}", c1-l2-s1-p2; 
    +"{p]", sPunc, +"{s]{l]", eol, +"[l}[s}[p}", c1-l2-s2-p1; 
    +"{l]", eol, +"[l}", c1-l2-s1-p1
  ).
-c1-l1-s1-p2 : text, 
  (
    pPunc, +"{p] [p}", c1-l1-s1-p1; 
    +"{p]", sPunc, +"{s] [s}[p}", c1-l1-s2-p1; 
    +"{l]", eol, +"[l}", c1-l2-s1-p2; 
    +"{p]{l]", pPunc, eol, +"[l}[p}", c1-l2-s1-p1; 
    +"{p]", sPunc, +"{s]{l]", eol, +"[l}[s}[p}", c1-l2-s2-p1
  ).
-c1-l1-s2-p1 : text, 
  (
    pPunc, +"{p] [p}", c1-l1-s2-p2; 
    +"{p]", sPunc, +"{s] [s}[p}", c1-l1-s1-p1; 
    +"{l]", eol, +"[l}", c1-l2-s2-p1; 
    +"{p]{l]", pPunc, eol, +"[l}[p}", c1-l2-s2-p2; 
    +"{p]", sPunc, +"{s]{l]", eol, +"[l}[s}[p}", c1-l2-s1-p1
  ).
-c1-l1-s2-p2 : text, 
  (
    pPunc, +"{p] [p}", c1-l1-s2-p1; 
    +"{p]", sPunc, +"{s] [s}[p}", c1-l1-s1-p1; 
    +"{l]", eol, +"[l}", c1-l2-s2-p2; 
    +"{p]{l]", pPunc, eol, +"[l}[p}", c1-l2-s2-p1; 
    +"{p]", sPunc, +"{s]{l]", eol, +"[l}[s}[p}", c1-l2-s1-p1
  ).
-c1-l2-s1-p1 : text, 
  (
    pPunc, +"{p] [p}", c1-l2-s1-p2; 
    +"{p]", sPunc, +"{s] [s}[p}", c1-l2-s2-p1; 
    +"{l]{c]", eol, +"[c}[l}", c2-l1-s1-p1; 
    +"{p]", pPunc, +"{l]{c]", eol, +"[c}[l}[p}", c2-l1-s1-p2; 
    sPunc, +"{p]{s]{l]{c]", eol, +"[c}[l}[s}[p}", c2-l1-s2-p1; 
    sPunc, +"{p]{s]{l]{c]", -#a?, end
  ).
-c1-l2-s1-p2 : text, 
  (
    pPunc, +"{p] [p}", c1-l2-s1-p1; 
    +"{p]", sPunc, +"{s] [s}[p}", c1-l2-s2-p1; 
    +"{l]{c]", eol, +"[c}[l}", c2-l1-s1-p2; 
    +"{p]", pPunc, +"{l]{c]", eol, +"[c}[l}[p}", c2-l1-s1-p1; 
    sPunc, +"{p]{s]{l]{c]", eol, +"[c}[l}[s}[p}", c2-l1-s2-p1; 
    sPunc, +"{p]{s]{l]{c]", -#a?, end
  ).
-c1-l2-s2-p1 : text, 
  (
    pPunc, +"{p] [p}", c1-l2-s2-p2; 
    +"{p]", sPunc, +"{s] [s}[p}", c1-l2-s1-p1; 
    +"{l]{c]", eol, +"[c}[l}", c2-l1-s2-p1; 
    +"{p]", pPunc, +"{l]{c]", eol, +"[c}[l}[p}", c2-l1-s2-p2; 
    sPunc, +"{p]{s]{l]{c]", eol, +"[c}[l}[s}[p}", c2-l1-s1-p1; 
    sPunc, +"{p]{s]{l]{c]", -#a?, end
  ).
-c1-l2-s2-p2 : text, 
  (
    pPunc, +"{p] [p}", c1-l2-s2-p1; 
    +"{p]", sPunc, +"{s] [s}[p}", c1-l2-s1-p1; 
    +"{l]{c]", eol, +"[c}[l}", c2-l1-s2-p2; 
    +"{p]", pPunc, +"{l]{c]", eol, +"[c}[l}[p}", c2-l1-s2-p1; 
    sPunc, +"{p]{s]{l]{c]", eol, +"[c}[l}[s}[p}", c2-l1-s1-p1; 
    sPunc, +"{p]{s]{l]{c]", -#a?, end
  ).
-c2-l1-s1-p1 : text, 
  (
    pPunc, +"{p] [p}", c2-l1-s1-p2; 
    +"{p]", sPunc, +"{s] [s}[p}", c2-l1-s2-p1; 
    +"{l]", eol, +"[l}", c2-l2-s1-p1; 
    +"{p]{l]", pPunc, eol, +"[l}[p}", c2-l2-s1-p2; 
    +"{p]", sPunc, +"{s]{l]", eol, +"[l}[s}[p}", c2-l2-s2-p1
  ).
-c2-l1-s1-p2 : text, 
  (
    pPunc, +"{p] [p}", c2-l1-s1-p1; 
    +"{p]", sPunc, +"{s] [s}[p}", c2-l1-s2-p1; 
    +"{l]", eol, +"[l}", c2-l2-s1-p2; 
    +"{p]{l]", pPunc, eol, +"[l}[p}", c2-l2-s1-p1; 
    +"{p]", sPunc, +"{s]{l]", eol, +"[l}[s}[p}", c2-l2-s2-p1
  ).
-c2-l1-s2-p1 : text, 
  (
    pPunc, +"{p] [p}", c2-l1-s2-p2; 
    +"{p]", sPunc, +"{s] [s}[p}", c2-l1-s1-p1; 
    +"{l]", eol, +"[l}", c2-l2-s2-p1; 
    +"{p]{l]", pPunc, eol, +"[l}[p}", c2-l2-s2-p2; 
    +"{p]", sPunc, +"{s]{l]", eol, +"[l}[s}[p}", c2-l2-s1-p1
  ).
-c2-l1-s2-p2 : text, 
  (
    pPunc, +"{p] [p}", c2-l1-s2-p1; 
    +"{p]", sPunc, +"{s] [s}[p}", c2-l1-s1-p1; 
    +"{l]", eol, +"[l}", c2-l2-s2-p2; 
    +"{p]{l]", pPunc, eol, +"[l}[p}", c2-l2-s2-p1; 
    +"{p]", sPunc, +"{s]{l]", eol, +"[l}[s}[p}", c2-l2-s1-p1
  ).
-c2-l2-s1-p1 : text, 
  (
    pPunc, +"{p] [p}", c2-l2-s1-p2; 
    +"{p]", sPunc, +"{s] [s}[p}", c2-l2-s2-p1; 
    +"{l]{c]", eol, +"[c}[l}", c1-l1-s1-p1; 
    +"{p]", pPunc, +"{l]{c]", eol, +"[c}[l}[p}", c1-l1-s1-p2; 
    sPunc, +"{p]{s]{l]{c]", eol, +"[c}[l}[s}[p}", c1-l1-s2-p1; 
    sPunc, +"{p]{s]{l]{c]", -#a?, end
  ).
-c2-l2-s1-p2 : text, 
  (
    pPunc, +"{p] [p}", c2-l2-s1-p1; 
    +"{p]", sPunc, +"{s] [s}[p}", c2-l2-s2-p1; 
    +"{l]{c]", eol, +"[c}[l}", c1-l1-s1-p2; 
    +"{p]", pPunc, +"{l]{c]", eol, +"[c}[l}[p}", c1-l1-s1-p1; 
    sPunc, +"{p]{s]{l]{c]", eol, +"[c}[l}[s}[p}", c1-l1-s2-p1; 
    sPunc, +"{p]{s]{l]{c]", -#a?, end
  ).
-c2-l2-s2-p1 : text, 
  (
    pPunc, +"{p] [p}", c2-l2-s2-p2; 
    +"{p]", sPunc, +"{s] [s}[p}", c2-l2-s1-p1; 
    +"{l]{c]", eol, +"[c}[l}", c1-l1-s2-p1; 
    +"{p]", pPunc, +"{l]{c]", eol, +"[c}[l}[p}", c1-l1-s2-p2; 
    sPunc, +"{p]{s]{l]{c]", eol, +"[c}[l}[s}[p}", c1-l1-s1-p1; 
    sPunc, +"{p]{s]{l]{c]", -#a?, end
  ).
-c2-l2-s2-p2 : text, 
  (
    pPunc, +"{p] [p}", c2-l2-s2-p1; 
    +"{p]", sPunc, +"{s] [s}[p}", c2-l2-s1-p1; 
    +"{l]{c]", eol, +"[c}[l}", c1-l1-s2-p2; 
    +"{p]", pPunc, +"{l]{c]", eol, +"[c}[l}[p}", c1-l1-s2-p1; 
    sPunc, +"{p]{s]{l]{c]", eol, +"[c}[l}[s}[p}", c1-l1-s1-p1; 
    sPunc, +"{p]{s]{l]{c]", -#a?, end
  ).

The output of transforming the plain-text input with the grammar above (and piping the result into the sed operation described in the initial comment) is:

[poem}
[c}[l}[s}[p}I think that I shall never see{l]
[l}A billboard lovely as a tree.{p]{s]{l]{c]
[c}[l}[s}[p}Perhaps,{p] [p}unless the billboards fall{p]{l],
[l}[p}I’ll never see a tree at all.{p]{s]{l]{c]
{poem]

If we modify the grammar to emit TexMECS markup instead of LMNL,^[29] the output is:

<poem|
<c|<l|<s|<p|I think that I shall never see|l>
<l|A billboard lovely as a tree.|p>|s>|l>|c>
<c|<l|<s|<p|Perhaps,|p> <p|unless the billboards fall|p>|l>,
<l|<p|I’ll never see a tree at all.|p>|s>|l>|c>
|poem>

A comparison of IXML grammars to tag The Open Road and Ozymandias invites us to consider three features:

Both poems have two hierarchies, one for poetic structure and one for linguistic structure. From this perspective the two poems are equivalent.
Both hierarchies in Ozymandias are more complex than their counterparts in Open Road. Specifically:
- The poetic-structure hierarchy for The Open Road has two levels: couplet and line. A poem in couplets can have any number of couplets, but there are only four possible couplet/line states, so that, for example, a combination like c1-l1 would repeat every fourth line.
  
  The poetic-structure hierarchy for Ozymandias has three levels: octave/sestet, quatrain/tercet, and line. Each of the fourteen lines of Ozymandias is in a different poetic state, that is, a different combination of those three levels, and this lets us use a 14-valued l (line) property as a surrogate for all existing combinations of values on the three tiers. For example, just l1 (line 1) implies o-q1-l1 (octave [there is exactly one], quatrain 1, line 1). The only possible transition from l1 is to l2, etc. This is more than the four possible combinations in Open Road, each with exactly one or two possible transitions (sometimes two because both c1-l2 and c2-l2 can transition either to a new first line of a different couplet or to the end state), but it is finite, which means that by itself (that is, as long as we don’t also factor in the linguistic structure) it does not preclude constructing grammar rules for all possible transitions.
- Linguistic structure: The Open Road has two levels: sentence and phrase. Ozymandias also includes quotes, which, in our (simplified) model, can occur wherever phrases can occur and which must contain one or more sentences.
The linguistic structure of Ozymandias is recursive. As we note above, lines 10–11 (My name is Ozymandias …) actually constitute a quote, and the only reason we did not tag it as such is that detecting it required NLP that could not realistically be implemented in a robust fashion with only IXML. Insofar as sentences can contain quotes and quotes must contain sentences, those two element types can nest infinitely. This is not the case with The Open Road, where neither hierarchy is recursive.

What makes the number of possible state transitions with Ozymandias infinite is the combination of multiple hierarchies and infinite recursion (in one hierarchy). With just one (but not both) of those conditions:

As our treatment of The Open Road shows, combining multiple hierarchies has a finite (even if potentialy large) number of possible transitions when there is no recursion.
As our IXML grammar for just the linguistic hierarchy in Ozymandias shows, infinite recursion is a not a problem with a single hierarchy because we do not need to spell out (flatten) the infinite number of possible patterns.
If we want both multiple hierarchies and recursion, we could enumerate all of the potential states if we artificially capped the depth of recursion. This, though, would effective make the model non-recursive.

Why isn’t it possible to write an IXML grammar that will track two independent hierarchies, one of which is genuinely recursive? The issue is that we would need to flatten the combinations, as we did with our Open Road solution, and because the recursion is infinite, a string that describes any possible state is infinitely long. For example: whether we move from a state with the same (current) sentence to one with a different sentence inside a quote is independent of whether that quote is inside an s1 or s2. That containing sentence, in turn, could be inside a quote, which is inside a sentence at a higher containing level, etc. Because the number of containing levels is infinite, the number of state combinations is not enumerable.

As we illustrate below, this combination is fully tractable with Invisible-Next.

Ozymandias in Invisible-Next with `CONCUR` and TexMECS output

As with our Hamlet experiment, above, an Invisible-Next grammar that manages the overlap in either The Open Road or Ozymandias looks very much like an IXML grammar with two-way ambiguity within the root.^[30] The only change we need to make is that we replace an or operator with a CONCUR operator, e.g., in the grammar for Ozymandias we change:

sonnet: (s; q)+; octave, -#a, sestet.

sonnet: (s; q)+ ~~ octave, -#a, sestet.

In the case of The Open Road, an IXML grammar that produces two-way ambiguity is:

ixml version "1.1".
{ shared by both hierarchies }
poem: sentence+; couplet++eol.
-eol: #a.

{ linguistic only }
sentence: Pphr*, phr, sPunc.
Pphr>phr: -phr, pPunc.
phr: (-" "; -#a)*, [L], ([L; "’ "]; #a)+.
-sPunc: ".".
-pPunc: ",".
-ws: -space; eol.
space: " ".

{ poetic only}
couplet: line, eol, line.
line: ~[#a]+.

Replacing the production for poem with the Invisible-Next CONCUR operator:

poem: sentence+ ~~ couplet++eol.

produces a grammar that emits a single output document with overlapping hierarchies. We have to tell our Invisible-Next processor to serialize the output as LMNL or TexMECS because overlapping markup cannot be serialized directly as XML. The Invisible-Next grammar that produces our LMNL or TexMECS output requires sixteen lines (we’ve omitted the IXML version statement, but kept the signposting comments and blank lines), providing almost a ninefold saving in line count and an immeasurable improvement in legibility over the 143-line IXML grammar, with the same functionality, above.

Experiment 3: Creating dynamic node names in Invisible-Next

A well-known limitation of IXML is that it is not possible to create node names dynamically:

You can change the names used for serialization, but you can’t make names that are based on content in the input. There’s no rule that will match a month name and produce a <March> element or a March attribute.

— Norm Tovey-Walsh 2025

More generally, XML element names in IXML output are based on the LHS of the productions, and it is not possible to use content from the input to specify a LHS name or alias. There are at least two ways we might work around that limitation using only IXML and other XML-related methods. The first solution uses IXML to create generic markup, which we then transform with XSLT to create element names based on input values. The second solution uses only IXML, at the expense of requiring static knowledge of the element names in advance.

Assume that the input we need to transform is the following plain text:

id: haentjensdekker2026
paper-title: On beyond Invisible XML: parsing with implicit (and explicit) next-generation markup
author: Haentjens Dekker, Ronald
author: Birnbaum, David J.
author: Buitendijk, Bram
collection-title: Proceedings of Balisage: The Markup Conference 2026
series-title: Balisage Series on Markup Technologies
volume: 31
publication-date: 2026
doi: 10.4242/BalisageVol31.HaentjensDekker010
url: https://doi.org/10.4242/BalisageVol31.HaentjensDekker0

Assume that the desired output is the following:

<record>
   <id>haentjensdekker2026</id>
   <paper-title>On beyond Invisible XML: parsing with implicit (and explicit) next-generation markup</paper-title>
   <author>Haentjens Dekker, Ronald</author>
   <author>Birnbaum, David J.</author>
   <author>Buitendijk, Bram</author>
   <collection-title>Proceedings of Balisage: The Markup Conference 2026</collection-title>
   <series-title>Balisage Series on Markup Technologies</series-title>
   <volume>31</volume>
   <publication-date>2026</publication-date>
   <doi>10.4242/BalisageVol31.HaentjensDekker010</doi>
   <url>https://doi.org/10.4242/BalisageVol31.HaentjensDekker0</url>
</record>

If we don’t require IXML to do all of the work, input like the example above could first be parsed into a graph with a parent node for each property and child nodes for the each of original key and value parts of the property. The parent nodes would all use the same name, and an IXML grammar that creates this type of parse result might be:

record: item++eol, eol?.
item: key, -kv-sep, value.
key: ~[":"]+.
value: ~[#a]+.
-kv-sep: -": ".
-eol: #a.

The output of that IXML process looks like:

<record>
   <item>
      <key>id</key>
      <value>haentjensdekker2026</value>
   </item>
   <item>
      <key>paper-title</key>
      <value>On beyond Invisible XML: parsing with implicit (and explicit) next-generation markup</value>
   </item>
   <!-- other items -->
</record>

We could then pass the preceding output through an XSLT transformation that turns each <item> element into the output we want. That XSLT might look like:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:math="http://www.w3.org/2005/xpath-functions/math" exclude-result-prefixes="#all"
  version="3.0">
  <xsl:template match="record">
    <!-- Don't copy possible ixml attributes -->
    <xsl:copy>
      <xsl:apply-templates/>
    </xsl:copy>
  </xsl:template>
  <xsl:template match="item">
    <xsl:element name="{key}">
      <xsl:value-of select="value"/>
    </xsl:element>
  </xsl:template>
</xsl:stylesheet>

and the output would look like:

<?xml version="1.0" encoding="UTF-8"?>
<record>
  <id>haentjensdekker2026</id>
  <paper-title>On beyond Invisible XML: parsing with implicit (and explicit) next-generation markup</paper-title>
  <author>Haentjens Dekker, Ronald</author>
  <author>Birnbaum, David J.</author>
  <author>Buitendijk, Bram</author>
  <collection-title>Proceedings of Balisage: The Markup Conference 2026</collection-title>
  <series-title>Balisage Series on Markup Technologies</series-title>
  <volume>31</volume>
  <publication-date>2026</publication-date>
  <doi>10.4242/BalisageVol31.HaentjensDekker010</doi>
  <url>https://doi.org/10.4242/BalisageVol31.HaentjensDekker0</url>
</record>

If, as an alternative, we want to create the final output directly with IXML, without requiring subsequent transformation, we can use a grammar like the following:

{ root }
record: item++eol, eol?.

{ or-list of all named fields}
-item: id; paper-title; author; collection-title; series-title; volume; publication-date; doi; url.

{ productions for named fields }
id: -"id", kv-sep, value.
paper-title: -"paper-title", kv-sep, value.
author: -"author", kv-sep, value.
collection-title: -"collection-title", kv-sep, value.
series-title: -"series-title", kv-sep, value.
volume: -"volume", kv-sep, value.
volume-title: -"volume-title", kv-sep, value.
publication-date: -"publication-date", kv-sep, value.
doi: -"doi", kv-sep, value.
url: -"url", kv-sep, value.

{ input formatting }
-kv-sep: -": ".
-value: ~[#a]+.
-eol: #a.

If we require IXML to do all of the work there is no escaping the need to create individual productions for each XML element type that we must be able to include in the output. We can, though, make the logic easier to understand by answering what types of elements do we need to create? in the or-list associated with item and answering how do we create those elements? by writing the productions according a simple and uniform template.

In Real Life, where our goal is to complete the task in a clear, robust, and maintainable way, we would, of course, use the first approach, the one that incorporates XSLT transformation. The IXML grammar with that approach is obviously simpler and easier to understand the the one that we use in our second approach, and it is also more capable and less brittle because it does not require static knowledge of the property names, which means that it avoids the risk that a user might accidentally omit or misspell one or more names. We include the second approach because it is the best we are able to do with IXML alone, and, as we explain below, Invisible-Next alone, unlike IXML, is able to create output element names dynamically, avoiding any need for either static knowledge of those names (method 2, above) or a separate transformation step (method 1, above).

Invisible-Next enriches the serialization part of the process to enable the dynamic creation of node names without static knowledge of those names in a grammar by borrowing a concept from attribute grammars and allowing the serialization process to use information from parent and child nodes in the parse forest to create element names in the output.^[31] It does this by first creating the same type of parse as IXML method 1, above, so that every element that needs a dynamic name in the output has the following basic structure in the parse result:

<item>
    <key>key string</key>
    <value>value string</value>
</item>

Invisible-Next then applies an attribute-grammar-like rule during serialization to say when you serialize an item node, the element name is taken from the child key node and the element value is taken from the child value node. This approach requires enriching the Invisible-Next syntax to enable it to convey this information to the serializer in the form of properties that are added to the nodes during parsing. An Invisible-Next grammar that can create the desired output directly might look like the following:

record: item++eol, eol?.
item<-key: key, kv-sep, value.
-key: -~[":"]+.
-value: ~[#a]+.
-kv-sep: -": ".
-eol: #a.

The starting point for this grammar is the generic IXML grammar above, the one that created item nodes with key and value children, and the principal difference between this grammar and IXML involves a new Invisible-Next operator, <-.^[32] The parse result is the same as with IXML, except that the new operator adds information to the result that an integrated attribute grammar can use during serialization. The meaning of <- is that it is a new type of aliasing, one that, instead of specifying the new name literally, says to take that name from the pattern that matches, in this case, key. Minus signs before the LHS and RHS of the rule for key on line 3 have the same meaning as in IXML, and in this case they ensure that neither the markup nor the matched content are part of the output. The new <- alias operator ignores the minus signs and uses whatever the subpattern key matched. Attribute grammars commonly require references to be local, and here the rule renames a parent item in the parse result according to the value of its child key node. In this case the new name happens to be the first child of the parent in the parse result, but the target could be any child.^[33]

Experiment 4: Distinguishing ordered and unordered content

An XML element hierarchy is an ordered tree, which means, among other things, that the order of sibling elements is syntactically meaningful. We may or may not care about that meaning, but two documents that differ only in the order of sibling elements are not deep-equal.^[34] Whether we are required to care about the order of sibling elements is often domain-specific, so that, for example, the order of <chapter> children of a <novel> element obviously matters, while we are not required to care about the order of triples in an RDF graph because in RDF semantics those triples are informationally unordered even when the graph is expressed with XML syntax. We might say informally that RDF triples are unordered because the order doesn’t affect the meaning, but when RDF is expressed in XML, two RDF graphs with the same triples in different orders are still not deep-equal. One way to think about the issue is that whether items are ordered may have a different answer depending on whether we look at the semantics of the model or its syntactic expression.^[35]

We can model XML sibling elements as ordered or unordered in a schema, but no matter what the schema says, the markup in an XML document instance is not capable of representing a difference between schema-ordered and schema-unordered content syntactically. That is, if we engage with an XML document instance in the absence of a schema, sibling elements are always ordered, and the markup syntax alone cannot tell us whether the schema required that order or whether the schema allowed unordered content and the order in the instance is accidental. We could incorporate something like an @ordered attribute to record the difference when needed, but that’s a semantic, and not a syntactic, approach. XML processing, such as with XSLT, can treat content as ordered or not, as required by domain semantics, but that becomes a feature of the application, and not of the document. Insofar as unordered content is a feature of some domain modeling, the ability to represent it syntactically, with markup, at the document level is a next-generation feature, that is, one that is not part of the design of XML.

TexMECS distinguishes ordered vs unordered content syntactically by using different tags: the TexMECS start-tag <e| and end-tag |e> contain ordered children, like XML tags, and the TexMECS start-tag <e|| and end-tag ||e> contain unordered children. This is a syntactic representation that is not dependent on semantics or schema rules. In the experiment below we’ll generate TexMECS markup for our examples.

If we turn from output markup to the processing of input, CFG productions on their own cannot describe unordered content except in a brute-force way. For example, if an XML element must contain exactly one instance each of elements A, B, and C, in any order, IXML could represent that requirement by fully listing all six permutations. At the same time, spelling out all possible combinations quickly becomes unrealistic, at least in a human-authored schema or grammar, as the number of possibilities grows.^[36]

For this experiment we started from the same plain text input as in the previous experiment, which we repeat here for convenience:

id: haentjensdekker2026
paper-title: On beyond Invisible XML: parsing with implicit (and explicit) next-generation markup
author: Haentjens Dekker, Ronald
author: Birnbaum, David J.
author: Buitendijk, Bram
collection-title: Proceedings of Balisage: The Markup Conference 2026
series-title: Balisage Series on Markup Technologies
volume: 31
publication-date: 2026
doi: 10.4242/BalisageVol31.HaentjensDekker010
url: https://doi.org/10.4242/BalisageVol31.HaentjensDekker0

The fields of a bibliographic record are often considered unordered, but the order of authors is usually considered informational. From that perspective, the TexMECS markup we might use for this record is something like:

<record||
  <id|haentjensdekker2026|id>
  <paper-title|On beyond Invisible XML: parsing with implicit (and explicit) next-generation markup|paper-title>
  <authors|
    <author|Haentjens Dekker, Ronald|author>
    <author|Birnbaum, David J.|author>
    <author|Buitendijk, Bram|author>
  |authors>
  <collection-title|Proceedings of Balisage: The Markup Conference 2026|collection-title>
  <series-title|Balisage Series on Markup Technologies|series-title>
  <volume|31|volume>
  <publication-date|2026|publication-date>
  <doi|10.4242/BalisageVol31.HaentjensDekker010|doi>
  <url|https://doi.org/10.4242/BalisageVol31.HaentjensDekker0|url>
||record>

Elements in this document have three types of content:

The children of an element are ordered. This applies to the authors element.
The children of an element are unordered. This applies to the root record element.
An element contains only one node, which means that the ordered vs unordered opposition is neutralized. This applies to all other elements. We use the ordered-content <e| and |e> tags here because Huitfeldt and Sperberg-McQueen 2003 refers to these as the tags for normal elements, that is, as the default.

In addition to distinguishing ordered from unordered content, this task also requires us to insert a wrapper element around the three author inputs. We can do that in IXML that emits regular XML output (that is, that doesn’t distinguish ordered from unordered content) by making three modifications to the IXML above:

We remove the author term from the RHS of the rule for item. We do not remove the production for author; we just remove the reference to it from the RHS of the item production.
We create a production for authors, which matches consecutive author lines.
We revise the root element to allow any number of regular items plus no more than one instance of authors.

The revised IXML, which produces the correct unambiguous result, looks like:

{ root }
record: item**eol, authors, item**eol.

{ or-list of all named fields except author, which is wrapped}
-item: id; paper-title; collection-title; series-title; volume; publication-date; doi; url.

{ productions for named fields, including wrapped ones }
id: -"id", kv-sep, value.
paper-title: -"paper-title", kv-sep, value.
author: -"author", kv-sep, value.
collection-title: -"collection-title", kv-sep, value.
series-title: -"series-title", kv-sep, value.
volume: -"volume", kv-sep, value.
volume-title: -"volume-title", kv-sep, value.
publication-date: -"publication-date", kv-sep, value.
doi: -"doi", kv-sep, value.
url: -"url", kv-sep, value.

{ productions for inserted wrappers }
authors: eol?, author++eol, eol?.

{ input formatting }
-kv-sep: -": ".
-value: ~[#a]+.
-eol: #a.

Here is how it works:

Removing author from the set of item elements means that individual author lines in the input will not be treated like children of the root record element.
Allowing authors as part of the RHS of the rule for the record root means that an authors node can be created there in the parse result whenever the RHS of the associated rule (which we describe below) is matched. This is a common idiom for inserting a wrapper around input items in IXML.
The new authors item will be created whenever one or more consecutive author line is encountered in the input, and it will wrap all contiguous author lines at that location. The one or more is a complication because it means that we can’t just add authors to the RHS of the item rule because that would make it repeatable, and if authors were repeatable, it would create ambiguity because the three consecutive author lines could then be wrapped with authors either individually or in various groups. The issue is that because IXML cannot model interleave directly, it becomes awkward to say that a record contains, in any order, zero or more instances of any individual member of the RHS of item plus no more than one instance of authors.

Our workaround is to say that we can have zero or more item components, any of which may repeat, followed by exactly one authors component, followed by another group of zero or more item components. The created authors component, in turn, contains one or more author lines.

The modification to the rule for record will cause it to fail if there are no author lines in the input, which will matter if we wish to allow input of that sort. This approach also requires that all author input lines appear adjacent to one another. This requirement is a consequence of the way IXML scannerless parsing works: the input is processed character by character, and once the processing has encountered, processed, and moved on from a sequence of author lines, it cannot add a later, non-consecutive author to the group. The following modification to the rule for record would allow multiple groups of authors in the input to produce multiple authors wrappers in the output:

record: item**eol, (authors, item++eol)+, eol?.

but that probably isn’t what we want; if the record has multiple authors, we probably want them wrapped in a single authors wrapper. In Real Life we wouldn’t ask IXML to be responsible for this sort of situation; for example, if we want to allow author input items to appear non-contiguously and get grouped inside a single wrapper in the output, we could let IXML tag the input lines and pipe the result to a subsequent XSLT transformation that would assemble non-contiguous authors inside a single output wrapper.

The preceding code emits XML, rather than TexMECS, which means, among other things, that it does not have access to markup that distinguishes ordered from unordered content syntactically. The following modification, which applies a strategy we used in earlier experiments, suppresses all of the XML tags and uses string insertions to add TexMECS tags. As in preceding experiments, because it is not possible to create a literal left angle bracket with an IXML string insertion, we use curly braces instead of angle brackets and pipe the IXML output first through sed to remove the XML tags that IXML requires for the root element around its output and then through tr to replace the curly braces with angle brackets. Here is the IXML that does that:

{ Run with: coffeepot -g:biblio-texmecs.ixml -i:biblio.txt | sed '1d;$d' | tr '{}' '<>' }
{ root }
root: +#a, record, +#a.
-record: +"{record||", +#a, item**eol, authors, item**eol, +#a, +"||record}", eol?.

{ or-list of all named fields except author, which is wrapped}
-item: id; paper-title; collection-title; series-title; volume; publication-date; doi; url.

{ productions for named fields, including wrapped ones }
-id: +"{id|", -"id", kv-sep, value, +"|id}".
-paper-title: +"{paper-title|", -"paper-title", kv-sep, value, +"|paper-title}".
-author: +"{author|", -"author", kv-sep, value, +"|author}".
-collection-title: +"{collection-title|", -"collection-title", kv-sep, value, +"|collection-title}".
-series-title: +"{series-title|", -"series-title", kv-sep, value, +"|series-title}".
-volume: +"{volume|", -"volume", kv-sep, value, +"|volume}".
-volume-title: +"{volume-title|", -"volume-title", kv-sep, value, +"|volume-title}".
-publication-date: +"{publication-date|", -"publication-date", kv-sep, value, +"|publication-date}".
-doi: +"{doi|", -"doi", kv-sep, value, +"|doi}".
-url: +"{url|", -"url", kv-sep, value, +"|url}".

{ productions for inserted wrappers }
-authors: +#a, +"{authors|", eol?, author++eol, +#a, +"|authors}", eol?.

{ input formatting }
-kv-sep: -": ".
-value: ~[#a]+.
-eol: #a.

Except for whitespace indentation, the output of applying this IXML grammar to our input produces TexMECS markup that matches the target output described above.

The following Invisible-Next grammar creates the same output as the IXML, above, and it improves on some, but not all, of the limitations of using IXML for this purpose:

{ root }
&record: &(item, authors)(eol).

{ interleave of all named fields except author, which is wrapped }
-item: &(id, paper-title, collection-title, series-title, volume, publication-date, doi, url).

{ productions for named fields, including wrapped ones }
id: -"id", kv-sep, value.
paper-title: -"paper-title", kv-sep, value.
author: -"author", kv-sep, value.
collection-title: -"collection-title", kv-sep, value.
series-title: -"series-title", kv-sep, value.
volume: -"volume", kv-sep, value.
volume-title: -"volume-title", kv-sep, value.
publication-date: -"publication-date", kv-sep, value.
doi: -"doi", kv-sep, value.
url: -"url", kv-sep, value.

{ productions for inserted wrappers }
authors: author++eol.

{ input formatting }
-kv-sep: -": ".
-value: ~[#a]+.
-eol: #a.

The following are the new Invisible-Next features:

The ampersand (&) before a nonterminal says that the child nodes in the parse result should be serialized as unordered, and we use that idiom in the LHS of the rule for record.
The ampersand before parenthesized content on the RHS (&( ... )) says the the items inside the parentheses are interleaved (in the RELAX NG sense of the term). Those items may or may not have repetition indicators; in this grammar none of them does, which means that each of them is required to appear exactly once.We use this spelling in two places:
- In the RHS of the rule for item we say that exactly one instance of each of the input line-types (except author) may appear in any order.
- In the RHS of the rule for record we say that the item nodes are interleaved with a single instance of authors, which is created below. Because the IXML syntax for inserting a separator requires repetition (either ** or ++) and in this case we want to specify a separator to be inserted between the interleaved items without repetition, we use a second parenthesized expression, after the &( ... ) primary interleave expression, to contain the separator. The RHS of the rule for record, then, says to interleave the value of the item production (which contains multiple items) with the value of the authors production (which contains the author input lines as a single block) and insert newline characters between them.

As with our previous new Invisible-Next operators, we regard the specific syntactic expressions (that is, operators) described here as candidates to be evaluated further. What we like about the Invisible-Next methods includes:

The ability to specify whether content is ordered or unordered.
The ability to express some patterns that are complex in IXML in a simpler manner through the use of interleave.

What we find less satisfying is that the strategy we adopted in experiment 3 to create dynamic element names during serialization is not easily expressed once interleave is involved. As a result, the grammar above is based on static knowledge of all input patterns. Trying to process those patterns with the dynamic naming strategy from our third experiment, without naming them, would make it challenging to treat author differently from the other input items. Our rule for authors would not be enough; it would create an ambiguity because it would not exempt input author lines from simultaneously being recognized by the dynamic rule that processes other input lines. A language with precedence might let us try to match author lines first for special handling, and create generic names only for lines that are not handled first by other rules, but Invisible-Next, like IXML, does not have precedence, and if multiple patterns match part of the input, that creates ambiguity, rather than deterministic decision-making. Alternatively, a language with a not operator might be able to exclude the string author from a general rule to create dynamic names, but Invisible-Next, like IXML, lacks this type of not operator.^[37] Integrating interleave with dynamic naming requires further investigation.

Results

What we learned about the `CONCUR` operator

LMNL can regard adjacent tags as simultaneous because tags can be understood as delimiting character ranges, and not as containing other ranges in an XML-like hierarchical sense.^[38] This means that the order of line and speech start- and end-tags in our LMNL Hamlet output might be regarded as non-informational. Nonetheless, for human legibility we wanted to avoid transitions like:

[line}Passing through nature to eternity.{line]
[line}{speech][speech}[speaker}HAMLET{speaker] Ay, madam, …

This requirement frustrated our ability to treat both line and speech transitions simply as milestones. For example, if we wanted always to rewrite an incoming metrical line transition (newline) in LMNL as {line]\n[line} and also always to rewrite an incoming speech transition as {speech] [speech}, it would not be possible, in response to a line event alone or a speech event alone, to emit {speech]{line]\n[line}[speech}. Our grammars work around this limitation successfully, but only by complicating the set of productions, and doing so in a way that does not scale well with the incorporation of additional overlapping hierarchies.

The most important detail of our use of IXML to emit non-XML markup with overlap and discontinuity is that the IXML grammars that ultimately produce our LMNL and TexMECs output are complex because they need to recognize and act on the interaction of two hierarchies, and not on each of them separately from the other. Non-ambiguous grammars that recognize transitions in these two hierarchies independently of each other are not supported directly and idiomatically in IXML, and we work around that limitation by matching patterns from both the metrical-line and the speech hierarchies in Hamlet or the sonnet and linguistic hierarchies in Ozymandias before recognizing and responding to a state that depends on interactions of those two types of patterns. Some of the relevant patterns are shared by the two hierarchies with the same meaning in terms of implicit structure, some are shared with different meaning, and some (partitions) are meaningful for only one hierarchy, but must nonetheless be included in rules for both because although IXML grammars can ignore parts of the input when creating output, they nonetheless must see and acknowledge the entire input character stream during parsing.

The support for a CONCUR operator in our Invisible-Next grammar means that our Invisible-Next productions are much less complex than their counterparts in our our IXML grammars that emit LMNL and TexMECS that encode overlap (and, for TexMECS, also discontinuity). The source of that simplicity is that Invisible-Next productions for one hierarchy do not have to include code that refers to a different hierarchy; Invisible-Next grammars instead describe each hierarchy independently of the other.

What we learned about ambiguity

Standard IXML parses the input into a forest, from which it then selects and serializes (or, rather, attempts to serialize) one tree. Invisible-Next, though, allows us to regard multiple parses of the same input not as ambiguity, but as concurrency. If we think of ambiguity as this or that and concurrency as both this and that simultaneously, we might model ambiguity as a graph from which we select exactly one path for serialization, and concurrency as a graph within which all paths are simultaneously available and selection among them is not part of the model. The preceding suggests two areas for comparison, input parsing and output serialization, as follows:

With respect to input parsing, as we write above, an Invisible-Next grammar production that uses the CONCUR operator describes two patterns in the RHS, both of which must be satisfied for the input to match the rule. Standard IXML cannot act on events in one overlapping hierarchy independently of events in another, although grammar authors can accumulate those events and act on them together (even if not very scalably), as we do in our IXML Hamlet and Ozymandias experiments, above.
A standard IXML forest and the graph that we construct when parsing with an Invisible-Next grammar are structurally similar in that in both cases a node in the representation of the parsing operation may have multiple in-edges and multiple out-edges. What is different is the semantics associated with multiple paths (alternative in IXML vs simultaneous with next-generation modeling) and the implications of those semantics for serialization (standard IXML cannot serialize multiple paths simultaneously because it must serialize a tree, a limitation that does not apply to our next-generation modeling). As a result, IXML cannot serialize overlapping hierarchies because it must serialize as XML, unless we treat the (non-XML) markup output as text that is emitted through insertions. This is considerably more cumbersome than the native IXML ability to emit well-formed XML.

The preceding addresses only the type of ambiguity that arises in IXML when the same input matches multiple patterns. It does not address some other types of ambiguity. Here are two examples:

Example #1: The following IXML grammar is ambiguous:

doc: CAT; ANIMAL.
CAT: "cat".
ANIMAL: "cat".

When used to parse the input cat, it would report ambiguity, with the following two parses:

<doc IXML:state="ambiguous">
   <CAT>cat</CAT>
</doc>

<doc IXML:state="ambiguous">
   <ANIMAL>cat</ANIMAL>
</doc>

The same grammar would also be ambiguous under our proposed enhancements for next-generation features. The difference is that our proposal would also support the following alternative grammar (where ~~ means CONCUR), where:

doc: CAT ~~ ANIMAL.
CAT: "cat".
ANIMAL: "cat".

The single unambiguous LMNL output of parsing the input cat against this grammar is [ANIMAL}[CAT}cat{ANIMAL]{CAT].^[39]

If we regard the context as overlap, that is, if we want to tag the text as both CAT and ANIMAL simultaneously, we could select the concur operator, and if we want to regard the document as having two competing parses, we could select the alternation operator. Our point with this example is that if we want to describe the situation as an alternative, the availability of a concur operator does not help avoid ambiguity because the concept of alternative, in this example, is inherently structurally ambiguous.

Example #2: Given an Invisible-Next grammar like the following:

doc: A+ ~~ B+.
A: [“x”]+.
B: [“x”]+.

This says that a doc contains one or more instances of A that run concurrently with one or more instances of B. Since the only content of both A and B is x, it means that every instance of the character x must be within both an A and a B. This, in turn, means that:

The input x is unambiguous and resolve to LMNL [A}[B}x{A]{B]. (Recall that the order of concurrent tags in serialized LMNL is not significant.)
The input xx (and input with more instances of x) is ambiguous. The input xx can be tagged with the preceding grammar in four ways:
- [A}[B}xx{A]{B] (one instance of A and one of B)
- [A}[B}x{B][B}x{A]{B] (one instance of A and two of B)
- [A}[B}x{A][A}x{A]{B] (two instances of A and one of B)
- [A}[B}x{A]{B][A}[B}x{A]{B] (two instances of A and two of B)

The examples above correspond to what Norm Tovey-Walsh 2025 calls vertical (our example #1) and horizontal (our example #2) ambiguity.^[40] Indeed, this type of horizontal ambiguity is present in each of our two hierarchies independently, so that, for example:

doc: A+.
A: ["x"]+.

with no concurrency, is also horizontally ambiguous.

The point of this example is that although concurrency is a way of understanding some instances of ambiguity, here we have both vertical and horizontal ambiguity even within a concurrent model.

What we learned about output

Whether to serialize as TexMECS, LMNL, XML, or something else is controlled, in our implementation, by a command-line option. Our default output is TexMECS, which is the only markup language we support that represents discontinuity with special markup syntax. Support for LMNL and XML output are still under development, pending decisions about managing features not supported directly with distinctive markup syntax in those languages. The features under discussion include:

Overlap: Currently supported only for TexMECS and LMNL serialization.
Self-overlap: Not currently supported for any serialization.
Discontinuity: Currently supported only for TexMECS serialization.
Self-discontinuity: Not currently supported for any serialization.
Unordered content: Currently supported only for TexMECS serialization.
Partitions: Not represented in markup in any serialization; must be specified in a schema. To the extent that discontinuity (but not self-discontinuity) is caused by partitions, the partition is recognized implicitly by the discontinuity, but whether all non-self-overlap is caused by partitions requires further study.

Where users need to model concurrency, parsing with a concurrency operator and modeling as a graph that can be serialized in its entirety offers a formal model that matches the mental model. Furthermore, it does so without structural ambiguity because instead of having to choose between coexisting paths it regards all paths as simultaneously available.

Challenges with input files that contain pre-existing markup

The IXML approach to LMNL processing illustrated in Piez 2026 Symposium starts with input that contains explicit LMNL markup delimiters, and the pipeline uses IXML to parse that input into xMNML (XML representation of Minimally Annotated Markup in LMNL, a LMNL subset). xMNML uses empty elements, attribute pointers, and other standard XML resources to work around the inability of XML to represent next-generation features (overlapping ranges in the case of LMNL) directly in the model, that is, in ways that do not rely on domain-specific interpretation and processing conventions. This parsing, using a library that Piez calls the Laminator, replaces Piez’s earlier XSLT and XPath Luminescent library, recognizing that IXML, which was not available when Luminescent was first developed, is well-designed for this type of task. Subsequent processing of the tagged LMNL input, after conversion via IXML to xMNML, can then use the standard XML toolkit. For example, as Piez has demonstrated, subsequent processing can use XSLT to transform an xMNML representation into HTML with SVG to render visualizations of overlapping structures using standard features of existing web browsers.

Because currently there are no robust toolkits for markup languages that support next-generation features, parsing such markup into an XML representation of the data model and then employing XML tools is a natural work-around for processing next-generation markup today. If, though, we imagine a future with processing toolkits (transformation, query, path expressions, etc.) for at least some popular next-generation markup languages, the construction of an XML representation of the data would not be necessary; a pipeline could parse tagged input into an internal, markup-syntax-agnostic representation of a data model that could then be processed in ways that would not depend on an XML data model, XML syntax, or XML tools.^[41]

This type of future system could be preconfigured to recognize a variety of markup languages, but because, like RELAX NG and Creole as implemented in Tennison 2007, it operates with prior information about and knowledge of language-specific markup tokens, it would nonetheless be constrained to handle only tokenizers that are supported explicitly. An Invisible-Next approach to recognition and validation, on the other hand, operates at the character level without applying any built-in, markup-language-specific distinction between markup and content during input parsing, which means that if the input already contains markup, a user-specified grammar could assume responsibility for recognizing and managing it. Insofar as Invisible-Next rules could be written for any markup language, such a system could be truly agnostic about input markup, and therefore equally applicable to any next-generation markup language (and, for that matter, also to XML).^[42]

Parsing XML input and emitting Markup-Next output is a special case of processing input with markup for historical reasons: the number of XML documents already in existence is much greater than the number of documents in Markup-Next languages, and where those XML documents need to support next-generation features, they have had to do so with workarounds because XML markup and XML schema languages alone do not support the representation of those features without recourse to interpretive semantics. This limitation in the design of XML as a markup language encourages users to migrate corpora that rely on domain-specific semantic workarounds for next-generation features to markup languages that support those features directly through markup itself and associated next-generation schema languages. That, in turn, invites us to consider how best to recognize XML workarounds and transform them, using Invisible-Next, to direct representation in Markup-Next.

There is, of course, no free lunch. A Schema-Next approach is limited to processing the specific markup and schema languages it knows about, but although the Invisible-Next approach is agnostic about markup and does not require a schema language beyond the rules implemented through an Invisible-Next grammar, someone nonetheless needs to write that grammar. End-users might, then, anticipate a tooling situation like the following:

Where resources might come to exist for relatively popular next-generation languages, such as LMNL or TexMECS, those resources, implemented as Schema-Next validators or Invisible-Next grammar modules, might reasonably meet the validation needs of end-users.
Where projects employ domain-specific markup languages for which schema validation and processing toolkits do not exist, the flexibility of the Invisible-Next methods, which expose parsing rules for recognizing markup more directly to users than Schema-Next methods, could meet validation needs, albeit at the expense of requiring users to write their own Invisible-Next grammars.
Insofar as the Schema-Next approach is only about validation, and not about inserting new markup into tagged (or not-yet-tagged) input, only an Invisible-Next approach would enable users both to recognize implicit structures and to insert additional explicit markup into documents and emit enriched output.

A key feature of our perspective and approach is that both Schema-Next validation and Invisible-Next markup generation can use exactly the same data structures and code base, which allows us to implement both Schema-Next and Invisible-Next tools as thin wrappers around that shared data model and code base. To the best of our knowledge this is not currently the case with existing XML tools, where the only RELAX NG validator in wide use (Jing) is implemented through parsing with derivatives (Clark 2002, Tennison 2007) and the IXML tools listed at https://invisiblexml.org/ appear to be implemented in other ways (e.g., with Earley or GLL parsers). We are not able to explore parsing input that already contains next-generation markup in this paper, but we intend to return to it in the future.

Visualizing multiple hierarchies

The literature about visualizing multiple hierarchies is large, and the types of visualizations that are applicable to the overlap and discontinuity issues exemplified by the text above include 1. face-to-face dual-tree layout, 2. single-root multitree, 3. hypergraph, and 4. modified swimlane diagram. We illustrate and discuss these below.

Face-to-face dual-tree visualization

A face-to-face dual-tree visualization of the data above looks like:

Figure 2: Face-to-face dual-tree layout

The leftmost and rightmost Hamlet nodes in the visualization above are the same root element. The gray rectangles in the middle represent the plain text input, one newline-delimited string after another. In those text boxes we have rendered the speaker names in a monospaced font that is left-aligned within its containing rectangle and we have rendered the newline-delimited input strings that represent spoken text with indentation and a serif font. We use typography here to communicate that the newline-delimited input lines can be recognized as typed; some are of type Speaker and others of type Spoken words.^[43]

Our decision to use typing, rather than, for example, parent Speaker and Spoken nodes, is somewhat arbitrary, and we could, alternatively, have eschewed the typing and instead insert a hierarchical level between Speech and the text lines (whether for both speakers and spoken text or only for speakers, treating spoken-text as a default). The point of this illustration is to describe the visualization in a general way; it is not to rationalize and choose among low-level modeling alternatives, any of which could be accommodated in a rendering of the two hierarchies on opposite sides of the (partially) shared text strings.^[44] The alternatives discussed in this paragraph are similar to some of the modeling decisions that are familiar from XML (e.g., element vs attribute, granularity of markup). Insofar as they affect the hierarchical levels they would affect the visualization, but the basic face-to-face dual-tree design can accommodate any of these variations.

We have chosen to model speaker names as part of only the hierarchy of speeches (colored red and on the left), which means that they are not connected at all to the hierarchy of metrical lines (colored blue and on the right). We made that decision because the lines that contain speaker names do not contribute to the metrical structure of the information, which invites us to consider them as fundamental to the organization of the speeches but irrelevant to the organization of the metrical lines. To the extent that we might care about the speakers of the parts of a metrical line, nothing prevents us from consulting information in the two hierarchies in a coordinated way; the point is not that we don’t care about the speakers of the parts of the metrical lines, but, rather, that we care about that as part of caring about the organization of speeches, and not fundamentally about the organization of meter.

The last five gray textual rectangles are modeled, to the left, as two speeches, one by Hamlet and one by Gertrude, and, to the right, as two lines, one split between Hamlet and Gertrude and the other by Gertrude alone. That is, at this location speeches and metrical lines overlap, and neither type wholly and consistently contains the other. Our exclusion of speaker input lines from the metrical hierarchy reflects our decision to model them as what we have called partitions, that is, units that, in a situation involving the overlap of two hierarchies, are part of one hierarchy and irrelevant to the other. This model differs from a perspective on overlap in which all text nodes must be part of all overlapping hierarchies.

We regard this visualization as clearer and more explicit that the other three, discussed below, with respect to representing the two overlapping hierarchical structures. Its greatest weakness in terms of visual language is that it relies on two root nodes, one at the far left and the other at the far right, although informationally those are the same node. This is not difficult to understand, but it is nonetheless the case that the visual language is inconsistent in this detail with the information structure. The face-to-face dual-tree visualization communicates clearly that most, although not all, leaf nodes (gray, in the middle) are part of both hierarchies, a detail that we find somewhat easier to see in the modified swimlane visualization discussed below. The overlap is easy to see as long as we look at both hierarchies at the same time, but we find this feature also easier to understand from the modified swimlane visualization. Finally, this visualization represents two overlapping hierarchies with no overlapping edge lines, which contributes greatly to the clarity of the communication, but it is not well suited to overlap that entails more than two hierarchies, which cannot easily be represented without crossing edge lines.

A face-to-face dual-tree visualization has sometimes been called a tanglegram or a cophylogeny (shortened to cophylo) plot, but those terms have also been applied to visualizations where the shared leaf nodes may be named (tagged) or ordered differently. That is not the case in this example, where the leaf nodes shared by both hierarchies also observe the same (document) order in both.

Single-root multitree

Figure 3: Single-root multitree layout

A multitree is a type of directed acyclic graph (DAG) that has large easily identifiable substructures that are trees.^[45] In this case the graph has a single root node (Hamlet) and shared leaf nodes, that is, this is a type of graph that is well-suited to visualizing overlapping hierarchies erected over largely shared text.^[46] To improve legibility we have used labeled ellipses (Orth 1, etc.) to represent the newline-delimited input strings, which are listed below the graph itself, with numbers corresponding to the nodes in the graph. Instead of locating the root at the top of the image and the input lines (or their numerical surrogates) at the bottom we could have rotated the graph ninety degrees, in which case we could more easily have written out the text in place, as we did with the face-to-face dual-tree visualization, above. In the earlier example that resulted in rendering the text in the middle of the image; with the single-root multitree the text would be arrayed with one input line above another vertically on one side of the image and a single multitree would be erected in a way that converged on a root node at the opposite side of the image. Whether to render the text in the graph itself or replace it with numbers that point to strings in a legend is independent of the choice between these first and second visualizations. Whatever the orientation of the image, whether we model the input text strings as wrapped in structural Orth nodes (in XML terms, e.g., <Speaker><Orth>GERTRUDE</Orth></Speaker>) or regard the Orth ellipses as visualization stand-ins for the text nodes, and not as parent nodes in the data model (e.g., in XML terms, as <Speaker>GERTRUDE</Speaker>), is a modeling decision, and any use of the visualization would be accompanied by documentation explaining those sorts of decisions. This decision, too, is independent of whether we adopt a face-to-face dual-tree or single-root multitree perspective.

The overlapping edge lines make this visualization inherently harder to read than Figure 2, and to mitigate that effect we incorporate design features that are intended to reduce the cognitive interference that arises when two structures are rendered in the same visual space. Specifically, the two hierarchies are colored differently and the top-level nodes of each (Speech, Line) are grouped on opposite edges of the canvas and enclosed in colored, dashed, rounded rectangles, which helps the viewer understand them as a distinct grouping (that is, hierarchy). Any visual interweaving of the Speech and Line nodes (for example, in an attempt to follow the order of the component text strings) would make it look as if the two hierarchies were ordered with respect to each other, which is not the case; the speeches are ordered only with respect to the order of their component input lines, the same is true of the metrical lines, and speeches and lines are, ultimately, ordered with respect to one another only as an artifact of the ordering of their text strings (or Orth nodes). We might think of this visualization, then, as corresponding to a hypergraph, which we discuss in more detail below.

In the single-root multitree we have somewhat arbitrarily decided not to type the text strings, and we include Speaker nodes to illustrate an alternative we discuss above, where speeches might be modeled as consisting of a Speaker node followed by plain text strings. As we mention above, we could have made a similar hierarchical decision with the face-to-face dual-tree visualization; the meaningful difference between the two visualizations is not the low-level hierarchical modeling, but the rendering of the two hierarchies as discrete (on opposite sides of the text strings in the middle in the face-to-face dual-tree visualization) vs interwoven (with overlapping edge lines in the single-root multitree visualization).

One strength of this visualization is that the single, shared root node is represented visually by a single node. Additionally, as we mentioned above, the face-to-face dual-tree visualization is well-suited to representing two hierarchies but not more than two because one of its principal virtues, the avoidance of overlapping edge lines, becomes unrealistic with three or more independent hierarchies. That is not the case with the single-root multitree visualization, where additional hierarchies would change only the extent of the overlapping. With that said, the overlapping web of lines with even two hierarchies makes this image harder to read than Figure 2, and that difficulty would increase with additional hierarchies.

At the same time, the single-root multitree visualization is better suited to overlapping hierarchies that intersect at locations other than the root and leaf nodes. Our example happens to rely on two completely independent hierarchies, where the only shared nodes are the root and the leaves. A different, more complex overlap situation that had shared nodes at different hierarchical levels could not realistically be represented with the complete visual separation of hierarchies in the face-to-face dual-tree model, and while such a structure might make the single-node multitree harder to read, it would not entail a fundamental contradiction between the model and the visualization.

Hypergraph

A hypergraph visualization of this example might look like:

Figure 4: Hypergraph

In the hypergraph visualization the Orth nodes are gray and are connected by directed edges because their order matters. The blue hyperedges denote metrical lines (note that nodes 1, 8, and 10, which are speaker names, do not have blue circles around them because they are not part of any metrical lines) and the red hyperedges denote speeches (which contain both speaker names and spoken lines). The metrical lines and speeches are not directly ordered because their order is a function of the order of their constituent Orth nodes. We’ve omitted the text of the lines to improve our focus on the nodes, regular edges, and hyperedges, but in a Real Life interpretive visualization we would either replace the numbers in the gray Orth nodes with the speaker names and spoken-line text or add a legend with numbered lines, as we did in the single-root multitree layout, above.

In most places the hypergraph is easy to understand, at least once one has learned its visual language, but the Speech node that connects Orth nodes 9 and 11, bypassing Orth node 10, which is a speaker name, involves visual overlap of the directed regular edges and both types of hyperedges. This is not exactly difficult to understand, but because it contains several types of overlapping information in a small space, we find it difficult to understand at a glance, even once we are comfortable with hypergraph visualizations. Replacing the numbers in the Orth nodes with the textual content, which we mention above, would make it easy to see which node contains exactly which text, but at the cost of undermining the visual consistency of the rounded, numbered circles, and in Real Life we would be inclined to retain just the numbers and attach a legend with the full text, as we discuss and illustrate above.

Modified swimlane

Figure 5: Modified swimlane

A traditional swimlane visualization plots responsibility areas on separate tiers (that is, in separate lanes) and draws directional arrows to illustrate the flow of information along and across tiers.^[47] In the visualization above we plot the content nodes (orthographic input lines) in a top lane, speeches in a middle lane, and metrical lines in a bottom lane. In Real Life we could either replace the numbers in the gray squares with the text of each input line or leave the numbering and attach a legend that maps the numbers to the associated text. As with the hypergraph, discussed immediately above, inserting the text into the squares in place of the numbers would make it easier to see exactly where each unit of text is located, but only at the expense of making the entire visualization busier and less uniform, and therefore ultimately harder to understand. Here, then, in Real Life we would opt for a legend, as we discussed in connection with the hypergraph, above.

We refer to ours as a modified swimlane because:

We use no directional arrows within a lane (except one in the bottom lane, about which see below) because in our visualization information flows from left to right and affects every node within each tier.
We use no directional arrows between lanes because in our visualization information flows across all tiers simultaneously, without the handing-off of responsibilities that characterizes a traditional swimlane representation.^[48]

As with the visualizations above, the orthographic input lines make up the most granular level of information, and the other tiers represent groupings of the input lines. We can contrast this to a tree diagram (such as the first and second visualizations discussed above), where the input lines represent sibling nodes at the same hierarchical level and the groupings on the other tiers represent, hierarchically, parents of each group of siblings, up to a common root. That tree hierarchy, such as it is, is quite flat (that is, both the Speeches and Lines tiers group the Orth tier items directly, without intermediate levels), which, were it not for the discontinuity of the split Line around a Speaker, might invite us to consider trees of that sort as more a representation of ranges, in the LMNL sense, than hierarchies (but see below about discontinuity). Meanwhile, the swimlane visualization conveys ranges in a natural way, although, like the trees, it also requires special handling of discontinuity. If we think of each tier as a set of distinct ranges projected over the same content:

The Speeches tier consists of three speeches, each corresponding to a speaker-name input line (nodes 1, 8, and 10) followed by one or more spoken input lines. Every input line is part of exactly one speech. A speech is colored pink, bordered in black, and numbered.
The Lines tier ignores Speaker input lines because we model Speaker lines as partitions, which means that they are not part of (and therefore not visible to) the Lines hierarchy.^[49] Metrical lines 1–6 and 8 correspond to single input lines. Metrical line 7 contains input lines 9 and 11. The black border shows that the covered orthographic input nodes constitute a single metrical line node, but only the nodes that are part of the metrical line itself have the blue background that we use to designate metrical lines, so in the area corresponding to input line 10, which is not part of the metrical line, the background color shows through a gap between input lines 9 and 11. The dotted gray lines and the dotted gray arrow between orthographic input lines 9 and 11 indicate that they behave as if contiguous, that is, as if there was no input line 10 between them. Insofar as input line 10 is a partition that is part of the Speeches hierarchy, but not the Metrical Line hierarchy, input line 10 effectively does not exist as far as metrical lines are concerned, and for all practical purposes, nothing separates the parts of L7, just as, on a different tier, nothing separates the parts of each of the three speeches.

Selecting a visualization

A visualization tells a story about data; story-telling is selective and organized narration, and not a data dump; our choice of visualization depends on the story we want to tell; and we may have more than one story to tell about the same data. For that reason, an implicit quest for an abstract best visualization would be misguided, and it is also unnecessary because there is no reason not to use multiple representations when we need to communicate different types of information that might be easiest to see through different visualizations.

With that said, for this exact data selection we find the face-to-face dual-tree layout easiest to understand. It has no overlapping lines, although it achieves the clarity only by repeating the root node, which introduces an undesirable visual distortion. Furthermore, the clarity would suffer were the input lines longer, if there were more types of them, or if they were hierarchically richer. For example, the excerpt that we selected here happens to have no stage directions, but elsewhere in the play stage directions may occur between or within speeches and lines, and if they were part of the story we wanted to tell, their incorporation would disturb the relatively simplicity of the visualization. Furthermore, as we mention above, the primary source of clarity for this visualization is its ability to avoid overlapping lines, a task that would become challenging, and perhaps impossible, if the story that we wanted to tell involved more than two hierarchies, a situation that might be conveyed more clearly with the modified swimlane visualization than with any of the others.

For all of those reasons, when we use graphic visualization we prefer to start with our communicate goals, and not with a toolkit. That is, instead of starting by selecting a visualization from a library of known types, we start with what are the stories I want to tell?, moving from there to how can I most effectively communicate those stories visually? We selected our input here with the goal of discussing overlap and discontinuity, and the preceding examples are intended to illustrate different ways in which we might seek to examine that goal through alternative visual perspectives.

Prototype

Mildly context-sensitive grammars (MCSG)

IXML processors generally use context-free grammars (CFG), but, in our implementation, both Schema-Next validation and Invisible-Next recognition or transformation rely instead on mildly context-sensitive grammars. In particular, our support for partial order (interleave) requires functionality that is not available naturally in a CFG and we use a MCSG where the Jing RELAX NG validator uses parsing with derivatives.^[50] We illustrate this functionality below.

Comparison between schema validator and markup generator

How validation and generation are similar and different

The Schema-Next and Invisible-Next technologies described above are similar on both the syntactic and modeling levels, which we elaborate below.

RELAX NG schema validation and IXML processing

Within XML technologies the roles of RELAX NG and IXML do not normally overlap. The normal role of RELAX NG is schema validation, that is, verifying whether an XML document with explicit markup conforms to a RELAX NG schema. The normal role of IXML is to add explicit markup to character-stream input, which is typically either plain text or non-XML structured text, such as JSON. IXML can read XML input as a character stream and it would be possible to use IXML processing to perform most of RELAX NG schema validation, that is, to construct an IXML grammar that implements most of the same modeling as a RELAX NG schema, but it would be peculiar to do so when the task is easier with RELAX NG, which, after all, was designed specifically for that purpose.

Although IXML always recognizes whether a specific document can be parsed against a specific grammar, complications arise with trying to use it to mimic schema validation because it was not developed specifically for that purpose. For example:

Unlike RELAX NG, IXML input processing does not come with a knowledge of XML markup conventions, which means that the IXML grammar, which reads the input as a character stream, would be responsible for distinguishing markup from content.
IXML lacks an interleave operator, which means that modeling unordered and partially ordered content would have to be simulated if expressed through an IXML grammar.
Although IXML is required to return only one path through the forest constructed during parsing, it needs to know whether there is more than one because it is expected to report ambiguity.^[51] Because RELAX NG is about validation, its only responsibility is to determine whether the instance is a document in the language described by the schema. Explicit XML markup describes a single tree, which means that ambiguous paths through a forest are not part of the RELAX NG processing model.
The output tasks of RELAX NG validation and IXML processing diverge, so although IXML processing includes determining whether an input document does or does not conform to an IXML grammar, and whether a well-formed output document can be emitted, parsing failures are not reported in the same way as they are with RELAX NG validation.

Purpose

Much as RELAX NG is intended to be used as a recognizer and IXML, as described in Invisible XML Specification, is intended to be used to enabl[e] authors to write documents and data in a format they prefer while providing XML for processes that are more effective with XML content, Schema-Next validation is intended to be used as a next-generation markup recognizer and Invisible-Next processing is intended to be used to add explicit next-generation markup. The considerations that distinguish RELAX NG processing from IXML do not interfere with our using a shared code base for parsing. Because the goals and behaviors of the two types of processing diverge, some other types of functionality cannot (or need not) be shared, but those differences do not impinge on the shared parsing. For example:

Our code base currently contains tokenizers for LMNL and TexMECs, which are used by our Schema-Next validation. Our Invisible-Next processing does not currently use the built-in tokenizers, so, as with IXML processing, a user-supplied Invisible-Next grammar would be responsible for recognizing markup tokens already in the input. We have not yet explored the extent to which the built-in tokenizers might be useful for Invisible-Next processing of input already tagged in LMNL or TexMECS.
Support for interleave is built into our code base, and therefore equally available to our Schema-Next validation and our Invisible-Next processing.
Our shared code base means that our Invisible-Next processing recognizes ambiguity. That recognition is not needed by our Schema-Next validation, but its presence does no harm.
Schema-Next validation and Invisible-Next processing requires different reporting modules for the results of the two types of operations. Those differences do not interfere with our using shared parsing technology for both.

Document domain

RELAX NG and IXML are designed to model the same basic document domain: a universe that includes either any possible XML document (RELAX NG) or XML that can subsequently be converted to any possible XML document.^[52] Both can describe any type of XML node and node content or value. RELAX NG can be used to validate any XML document and IXML can be used to add XML markup to non-XML input. One difference in what the two technologies can and cannot model is that RELAX NG can describe interleave idiomatically, while IXML, which lacks an interleave operator, would have to simulate it.^[53] There are some differences in syntax between the two technologies, to be sure, and we discuss those below, but the domain of XML documents is substantially the same. Similarly, validation and the insertion of explicit markup in the domain of a next-generation markup like LMNL or TexMECS also use a shared model, and our implementation of a shared code base builds on that common model.

Input and output

RELAX NG incorporates a tokenizer that recognizes XML markup tokens during input, while IXML input tokenization is character based, and the same difference applies to our Schema-Next validation and our Invisible-Next markup generator.^[54] Similarly, IXML incorporates code to create XML markup tokens during output and our code base contains code that knows how to emit LMNL and TexMECS markup tokens.^[55] Currently our code base uses different code for recognizing markup tokens during input and creating them during output; we have not yet explored whether our use of a common code base offers an opportunity for sharing code between input markup recognition and output markup generation.

Grammar and syntax

Both RELAX NG and Schema-Next, on the one hand, and IXML and Invisible-Next processing, on the other, rely on grammar productions that describe nonterminals (LHS) in terms of terminals, other nonterminals (RHS), and operators. Both support rules for elements (or something similar, such as LMNL ranges), attributes (or LMNL annotations), and (sub)patterns (not represented directly by markup tokens). There are, at the same time, conspicuous differences in the syntactic features used for validation and for markup generation. For example:

Although eventual node types during serialization (e.g., elements, attributes) are encoded, through marks on nonterminals, IXML is not required to distinguish among them during parsing. This is why, for example, an IXML grammar can define an attribute node that is the root of a parse tree, and the parsing itself will succeed, although the result cannot be serialized, and will ultimately fail with D05.^[56]
The element and attribute operators in RELAX NG and Schema-Next allow nested structures on the RHS, so that, for example, an entire RELAX NG schema can be expressed as a single nested rule, an organizational pattern known as Russian doll.^[57] For example:
```
start = element doc { 
    element section {
        element p { text }+
    }+
}
```
says that a <doc> root element contains one or more <section> children, each of which contains one or more <p> children, each of which contains plain text. The keyword start identifies the root element as <doc>, which contains, as its RHS, a production for a <section>, which contains, as its RHS, a production for a <p> element. There are no labels for the elements; labeling is used only when a label (not full production rule) used in a content model is, in turn, described in its own separate production. The following schema, which uses labels instead of nested productions, is equivalent to the one above:
```
start = doc
doc = element doc { section+ }
section = element section { p+ }
p = element p { text }
```
Because IXML and Invisible-Next lack the element and attribute operators of the schema languages, the RHS of productions in the IXML and Invisible-Next cannot embed other complete (both LHS and RHS) productions. This means that IXML and Invisible-Next grammars are necessarily flat, while the degree of nesting in a RELAX NG or Schema-Next schema is under the control of the developer.
IXML and Invisible-Next support the ++ and ** repetition-with-separator operators, which are useful for resolving modeling that might otherwise be challenging to express unambiguously.^[58] These operators are not used in Schema-Next, but our code base needs to recognize them to support Invisible-Next.^[59]
Although both RELAX NG (and Schema-Next) schemas and IXML (and Invisible-Next) processing might appear to support renaming elements and attributes, the similarity is superficial. Specifically, labels used on the LHS of compact-syntax productions (e.g., the para in RELAX NG para = element p { text }) are similar to variable name bindings, which are relevant only in pre-compilation source. They are never part of the compiled RELAX NG grammar and therefore are not subject to renaming. While to a user it might appear that a para in the content model is being rewritten as a p, an LHS label is a feature only of the compact syntax, and not of the tree.^[60]

When parsing an IXML (or Invisible-Next) grammar, on the other hand, the name of a node in the parse forest is the name of the grammar production that produced it. In IXML 1.0 that name value is part of the serialization of the node. IXML 1.1 permits renaming (e.g., A>B: "x"), but the name associated with the node in the parse forest is the name before renaming, and the name after renaming is a feature of serialization. We can see this when we look at the Coffeepot output of the result of compiling an IXML grammar that uses renaming. For example, the IXML grammar:
```
doc: A.
A>B: "x".
```
compiles to:
```
1.  $$ ::= doc
2. doc ::= A
3.   A ::= 'x'
```
When we parse a document with the value x against this grammar, it produces:
```
<doc>
   <B>x</B>
</doc>
```
Note that although the element name B is used during serialization (and is therefore associated with A in some way), the name of the node in the grammar after compilation is A.^[61]

The difference between the RELAX NG (and Schema-Next) vs IXML (and Invisible-Next) behavior may appear superficial to an end-user, who might reasonably understand both as sharing the meaning of call it A in an LHS but spell it B in the output. The difference nonetheless matters for application development because an IXML (and Invisible-Next) renamed node has (in the language of Invisible XML Specification (current draft) quoted above) one name in the parse forest and a different name in serialization, while an apparently renamed node in a RELAX NG (and Schema-Next) schemas has the same name after parsing as it will use when it performs validation.

The following are apparent or historical differences, and in these cases RELAX NG and IXML have similar capabilities, even though those may be (or have previously been) expressed differently syntactically:

RELAX NG and Schema-Next allow multiple start values, e.g.,
```
start = element start { A | B}
```
IXML and our Invisible-Next system support this same feature through suppressing the serialization of an element node for the root of the grammar, as in:
```
-start: A; B.
A: "a".
B: "b".
```
Repetition indicators are greedy in RELAX NG, while in IXML they they match all possible patterns, and not just the longest (greedy) or shortest (lazy or reluctant).^[62] This is a common source of horizontal ambiguity.^[63] The addition of support for greedy matching of repeated patterns is under active discussion within the IXML Community Group.

What those similarities and differences mean for tool development

Our approaches to schema validation (Schema-Next) and to adding markup (Invisible-Next) are sufficiently similar that we found it useful to explore the development of a shared data model and a shared code base. Below we discuss where Schema-Next processing and our Invisible-Next processing are similar and where they are different, as summarized in the following data-processing pipeline:.

Figure 6: Data-processing pipeline for our Schema-Next and Invisible-Next applications

The visual language of the image above is:

Topology expresses the computation flow
Shape expresses the kind of artifact
- Document: input or output document
- Parallelogram: non-document result (output or error)
- Rounded rectangle: process
- Unrounded rectangle: interim data structure
Color expresses subsystem ownership
- Blue: schema validation
- Green: markup generation
- Lavender: shared
Subgraphs express conceptual grouping
Typography expresses hierarchy (uniform font and typesize, except for larger labels on subgraph groups)

Parsing user-supplied schema or grammar (shared)

The first processing step reads a Schema-Next schema or Invisible-Next grammar and parses it into a high-level grammar model that represents the features of both types of input in a unified way. Both a Schema-Next schema and an Invisible-Next grammar describe expectations for nodes of different types by using character classes and ranges and high-level operators (and (group), or (alternatives), oneOrMore (optional), zeroOrOne (optional) zeroOrMore (optional), interleave, concur, partition, etc.). Invisible-Next separators ** and ++ are rewritten directly as zeroOrMoreWithSeparator and oneOrMoreWithSeparator. This stage translates reserved-character notation, which would be challenging for the eventual low-level parser, into keyword operators. The intelligent interpretation of those keyword operators happens later, as we describe below.

High-level schema or grammar model (shared)

The output of the preceding stage is a high-level description of the types of documents that are part of the language, whether for recognition and validation (Schema-Next) or for generation (Invisible-Next). This step abstracts away syntactic differences between Schema-Next schemas and Invisible-Next grammars. The operators in the input schema or grammar become nodes in the tree of expectations (expression trees), alongside elements and attributes.

Schema or grammar compiler (shared)

The preceding step is compiled down into an MCSG with a set of linear production rules (no operators and no node definitions). The model for this low-level grammar is implemented with optional features, which is to say that it’s a shared model, but which optional features are used depends on whether we are performing validation or adding markup. Specifically:

For Schema-Next validation we know to expect markup tokens (e.g., start- and end-tokens), while for Invisible-Next processing there is no incoming markup.
For Invisible-Next, processing rules have properties that define whether they should result in serialization as elements, attributes, or patterns.^[64] For Schema-Next processing there is no output serialization.

Tokenizer for tagged input (Schema-Next validation only)

The tokenizer divides the input stream into markup tokens and text tokens, based on built-in tokenizers for (currently) LMNL (MNML version) and TexMECS.^[65] Input for Invisible-Next processing does not have markup tokens, and therefore does not use the tokenizer module.^[66]

Parser to parse input file against compiled schema or grammar (shared)

The input is a character stream for Invisible-Next processing and the result of the tokenization step (above) for schema validation. Both use an MCSG and produce:

A GSS (graph-structured stack, also called a multistack) of expectations that stores information about concurrency, interleave, and, just for the generator, ambiguity.
A parse forest. Currently, this is used only for Invisible-Next processing, but it might be used for Schema-Next validation in the future to enhance error messaging based on information available when invalidity or other types of errors are detected.

The validator ends here.

Serializer (Invisible-Next processing only)

The serializer serializes and emits one or more paths through the parse forest, with next-generation features. We currently have serializers for LMNL, TexMECS, and XML^[67]

Discussion

As this section has shown, despite the patent differences in the XML ecosystem between the roles of schema validation, on the one hand, and the addition of explicit markup, on the other, the tasks have much in common. Those similarities create an opportunity to develop tools that rely on a shared data model and a largely shared code base.

There is, at the same time, good reason that the XML toolkit does not include validation and markup-generation tools that employ a shared data model and code base. These include:

Validation and markup generation are very different tasks. Those genuine and profound differences may have made it difficult to ask what the two have in common and whether they might be approached in a coordinated way.
The emergence of RELAX NG and the robust and highly performant Jing implementation thereof makes XML schema validation largely a solved problem. IXML is a much younger technology, but it has already acquired several robust and highly performant implementations, and while the language continues to evolve, tooling to support it can also reasonably be regarded as a settled problem—or, at least, a problem that is well on course toward being settled. As a result, there is little reason to deviate from established approaches to RELAX NG and to IXML; both are well served by the available approaches and the available implementations of those approaches.

The same cannot be said for next-generation markup processing, which is very much an emerging and not-yet-settled field. Validation and markup generation for next-generation languages are necessarily new, which encourages new approaches in a way that does not obtain, at least to the same extent, in the case of the far more mature XML technologies.
In the XML ecosystem, the support for interleave in RELAX NG, but not in IXML, leads to the use of different underlying grammar technologies, a context-free string grammar for IXML and parsing with derivation, using a tree grammar, for RELAX NG. Our decision to approach both tasks with an MCSG removes this fundamental difference and makes it easier to see what the specific MCSG implementations might have in common.

Conclusions

The impetus for this paper was our interest in exploring what Invisible-Next processing might look like for next-generation markup languages. We approached this question by starting from the MCSG concept that informed Haentjens Dekker et al. 2025 and applying it to Invisible-Next processing.

Our observations during the course of these explorations included the following:

Our work with Hamlet took us through identifying overlap, creating independent IXML grammars for the two hierarchies, exploring the consequences of attempting to merge those hierarchies, and creating IXML grammars that were capable of recognizing and serializing LMNL and TexMECS markup of the overlap. These activities encouraged us to recognize the value of being able to model overlap as concurrency, rather than as ambiguity, and of the ability of processing with an MCSG to represent concurrency in a more robust and scalable way than IXML.
In our work with Hamlet we also identified discontinuity that was caused by a partition, that is, a portion of text that should be visible to only one hierarchy. Not all discontinuity necessarily involves partitions, but our experiment showed that we were able to specify a partition in a grammar and apply the relevant grammar rules during parsing to create appropriate markup. This part of the experiment was limited to TexMECS output because LMNL (and XML) do not have markup delimiters that distinguish discontinuous components unambiguously.^[68]
We found that although our Invisible-Next syntax borrows, to the extent possible, from IXML, non-XML concepts in next-generation markup required the addition of new operators (for unordered content [that is, interleave] and concurrency) and marks (for partitions and unordered output).
Although not all of the details have been implemented fully in the current version of our prototype (see below, under Potential goals), we confirmed that a MCSG can represent all next-generation features that we targeted, that is, concurrency, discontinuity, and unordered content.
Invisible-Next describes concurrent hierarchies independently of each other. IXML can approximate this type of modeling by using ambiguity, but only up to a point. Specifically, in IXML all overlapping hierarchies must parse all content, and we would then have to use the IXML - operator to suppress content items in the hierarchy where they should not be present. With a partition-aware MCSG grammar, on the other hand, partitions are recognized during parsing and the parse forest itself distinguishes hierarchies where a partition is included from those where it is excluded. The Invisible-Next approach thus matches the mental model of a partition as content that is not part of some hierarchies, rather than content that is present but not serialized.
We explored four visualization strategies for next-generation markup involving concurrency, discontinuity, and partitions. Although no two-dimensional visualization can fully, clearly, and idiomatically express all aspects of a next-generation document, our experiment provided a context for assessing how the four methods tell stories about our sample document in similar but different ways.
While we understood from the beginning that there was likely to be reusable code as we moved from validation to markup generation, the extent of the opportunities for reuse was unexpected, and led us to regard the potential for shared modeling and development as an additional goal of our explorations.

Potential goals for further work include:

Although we were not able to complete experiments with Invisible-Next parsing that already contains markup (whether XML or Markup-Next), we began theorizing about how our ongoing work might address that type of task. Existing XML documents that used workarounds for overlapping or discontinuous structures are likely candidates for Invisible-Next processing that recognizes the existing XML markup while parsing and emits Markup-Next that supports those structural features directly, without recourse to workarounds.
LMNL, TexMECS, and XMLs have different serialization capabilities (for example, only TexMECs can represent unordered content and discontinuity with markup tokens), and how to represent features in a serialization that lacks direct, syntactic markup delimiters for them requires further study. Our implementation already includes serializers for all three, but we have not yet implemented a strategy for selecting a serialization method at run-time in our command-line application.
Our Schema-Next language also models self-overlap, that is, overlaps of multiple elements of the same type. This sort of phenomenon is well known from annotation applications (such as https://web.hypothes.is/), where users might highlight overlapping strings of text and associate annotations with them. In our Schema-Next language we adopted the Creole keywords concurOneOrMore and concurZeroOrMore for that purpose, with no associated operators.

We have not yet explored the recognition and management of self-overlap and self-discontinuity in Invisible-Next. We anticipate being able to engage with these document-structure features once we have had the opportunity to develop additional experiments.
Our current implementation recognizes partitions in one branch of a concurrent structure, but not in both, although partitions in multiple branches are possible, especially as the number of concurrent hierarchies grows. That limitation results from our not yet having fully implemented our management of concurrency, partitions, and sets. These responsibilities should be located entirely in the compiler and the generated MCSG, but currently, because our compiler is not yet complete, our GLL parser is required to deal directly with some next-generation features. The current incorrect location of some of these operations in the GLL parser impinges on the modularity of the system (that is, the ability to change compiler and compiled grammar decisions) because our MCSG is not yet fully linearized. To remedy these limitations we are currently working on relocating these features out of the GLL parser and into the compiler.
A complete next-generation tool stack requires transformation and query languages, comparable in their roles to XSLT and XQuery in the XML world. We have not yet begun designing these languages, but they remain part of our comprehensive plan for supporting next-generation markup. Because transformation and query languages work with the data model, and not the serialization, the data model must be consistent with the mental model of the document if those languages are to be able to operate over it in a way that supports transparent reasoning. This consideration motivates, for example, our decision to prioritize excluding representations that cannot be serialized (such as duplicate attribute names on an element instance in XML) from the parse result (that is, the data model), instead of regarding them fundamentally as serialization errors, as Invisible XML Specification does.

Appendix A. Grammar terminology

A grammar is expressed as a set of rules (called productions) and map from a left-hand-side (LHS) to a right-hand-side (RHS). For example, the IXML production:

article: metadata, content.

says that an article can be matched by a sequence, in the input, of whatever matches a metadata followed by whatever matches a content.

The purpose of a production is to define the LHS according to the RHS. A grammar distinguishes nonterminals, like article, metadata, and content in the example above, from terminals. Terminals are literal values or reserved words that stand in for literal values, and that therefore do not need to be interpreted in their own productions. Nonterminals, on the other hand, require explanation by means of productions. All nonterminals must expand eventually (that is, by expanding the nonterminals in the RHS of their productions recursively) to terminals.

Syntactic operators that represent repetition, optionality, and alternation are common features of higher-level, human-facing grammars, such as XML schema languages and IXML. A lower-level grammar can be compiled for more convenient machine processing by rewriting the productions to remove those operators, and also introduce epsilon (ε), a terminal that signifies that that portion of a rule can be matched against the empty string. We discuss what we mean by higher and lower level in the Schemas and grammars section of Haentjens Dekker et al. 2025.

A context-free grammar (CFG) has a single symbol on the LHS and a single sequence of terminal and nonterminal symbols on the RHS.

A mildly context-sensitive grammar (MCSG) also has a single symbol on the LHS, but it may have two sequences on the RHS. The relationship between the two sequences on the right is similar (although not identical) to RELAX NG interleave. RELAX NG interleave is often understood as all of these items in any order, but in addition to that type of unordered content, interleave is also designed to model partially ordered content. See the section titled Partially ordered content: how Interleave works in Haentjens Dekker et al. 2025 for details. An MCSG rule can be matched only if all parts of both sequences on the RHS are matched in a way that conforms to the ordering of each of the individual sequences, and the rule may incorporate an ordering constraint that describes how the two sequences can be interleaved.

Creole is a proposed schema language for next-generation markup languages, originally designed within the framework of the LMNL project. It is described in Tennison 2007.

In the Schemas and grammars section of Haentjens Dekker et al. 2025 we distinguished the term schema, for productions authored by users (as in a RELAX NG schema), and grammar, for a constrained set of productions, optimized for machine interpretation and processing, that is generated programmatically by compiling the schema into a grammar. While schema design prioritizes human legibility, grammar design prioritizes efficient computational implementation. That distinction is comparable to the one in Norm Tovey-Walsh 2023 between Extended Backus-Naur forms (EBNF) and Backus-Naur forms (BNF), which explains that BNF grammars are simpler to reason about and simpler for implementations to process. EBNF grammars are easier to author. The How it works section of Coffeepot similarly distinguishes an Invisible XML grammar, authored by a human with EBNF-like syntax and comparable to what we called a schema, and the Earley grammar into which it is compiled, expressed with BNF-like syntax and comparable to what we called a grammar. Because the established term in Invisible XML Specification for the set of productions authored by a human is not schema, but grammar, for the present report we modify the terminology from Haentjens Dekker et al. 2025 and, in this discussion of Invisible-Next processing, we use the terms Invisible-Next grammar (instead of schema) and compiled grammar (instead of just grammar).^[69]

Appendix B. Failure vs error in IXML processing

There are four possible types of outcome of an IXML operation, with the following behaviors described in Invisible XML Specification:

Success in parsing the grammar, parsing the input against the grammar, and serializing a parse result returns well-formed XML according to the rules of the grammar.^[70]
Failure to parse the grammar is a static error, with the error number reflecting the reason for the problem. Except for the error number, no specific returned result, messaging, or exit code is described.
Success in parsing the grammar but failure to parse the input against it is not described as an error condition. It returns XML with an @ixml:state attribute that includes the string failed. The root element and content of the XML result are not described, except that the specification suggests optional helpful content. Because this case is not regarded as an error, no error number is described.
Success in parsing the grammar, success in parsing the input against it, but failure to serialize the parse result as well-formed XML is a dynamic error, with the error number reflecting the reason for the problem. Except for the error number, no returned result, messaging, or exit code is described.^[71]

We identify three places in which IXML and Invisible-Next processing might fail to emit a tagged version of the output:

The grammar cannot be parsed. The requirement that IXML grammars be valid according to the rules described in Invisible XML Specification is similar to the requirement that RELAX NG schemas be valid according to the rules described in the RELAX NG specification.
The grammar itself can be parsed but the input cannot be parsed successfully against it. This is similar to the use of traditional XML schema validation to determine whether documents do or do not conform to structures represented in a schema, but see also the more detailed exploration of failure vs error in IXML below.^[72]
The grammar itself can be parsed and the input can be parsed against it, but the result cannot be serialized as well-formed XML.^[73]

The first and third of the preceding types of failure are regarded as IXML errors, but the second, perhaps surprisingly, is not. An IXML processor must report the failure, but there is no associated numbered error message. Invisible XML Specification says:

If the parse fails, some XML document must be produced with IXML:state on the document element with a value that includes the word failed. The document should provide helpful information about where and why it failed; it may be a partial parse tree that includes parts of the parse that succeeded.

Meanwhile, the failure to parse an XML document against a RELAX NG schema that is, itself, valid is regarded as an error in RELAX NG; see, on this topic, the section entitled Error handling in Clark 2002. We find this discrepancy between what IXML and RELAX NG consider to be an error surprising and confusing for the following reason.

The primary purpose of IXML processing, as described in Invisible XML Specification, is:

Invisible XML (IXML) is a method for treating non-XML documents as if they were XML, enabling authors to write documents and data in a format they prefer while providing XML for processes that are more effective with XML content.

The computational pipeline that implements this functionality has three steps:

Read and parse an IXML grammar.
Read and parse an input document against that grammar.
Serialize the result of parsing the input document against the grammar as well-formed XML by writing explicit markup into the parse results according to rules specified in the grammar.^[74]

IXML treats failure in the first of these three stages (cannot parse and interpret the IXML grammar) as a static error, it treats failure in the third stage (cannot emit well-formed XML) as a dynamic error, but it treats failure in the second stage as Not An Error—that is, Invisible XML Specification does not refer to failure as an error and it does not associate a numbered error message with this failure. We find this behavior unintuitive because:

When IXML is used for its typical purpose of interpreting and processing non-XML input as XML, a lack of success in any of the three stages equivalently prevents the process from satisfying that purpose. A user might reasonably focus on the common frustrated outcome and not expect some lack-of-success situations to be modeled as errors and others as non-error failures.
When IXML processing is used only as a recognizer (a use that is not, by the way, described in Invisible XML Specification as a goal of the technology), users might expect RELAX-NG-like recognizer behavior, and therefore error reporting (not just failure reporting) if the input document cannot be recognized as matching the grammar that is used to (attempt to) interpret it.

For these reasons, our Invisible-Next application regards failure to parse an input document against a supplied grammar as a dynamic error.

References

[Bethan Tovey-Walsh 2026] Tovey-Walsh, Bethan. Document on ambiguity. 2026-02-04. https://lists.w3.org/Archives/Public/public-ixml/2026Feb/0040.html

[Brabrand et al. 2010] Brabrand, Claus, Robert Giegerich, and Anders Møller. 2010. Analyzing ambiguity of context-free grammars. Science of computer programming, v. 75, n. 3, pp. 176–91. https://cs.au.dk/~amoeller/papers/ambiguity/. doi:https://doi.org/10.1016/j.scico.2009.11.002

[Brüggemann-Klein 2020] Brüggemann-Klein, Anne. Four basic building principles (patterns) for XML schemas. Presented at Balisage: The Markup Conference 2020, Washington, DC, July 27 - 31, 2020. In Proceedings of Balisage: The Markup Conference 2020. Balisage Series on Markup Technologies, vol. 25 (2020). doi:https://doi.org/10.4242/BalisageVol25.Bruggemann-Klein01

[Clark 2002] Clark, James. An algorithm for RELAX NG validation. 2002. https://relaxng.org/jclark/derivative.html

[Coffeepot] CoffeePot An Invisible XML processor. Version 3.3.11 https://docs.nineml.org/current/coffeepot/. Accessed 2026-07-04.

[Furnas and Zacks 1994] Furnas, George W. and Jeff Zucks. Multitrees: enriching and reusing hierarchical structure. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '94), 330–36. doi:https://doi.org/10.1145/191666.191778

[Haentjens Dekker and Birnbaum 2017] Haentjens Dekker, Ronald, and David J. Birnbaum. It's more than just overlap: Text As Graph. Presented at Balisage: The Markup Conference 2017, Washington, DC, August 1 - 4, 2017. In Proceedings of Balisage: The Markup Conference 2017. Balisage Series on Markup Technologies, vol. 19 (2017). doi:https://doi.org/10.4242/BalisageVol19.Dekker01

[Haentjens Dekker et al. 2018] Haentjens Dekker, Ronald, Elli Bleeker, Bram Buitendijk, Astrid Kulsdom and David J. Birnbaum. TAGML: A markup language of many dimensions. Presented at Balisage: The Markup Conference 2018, Washington, DC, July 31 - August 3, 2018. In Proceedings of Balisage: The Markup Conference 2018. Balisage Series on Markup Technologies, vol. 21 (2018). doi:https://doi.org/10.4242/BalisageVol21.HaentjensDekker01

[Haentjens Dekker et al. 2025] Haentjens Dekker, Ronald, David J. Birnbaum, Bram Buitendijk and Joris J. van Zundert. A schema language and parser for next-generation markup languages. Presented at Balisage: The Markup Conference 2025, Washington, DC, August 4 - 8, 2025. In Proceedings of Balisage: The Markup Conference 2025. Balisage Series on Markup Technologies, vol. 30 (2025). doi:https://doi.org/10.4242/BalisageVol30.HaentjensDekker01

[Huitfeldt and Sperberg-McQueen 2003] Huitfeldt, Claus, and C. M. Sperberg-McQueen. TexMECS: An experimental markup meta-language for complex documents. Working paper of the project Markup Languages for Complex Documents (MLCD), University of Bergen. January 2001, rev. October 2003. http://mlcd.blackmesatech.com/mlcd/2003/Papers/texmecs.html

[Invisible XML Specification] Invisible XML Specification. Editor: Steven Pemberton, CWI, Amsterdam. Version: 2022-06-20. https://invisiblexml.org/1.0/

[Invisible XML Specification (current draft)] Invisible XML Specification. Editor: Steven Pemberton, CWI, Amsterdam. Community Group Editorial Draft, 2026-02-03. https://invisiblexml.org/current/

[Jing] Clark, James. Jing. A RELAX NG validator in Java. 2001–08. https://relaxng.org/jclark/jing.html

[Jinks 2004] Jinks, Pete. Notations for context-free grammars. https://www.cs.man.ac.uk/~pjj/bnf/bnf.html

[Kunth 1968] Knuth, Donald E. 1968. Semantics of context-free languages. Mathematical systems theory. 2 (2): 127–145. Archived copy at https://web.archive.org/web/20200519113619/https://www.csee.umbc.edu/courses/331/fall16/01/resources/papers/Knuth67AG.pdf. doi:https://doi.org/10.1007/BF01692511

[Kunth 1990] Knuth, Donald E. The genesis of attribute grammars. Proceedings of the international conference on Attribute grammars and their applications (1990), LNCS, vol. 461, 1–12. https://www.cs.miami.edu/home/odelia/teaching/csc419_spring17/syllabus/Knuth_AttributeHistory.pdf doi:https://doi.org/10.1007/3-540-53101-7_1

[La Fontaine 2021] La Fontaine, Robin. Element order is always important in XML, except when it isn’t. Presented at Balisage: The Markup Conference 2021, Washington, DC, August 2–6, 2021. In Proceedings of Balisage: The Markup Conference 2021. Balisage Series on Markup Technologies, vol. 26 (2021). doi:https://doi.org/10.4242/BalisageVol26.LaFontaine01

[Norm Tovey-Walsh 2023] Tovey-Walsh, Norm. Ambiguity in iXML: and how to control it. Presented at Balisage: The Markup Conference 2023, Washington, DC, July 31–August 4, 2023. In Proceedings of Balisage: The Markup Conference 2023. Balisage Series on Markup Technologies, vol. 28 (2023). doi:https://doi.org/10.4242/BalisageVol28.Tovey-Walsh01

[Norm Tovey-Walsh 2025] Tovey-Walsh, Norm. Writing Invisible XML Grammars. December 16, 2025. https://www.xml.com/articles/2025/12/16/invisible-xml-grammars-update/

[Pemberton 2026] Pemberton, Steven. Invisible XML case studies. Presented at XML Prague, 2026-06-06. https://homepages.cwi.nl/~steven/Talks/2026/06-06-prague-tutorial/

[Piez 2026 Balisage] Piez, Wendell. Scholia 2026: DIY study aids with TEI, XProc, browser and printer. Presented at Balisage: The Markup Conference 2026, Washington, DC, August 3 - 7, 2026. In Proceedings of Balisage: The Markup Conference 2026. Balisage Series on Markup Technologies, vol. 31 (2026). doi:https://doi.org/10.4242/BalisageVol31.Piez01

[Piez 2026 Symposium] Piez, Wendell. Overlap meets Invisible XML. Presented at The First International Symposium on Invisible XML, 26/27 February 2026. https://github.com/wendellpiez/Laminator/tree/main/papers/iXMLSymposium2026

[Rummler and Brache 2013] Rummler, Geary A. and Alan P. Brache. 2013. Improving performance: How to manage the white space on the organizational chart. Third edition. San Francisco: Jossey-Bass.

[Sperberg-McQueen 2010] Sperberg-McQueen, C. M. 2010. (Formal model for MOC catalog in Alloy.) http://mlcd.blackmesatech.com/mlcd/2010/A/moc-catalog.als

[Sperberg-McQueen and Huitfeldt 2000] Sperberg-McQueen, C. M., and Claus Huitfeldt. 2000. GODDAG: a data structure for overlapping hierarchies. Paper given at Digital Documents: Systems and Principles. 8th International Conference on Digital Documents and Electronic Publishing, DDEP 2000, 5th International Workshop on the Principles of Digital Document Processing, PODDP 2000, Munich, Germany, September 13–15, 2000. Published in DDEP-PODDP 2000, ed. P. King and E.V. Munson. Lecture Notes in Computer Science 2023. Berlin: Springer, 2004, pp. 139–60. Available on the Web at http://www.w3.org/People/cmsmcq/2000/poddp2000.html. doi:https://doi.org/10.1007/978-3-540-39916-2_12

[Sperberg-McQueen and Huitfeldt 2008] Sperberg-McQueen, C. M., and Claus Huitfeldt. Containment and dominance in Goddag structures. Processing Text-Technological Resources, Bielefeld, March 2008. https://cmsmcq.com/2008/bielefeld/slides.html

[Sperberg-McQueen and Huitfeldt 2008a] Sperberg-McQueen, C. M., and Claus Huitfeldt. Markup Discontinued: Discontinuity in TexMecs, Goddag structures, and rabbit/duck grammars. Presented at Balisage: The Markup Conference 2008, Montréal, Canada, August 12 - 15, 2008. In Proceedings of Balisage: The Markup Conference 2008. Balisage Series on Markup Technologies, vol. 1 (2008). doi:https://doi.org/10.4242/BalisageVol1.Sperberg-McQueen01

[Tennison 2007] Tennison, Jeni. Creole: Validating Overlapping Markup. Presented at XTech 2007. https://www.princexml.com/howcome/2007/xtech/papers/output/0077-30.pdf

[Tennison 2008] Tennison, Jeni. Overlap, containment and dominance. Jeni’s musings, 2008-12-06. http://www.jenitennison.com/2008/12/06/overlap-containment-and-dominance.html

[Tennison and Piez 2002] Tennison, Jeni and Wendell Piez. The Layered Markup and Annotation Language (LMNL). Presented at Extreme Markup Languages® 2002 (Montréal, Canada). 2002.

[Verwer 2026] Verwer, Nico. Between invisible XML and visible XML. Presented at The First International Symposium on Invisible XML, 26/27 February 2026. https://invisiblexml.org/events/symposium2026/slides/Between_Invisible_XML_and_Visible_XML_-_Nico_Verwer.pdf

[xMNML grammar] Piez, Wendell. xMNML grammar. https://github.com/wendellpiez/Laminator/blob/c9b5566c3529bbf6de4e1f804cb02d32793c8ae9/lib/xMNML/rules/xMNML.rnc. Accessed 2026-09-03.

^[1] We are grateful to Bethan Tovey-Walsh for her generous responses to some of the ideas we discuss here, for providing some of the IXML grammars that we used as we developed those ideas, and for her careful and constructively critical reading of an earlier version of this paper.

^[2] For a discussion of when ambiguity might be a desirable feature of an IXML grammar see Bethan Tovey-Walsh 2026.

^[3] See “Challenges for text modeling” in Haentjens Dekker and Birnbaum 2017.

^[4] See “Semantics versus application level” in Haentjens Dekker and Birnbaum 2017.

^[5] We employ, somewhat arbitrarily, the spelling IXML (instead of ixml or iXML). Invisible XML Specification does not recommend a spelling and uses both primarily ixml, but also IXML in one location. The IXML portal at https://invisiblexml.org/ uses (in text that does not cite publication or application titles) primarily ixml, but also iXML in one place. The latest working draft of the specification (Invisible XML Specification (current draft)) uses ixml consistently, except for IXML in one place.

^[6] The input to an IXML process may be plain text or it may include characters that function as markup in an alternative structured markup language, such as JSON. Cf. the examples in Invisible XML Specification of input that may be not only plain text, but also formats such as JSON, CSS, URLs, or structured mathematical notation.

^[7] The terms recognizer and validator can be used similarly, but we employ the former to emphasize a Boolean outcome (successful or unsuccessful interpretation of the input according to a grammar) and the latter to point to the possibility of more detailed messaging, in case of failure, that can help a user locate and redress mistakes in either the input or the grammar.

^[8] For example, the system described in Piez 2026 Symposium reads and parses a document with LMNL input and emits a document with XML markup.

^[9] Our perspective on assumed correct vs check whether correct is informed by Pemberton 2026, which distinguishes:

Permissive grammars: we assume the input is correct
Strict grammars: we check that the input is correct.

^[10] Invisible XML Specification gives the following example:

An exclusion is an inclusion preceded by a tilde ~. For example, ~["{}"] matches any character that is not an opening or closing brace.

^[11] RELAX NG does not permit interleave that shares operands. For example, the following RELAX NG schema:

start = doc
doc = element doc { a & b }
a = x, y
b = x, z
x = element x { text }
y = element y { text }
z = element z { text }

is not valid because a and b both include x. In keeping with the spirit of IXML, which permits ambiguity, we allow this pattern, which can be matched by

x, x,
                        y, z

and others. In the case of x, x, y, z there is structural ambiguity concerning which x matches which of a and b with no way to decide, that is, with two paths in the parse result that happen to be serialized identically.

^[12] What happens in case the LHS of a rule asks for unordered content in the output and the output language is not designed to represent ordered vs unordered content syntactically is an open question that requires further study.

^[13] IXML defines three marks: ^ for nodes to be serialized as elements (this mark is optional because serialization as an element is the default), @ for nodes to be serialized as attributes, and - for nodes to be removed from the serialization and replaced by their children. Our selection of ^^ to represent a partition reflects our understanding that partitions are element-like.

^[14] It is unfortunate that Invisible XML Specification allows alternative operators for the same concept, specifically : and = to separate the LHS and RHS of a production and | and ; for an or-group. The provision of alternatives does not contribute to the expressivity of the language and it reduces the inventory of operators that might be available for other purposes.

^[15] We base this example on the one from Peer Gynt in Sperberg-McQueen and Huitfeldt 2008.

^[16] In our analysis, the fact that a speech contains multiple input lines is not significant for the speech hierarchy, but if we should later decide otherwise, nothing prevents us from tagging the input lines as, for example, Orth nodes (representing orthographic input lines) and connecting the speech nodes to Orth nodes, which, in turn, are connected to the leaf nodes, instead of connecting the Speech nodes directly to the textual leaves.

^[17] Because Creole was developed within the range-oriented LMNL framework, Tennison 2007 uses 1) range where we use element and 2) element where we use partition.

^[18] The complete overlap of physical (pages and lines) and content-based (chapters and paragraphs) may involve simplification, at least with documents that have running headers or page numbers or other bits of text that can reasonably be considered part of the page hierarchy (and perhaps even an important part) but not of the content-based hierarchy. An editor might work around that issue by encoding running headers or page numbers as attribute values, but that translates what is plainly physical data (written digits) out of the textual leaf nodes. That translation may make good analytic sense, but every character written on the page is part of the content of (not metadata about) the page is also a defensible interpretation. From that perspective, we might reasonably regard the page numbers as partitions that are visible only to the page hierarchy, but not to the content-based hierarchy.

^[19] We could, to be sure, choose to model competitive, interruptive speech as an overlap of, say, <abbott> and <costello> elements, but a more accurate representation would capture the fact that the overlap is between elements of the same type (e.g., <speech>), and the identity of the speakers is metadata about the individual speeches, and not an aspect of their type-level ontology.

^[20] There are, to be sure, lines in the Shakespeare drama corpus that are not iambic pentameter. Some are in other verse meters, such as the famous trochaic tetrameter witches’ chant (Double, double, toil and trouble) in Macbeth; others cannot be parsed as any regular meter, such as Dogberry’s speeches in Much ado about nothing. The robust recognition of all types of structural lines, which would be a requirement for an application that could recognize and tag such lines, is not a natural task for IXML alone.

^[21] For example, English react has two syllables but reach has one.

^[22] No other structural modifications to the two single-hierarchy grammars are required, except ensuring that no nonterminal is defined more than once. In our example the only nonterminals defined in both single-hierarchy grammars are defined identically, which means that we can satisfy the requirement in question by removing the duplicates.

^[23] Concerning MNML see Piez 2026 Symposium and Piez 2026 Balisage.

^[24] It is theoretically possible for a line of iambic pentameter to end with a monosyllable as the eleventh syllable, which means that we would have 10word 11word at the end of the input line, which our grammar cannot match. As a simplification in the interest of clarity we omit that possibility here.

^[25] An IXML processor cannot serialize concurrency because it is restricted to serializing XML.

^[26] https://github.com/wendellpiez/Laminator/blob/c9b5566c3529bbf6de4e1f804cb02d32793c8ae9/sources/Sonneteer/sonnets/ozymandias.lmnl

^[27] Historical publications of the poem diverge in their representation of the quotations (and in other small ways). For example:

The use of quotation marks in Shelley’s 1817 handwritten fair copy draft matches Piez’s text. (https://en.wikipedia.org/wiki/Ozymandias#/media/File:Ozymandias_Shelley_draft_c1817.gif)
The first publication, in the British weekly The Examiner, opens quotation marks on line 2 and never closes them. It also wraps paired quotation marks around line 10 (only), although line 11 is also plainly part of the embedded quotation. (https://en.wikipedia.org/wiki/Ozymandias#/media/File:Ozymandias_The_Examiner_1818.jpg)
The 1819 edition wraps quotation marks around the inner quotation (both lines) but does not use quotation marks for the outer quotation, which simply begins after a colon on line 2. (https://babel.hathitrust.org/cgi/pt?id=dul1.ark:/13960/t1bk2mq33&view=1up&seq=106)

^[28] The poem consists of a combination of sentences (s) and quotes (q) in any order. This particular poem is a single sentence, but the rule for sonnet allows for other natural top-level linguistic organization.

Quotes contain one or more sentences. In Real Life quotes may contain other structures, such as phrases and other quotes, but our grammar has been artificially constrained to match the structure of this particular poem. We adopted this compromise in the interest of legibility; our focus is on navigating overlapping hierarchies and not on modeling any and all possible linguistic structures.

Phrases (phr) have optional initial whitespace (spaces and newlines) followed by plain text (defined as an initial letter, then any combination of letters and whitespace, and then a final letter). This definition helps manage interphrase spacing in the input, and the requirement that phrases end in a letter lets us match phrase-final punctuation (pPunc) and sentence-final punctuation (sPunc) separately.

Sentences (s) start with zero or more quotes or instances of phrp, which is the textual part of a phrase followed by pPunc, that is, by phrasal, but not sentence, punctuation. Commas are the most common phrasal punctuation mark, and in Real Life they do not always mark phrase boundaries, but they do in this poem, and as another compromise we interpret every instance of comma (also semicolon and colon) as a phrase transition.

Sentences end with either the textual part of a phrase followed by sentence punctuation (sPunc) or a quote. This reflects the fact that phrases are separated from other phrases by phrasal punctuation, but the last phrase in a sentence contains no phrasal punctuation, and it is the sentence-final punctuation that implies its end. A sentence that ends in a quote does not end in sPunc; this is the case with outermost sentence in this poem, which contains a single phrase (phrp) followed by a single quote.

^[29] Because IXML cannot easily emit angle brackets that are not part of well-formed XML markup, our IXML to serialize TexMECS emits curly braces instead of angle brackets, and we then replace them with td:

coffeepot -g:open-road-ixml-all-transitions-ixml-syntax-to-texmecs.ixml -i:open-road.txt | sed '/^</d' | tr '{}' '<>'

^[30] In the Hamlet experiment one parse identified speech structure and the other parse identified metrical structure; in both of the poems, one parse identifies poetic structure and the other parse identifies linguistic structure.

^[31] Attribute grammars were introduced by Donald Knuth in Kunth 1968. See also his Kunth 1990 for the origin story.

^[32] As with other new Invisible-XML operators, we are not committed to using <- to implement this functionality, but what we like about it as a candidate is that 1) it is not permitted in IXML, and therefore does not conflict with spellings that already have a meaning in IXML, and 2) it suggests take the new name of the node on the left of the <- operator, during serialization, from whatever matched the nonterminal to the right of that operator. Using something like -> to spell the new operator would conflict with IXML because a hyphen is an XML name character, so item->key in IXML already means the name of this element is item- and the alias used in serialization is the string key. Hyphen is not a name start character, so it creates no conflict with what is possible in IXML.

^[33] This aspect of the behavior is similar to the way IXML allows any part of a RHS (not just the first item) to be serialized as an attribute on an element named by the LHS. The child nodes are equally accessible because the renaming happens during serialization, after the entire parse result has been created.

^[34] Meanwhile, XML attributes are unordered, which means, among other things, that two documents that differ only in the order in which attributes appear in a serialization are deep-equal.

^[35] See also the discussion of order and XML in La Fontaine 2021.

^[36] IXML could avoid listing all permutations in full by grouping those that share a prefix and factoring out the prefix. For example, there are six possible permutations of A, B, and C (ABC, ACB, BAC, BCA, CAB, and CBA), and the following IXML grammar will match a sequence of newline-separated instances of any of those six patterns, but not any other combination of the constituent letters (e.g., those with repetition, omission, fewer or more than three items):

doc: (permutation)++eol.
permutation: 
  A, (B, C; C, B);
  B, (A, C; C, A);
  C, (A, B; B, A).
A: "A".
B: "B".
C: "C".
eol: #a.

This grammar is equivalent to one that spells out all six combinations directly, but it isn’t much of an improvement in terms of usability. The reasons that we don’t want to spell out all combinations are that it’s tedious, it’s error prone, and it doesn’t scale gracefully, and this alternative has all of those same limitations. We could generate the production in question programmatically, which may be less tedious (or, at least, more interesting) than just typing it out, but code is typically read more often than it’s written, and however this alternative might have been produced, we find it harder to understand than a plain list of the six combinations.

A common workaround for this limitation is the use of a repeatable or-group, although by itself a repeatable or-group would, incorrectly, allow an individual element to repeat. Supplementing a repeatable or-group in a RELAX NG schema with a separate Schematron rule that prohibits the repetition of individual items is a common alternative to interleave for modeling unordered content.

^[37] IXML has a character-level exclusion operator (~), which means that, to avoid matching author, we could match any string that does not have a combination of a as its first character plus u as its second character, etc., but this type of brute-force matching becomes increasingly difficult to write and to understand as the number of characters grows. For example, the following IXML grammar will tag lines of input according to whether they are exactly equal to the string authoror not:

root: (author;other)++eol,eol?. 
author: "author". 
other: ~["a"],[L]* ; 
  ("a",(~["u"; #a],[L]*)?) ; 
  ("au",(~["t"; #a],[L]*)?) ; 
  ("aut",(~["h"; #a],[L]*)?) ; 
  ("auth",(~["o"; #a],[L]*)?) ; 
  ("autho",(~["r"; #a],[L]*)?) ; 
  "author",[L]+. 
-eol: #a.

With input:

a
au
aut
auth
autho
author
authority
editor

the output is:

<root>
   <other>a</other>
   <other>au</other>
   <other>aut</other>
   <other>auth</other>
   <other>autho</other>
   <author>author</author>
   <other>authority</other>
   <other>editor</other>
</root>

^[38] As far as we have been able to determine, the relationship between dominance and containment in LMNL has not been worked out in detail. The only publication we have been able to locate that focuses on the topic in Tennison 2008.

^[39] If we want to say that an ANIMAL contains a CAT we would model this in Invisible-Next just as we would model it in IXML:

doc: ANIMAL.
ANIMAL: CAT.
CAT: "cat".

^[40] As Norm Tovey-Walsh 2025 notes, the terms horizontal ambiguity and vertical ambiguity were introduced by Brabrand et al. 2010.

^[41] We bracket for now the question of what this internal data model might look like. Next-generation languages like TAGML (representing a TAG model; Haentjens Dekker and Birnbaum 2017 and Haentjens Dekker et al. 2018), TexMECS (representing a GODDAG model; Sperberg-McQueen and Huitfeldt 2000), and LMNL (representing a LMNL model; Piez 2026 Symposium) model relationships among markup units in somewhat different ways, even where the underlying conceptual features (such as overlap) are the same or similar. We see no reason that these differences cannot converge on a model that reconciles and accommodates them, but we regard the elaboration of such a model as a separate, although related topic, and one that we intend to address elsewhere.

^[42] Concerning the possibility and utility of applying Invisible-Next processing to documents that already contain markup see Verwer 2026, which applies IXML-like processing to XML input.

^[43] The input to our processing is a stream of characters, and therefore not pre-typed. Human readers can distinguish speaker lines from spoken-text lines typographically because speaker lines in the input are entirely in upper case and spoken-text lines are not. The IXML and Invisible-Next grammars above use the same distinction to process the two types of input lines differently.

^[44] The fact that we have typed and consecutive text nodes illustrates a difference between our modeling and XML, in which text nodes do not have names or attributes (two common ways of representing type information in XML) and cannot occur adjacent to one another (e.g., what might originate in an XSLT stylesheet as adjacent sibling text nodes are automatically interpreted as a single merged text node). Nothing about this visualization is incompatible with markup that supports a more XML-like perspective on text nodes, should we wish to assume one.

^[45] First described in Furnas and Zacks 1994. Directed graphs are often drawn with directional edges, that is, edges with arrowheads that indicate a source-to-target direction. We follow Furnas and Zacks 1994 in omitting the arrowheads, which can be considered superfluous in this case because the directionality is implied: parents are rendered above children vertically.

^[46] Strictly speaking, the face-to-face dual-tree visualization above is also a type of multitree, and it is discussed in the foundational Furnas and Zacks 1994 article.

^[47] Swimlane diagrams are often traced back to the cross-functional process map that originated in Rummler and Brache 2013. Those authors did not use the term swimlane, and the origin of that term is obscure. The term activity partition is used for this sort of activity grouping in UML.

^[48] Whether the term swimlane is appropriate for a traditional visualization, where information flows among lanes, is questionable. The point of lanes in a swimming pool is to keep each swimmer entirely within a single, separate lane.

^[49] The interruptions also raise questions about the suitability of the term swimlane for this visualization. Acutal swimmers get from one end of a lane to another by crossing all of the water in between, without interruption.

^[50] Processing interleave requires keeping track of the state in all interleaved sequences simultaneously. This can be accommodated, at least in principle, in a CFG, but only by enumerating all combinations of those states, a strategy that becomes unmanageable as the number of sequences and number of states in each sequence grows.

^[51] Invisible XML Specification says that a IXML processor should report ambiguity, and that processors may vary in whether input is detected as ambiguous or not. This technically means that IXML parsers are not required to detect or report ambiguity when it is present, but, as far as we know, all current IXML applications detect and report ambiguity.

Ambiguity is a property of grammars, and not of input documents, but parsing with an ambiguous grammar may produce multiple paths through the forest with some input and not with other input. If parsing a specific input document against an ambiguous grammar produces more than one path, Invisible XML Specification strongly suggests (should, but not must) that a conformant processor report the presence of ambiguity.

^[52] IXML does not allow users to create namespaced information. Additionally, the types of XML markup it can create may be limited because of the parser design (for example, IXML can rearrange content only in limited ways). It is nonetheless the case that IXML can transform arbitrary character-stream input into an XML document that can serve as a step toward creating domain-specific markup.

^[53] RELAX NG is able to support an interleave operator because RELAX NG processing uses tree grammars, which are able to track state in a non-linear manner. IXML processing is implemented with context-free string grammars, which do not easily accommodate the multiple contexts required by interleave.

^[54] A tokenizer, or lexer, reads character-stream input and groups the characters to recognize higher-level predefined tokens. Norm Tovey-Walsh 2025 writes that:

The lexer decides what parts of the input are the actual tokens. It might remove insignificant whitespace, strip out comments, normalize strings, etc. The parser then operates on those tokens. […] If Invisible XML was going to work this way, it would have to provide two sets of rules, one for lexing and one for parsing. That would add all kinds of complexity to the language. Instead, Invisible XML has the simplest possible lexer: your input string becomes a sequence of characters. What you’re matching in rules are the individual characters of the input: every single one of them.

A single character might technically be regarded as an input token, but in the absence in IXML of grouping characters into higher-level predefined tokens and the absence of separate lexing and parsing stages in the processing, it might be reasonable to say not just that an IXML processor uses a different type of tokenizer, but that it has no tokenization step that is separable from parsing at all. The technical term for this is scannerless parsing.

^[55] Our application also includes code for emitting XML markup, originally developed to explore the creation of XML with traditional workarounds to represent next-generation features. We have not prioritized the further development of this feature because it requires decision-making that we have not yet explored adequately, such as, for example, how to determine which hierarchies to flatten in the case of overlap.

^[56] For example, the following grammar attempts to create a root node that is an attribute, a structure that is not permitted in XML and that is recognized as dynamic error D05 in Invisible XML Specification:

@doc: A, B.
@A: "a".
B: "b"

The Early grammar compiled for this by Coffeepot is:

1.  $$ ::= doc
2. doc ::= A, B
3.   A ::= 'a'
4.   B ::= 'b'

(note that the grammar does not mention elements or attributes), but when Coffeepot attempts to serialize the result of applying this grammar to matching input, it correctly reports an error: The root node (‘doc’) must not be marked ‘@’.

^[57] The term Russian doll is based on the matryoshka, a set of nested dolls that is famous in Russian folk art. For a description of the types of patterns that can be used (alone or combined) to organize a schema (written for XML Schema but largely applicable also to RELAX NG) see Brüggemann-Klein 2020.

^[58] Challenging is somewhat a matter of personal preference, but the versions without separator operators are objectively more verbose (more symbols) and more complex syntactically (requiring parenthesized subgroups). Compare: A++B vs A, (B, A)* and C**D vs (C, (D, C)*)?.

^[59] Separator notation is not strictly necessary even for IXML, since any expression that uses it can be rewritten without it (and is rewritten during compilation), but its use in the grammars authored by humans can improve the legibility of the productions.

^[60] The LHS labels might be retained when the schema is processed, since they could be useful for error reporting, but the name property of the node in the tree is based only on the value after the element or attribute keyword, and not on the value in the label on a possible LHS.

^[61] The only aspects of an IXML process that must be exposed to the user are the user-supplied grammar, the user-supplied input document, and, in the absence of error, an eventual serialization that conforms to the requirements of the specification. It is, then, possible for an IXML implementation to do whatever it wants behind the scenes, including ignoring the rule name as specified in the original grammar in favor of the name after rewriting, or in favor of some other identifier. Such implementation details are limited only by the imagination of the developer.

The rename feature did not exist in IXML until version 1.1. In Invisible XML Specification (current draft) (under Constructing XML) the roles of the name and the result of renaming (called an alias) are described as follows:

A parse node is one of:

a nonterminal, which has a name, an optional alias, and children,

a terminal, which has a string,

an insertion, which has a string.

Since the parse forest as such need not be exposed to users, nothing prevents an implementation from using the alias in the user-supplied grammar as the name of a node in the parse forest, but the specification understands (or, at least, permits as an understanding) that the name, as a technical feature of the node (supplied alone or before a >) is distinct from the alias (supplied after a >) in the LHS.

^[62] We could refer to this IXML type of matching as complete, exhaustive, or enumerative, all of which suggest that there may be more than one result. We could also describe it as ambiguous, branching, or non-deterministic, since it results in multiple transitions in the parse result.

^[63] For example, an IXML grammar like:

doc: A, B.
A: "x"*.
B: "x"*.

can parse xxx in three ways:

<ixml parses="3" totalParses="3">
<doc xmlns:ixml='http://invisiblexml.org/NS' IXML:state='ambiguous'>
   <A>x</A>
   <B>x</B>
</doc>
<doc xmlns:ixml='http://invisiblexml.org/NS' IXML:state='ambiguous'>
   <A/>
   <B>xx</B>
</doc>
<doc xmlns:ixml='http://invisiblexml.org/NS' IXML:state='ambiguous'>
   <A>xx</A>
   <B/>
</doc>
</IXML>

^[64] Patterns (a type of node in the tree) can include nonterminals (other productions), string literals, character classes, character ranges, or numeric character references (e.g., #a), and epsilon. These all express expectations for the input. When input matches a pattern, the pattern is added to the parse forest and typed as a named pattern (which tells the eventual serializer to ignore it) and the input is added to parse forest as children of the named-pattern node..

^[65] For schema validation we have prioritized next-generation markup, rather than XML, but plan also eventually to include a tokenizer for XML input.

^[66] As we note above, text with explicit markup can serve as input to Invisible-Next processing, but the input is ingested as a character stream, and any recognition of existing markup is part of the user-supplied grammar, rather than a tokenizer supplied by the tool.

^[67] Currently our XML serialization works only in the absence of concurrency. While XML does not support overlapping markup, an XML serializer might be able to create well-formed XML markup according to one of the work-around strategies currently available, such as creating flat output, where the children of the root node are a single sequence of text nodes and empty elements, with attributes on those elements to indicate how they mark the starts and ends of possibly overlapping elements.

^[68] The discontinuity is expressed with markup in TexMECS, although the information that the intervening text is specifically a partition would need to be a feature of an eventual schema because TexMECS does not have separate markup specifically for partitions.

^[69] We avoid the terms EBNF and BNF because different applications may use them to refer to slightly different sets of syntactic rules, about which see Jinks 2004.

^[70] Because IXML makes it possible to suppress all input characters and to emit, through the use of version 1.1 insertions, supplied textual content, an IXML grammar can emit XML that bears no meaningful similarity to the input. The wording in Invisible XML Specification implies that this behavior, although not mentioned explicitly, is not considered typical:

Invisible XML (ixml) is a method for treating non-XML documents as if they were XML, enabling authors to write documents and data in a format they prefer while providing XML for processes that are more effective with XML content.

^[71] Because Invisible XML Specification describes expected behavior in these four cases only partially, it is not surprising that although the three command-line tools that we had available (Coffeepot, Markup Blitz, xmq) conform to what the spec requires, they do not agree fully in what they report. We tested all four conditions, as follows, running the operations at the zsh command line and checking the exit code with echo $?:

Comparison of command-line ixml processing tools

Grammar correct?	Input matches grammar	Output can be serialized	Result category	Coffeepot 3.3.13	Markup Blitz 1.12-SNAPSHOT	xmq 4.1.0-modified
Yes	Yes	Yes	stderr	N/A	N/A	N/A
			stdout	Valid XML	Valid XML	Valid XML
			Exit code	0	0	0
No	N/A	N/A	stderr	Prose, no error code	Prose, error code, stack trace	None
			stdout	None	None	<IXML IXML:state="failed">, no error code, includes "syntax error" and input location
			Exit code	0	1	1
Yes	No	N/A	stderr	None	None	None
			stdout	<fail IXML:state="failed"> with xml description	<IXML IXML:state="failed"> with prose description	<IXML IXML:state="failed"> with xml description
			Exit code	1	0	1
Yes	Yes	No	stderr	Prose, no error code	None	None
			stdout	None	<IXML IXML:status="failed" IXML:error-code="[value]"> with prose in body	Malformed XML, no mention of error
			Exit code	3	0	0

(xmq serialization of malformed XML has been logged as a bug and is scheduled for correction.)

To summarize, all three processors conform to the requirements of the spec in all cases, but in divergent ways:

Correct grammar, input matches: all produce the same result
Incorrect grammar: All report an error, but in divergent ways, on divergent file handles, and with divergent exit codes
Correct grammar but input does not match: All emit XML (with divergent root element names) on stdout with the required @ixml:state attribute value of failed. Divergent exit codes.
Correct grammar, input matches, but output cannot be serialized: Divergent result, file handles, and exit codes.

^[72] Because IXML is defined according to an IXML grammar, case 1 and case 2 both involve failure to parse an instance against a grammar. The two situations are nonetheless meaningfully different for an end-user because if phase 1 does not succeed, the user must adjust the user’s own grammar, which is effectively the input being parsed against the IXML grammar for IXML, while if phase 2 does not succeed, the user may adjust either the user’s own grammar or the input document.

^[73] An ambiguous IXML grammar may produce both parses that can be serialized as well-formed XML and those that cannot, and an IXML processor may choose any parse to (attempt to) serialize for output. This means that if an IXML tool chooses a parse arbitrarily (as it is permitted to do), whether the operation reports success or failure may depend on which parse is selected for serialization. This indeterminacy could be mitigated by requiring the tool to choose only among successful parses, if any are available, for serialization; such behavior is not required by the IXML specification, but it is not prohibited because it is consistent with the requirement that the process emit one or more parses. The exact language of Invisible XML Specification about this issue is that:

Processors must accept and parse any conforming grammar, and produce at least one parse of supplied input that matches the grammar starting at the root symbol. If more than one parse results, one is chosen; it is not defined how this choice is made, but the resulting serialization should include the attribute ixml:state on the document element with a value that includes the word ambiguous.

An exact value of the @ixml:state attribute is not provided; the only expectation is that the value should include the word ambiguous.

^[74] In addition to XML markup, IXML may also insert plain text into the output according to insertions present in the grammar.

Author's keywords for this paper:

next-generation markup; overlap; discontinuity; grammar; LMNL; TexMECS; TAGML; IXML; Creole

Ronald Haentjens Dekker

Researcher

DH Lab, Huygens Institute for the History of the Netherlands

`<ronald.dekker@di.huc.knaw.nl>`

Ronald Haentjens Dekker is a researcher and software architect at DHLab and the Huygens Institute for the History of the Netherlands, part of the Royal Netherlands Academy of Arts and Sciences. As a software architect, he is responsible for translating research questions into technology or algorithms and explaining to researchers and management how specific technologies will influence their research. He has worked on transcription and annotation software, collation software, and repository software, and he is the lead developer of the CollateX collation tool. He also conducts workshops to teach researchers how to use scripting languages in combination with digital editions to enhance their research.

David J. Birnbaum

Professor Emeritus

Department of Slavic Languages and Literatures, University of Pittsburgh

`<djbpitt@gmail.com>`

David J. Birnbaum is Professor Emeritus of Slavic Languages and Literatures at the University of Pittsburgh. He has been involved in the study of electronic text technology since the mid-1980s, has delivered presentations at a variety of electronic text technology conferences, and has served on the board of the Association for Computers and the Humanities, the editorial board of Markup languages: theory and practice , and the Text Encoding Initiative Technical Council. Much of his electronic text work intersects with his research in medieval Slavic manuscript studies, but he also often writes about issues in the philosophy of markup.

Bram Buitendijk

Software Engineer

Digital Infrastructure, KNAW Humanities Cluster

`<bram.buitendijk@di.huc.knaw.nl>`

Bram Buitendijk is a software engineer in the Research and Development team at the Humanities Cluster, part of the Royal Netherlands Academy of Arts and Sciences. He has worked on transcription and annotation software, collation software, and repository software.

Balisage Paper: On beyond Invisible XML: parsing with implicit (and explicit) next-generation markup[1]

Ronald Haentjens Dekker

<ronald.dekker@di.huc.knaw.nl>

David J. Birnbaum

<djbpitt@gmail.com>

Bram Buitendijk

<bram.buitendijk@di.huc.knaw.nl>

Table of Contents

Research goals and questions

Introduction

Overview

Three goals of invisible markup

Experiments: Invisible markup, ambiguity, and concurrency

Added operators in Invisible-Next: interleave, concur, partition, unordered

Experiment 1: Hamlet

Hamlet in separate IXML grammars

Hamlet in one IXML grammar using alternative operator

Hamlet in IXML with complete LMNL and TexMECS output

LMNL output

TexMECS output

Hamlet in Invisible-Next with CONCUR and TexMECS output

Experiment 2: Ozymandias

Ozymandias in separate IXML grammars

Ozymandias in one IXML grammar using alternative operator

Ozymandias (or surrogate) in IXML with complete LMNL and TexMECS output

Ozymandias in Invisible-Next with CONCUR and TexMECS output

Experiment 3: Creating dynamic node names in Invisible-Next

Experiment 4: Distinguishing ordered and unordered content

Results

What we learned about the CONCUR operator

What we learned about ambiguity

What we learned about output

Challenges with input files that contain pre-existing markup

Visualizing multiple hierarchies

Face-to-face dual-tree visualization

Single-root multitree

Hypergraph

Modified swimlane

Selecting a visualization

Prototype

Mildly context-sensitive grammars (MCSG)

Comparison between schema validator and markup generator

How validation and generation are similar and different

RELAX NG schema validation and IXML processing

Purpose

Document domain

Input and output

Grammar and syntax

What those similarities and differences mean for tool development

Parsing user-supplied schema or grammar (shared)

High-level schema or grammar model (shared)

Schema or grammar compiler (shared)

Tokenizer for tagged input (Schema-Next validation only)

Parser to parse input file against compiled schema or grammar (shared)

Serializer (Invisible-Next processing only)

Discussion

Conclusions

Appendix A. Grammar terminology

Appendix B. Failure vs error in IXML processing

References

Author's keywords for this paper:

<ronald.dekker@di.huc.knaw.nl>

<djbpitt@gmail.com>

<bram.buitendijk@di.huc.knaw.nl>

Balisage Series on Markup Technologies

Balisage Paper: On beyond Invisible XML: parsing with implicit (and explicit) next-generation markup^[1]

`<ronald.dekker@di.huc.knaw.nl>`

`<djbpitt@gmail.com>`

`<bram.buitendijk@di.huc.knaw.nl>`

Hamlet in Invisible-Next with `CONCUR` and TexMECS output

Ozymandias in Invisible-Next with `CONCUR` and TexMECS output

What we learned about the `CONCUR` operator

`<ronald.dekker@di.huc.knaw.nl>`

`<djbpitt@gmail.com>`

`<bram.buitendijk@di.huc.knaw.nl>`