Balisage Paper: Hyper, Multi, or Single? Thinking about Text in Graphs and Trees

Data models

Text features

In earlier publications, we distinguished at least three complex features with which digital scholarly editors have to deal on a regular basis: overlapping or concurrent hiearchies, discontinuous text, and non-linear structures (Haentjens Dekker and Birnbaum 2017, Haentjens Dekker et al. 2018). Expressing concurrent or overlapping structures is, if not the biggest, surely the most famous and most debated obstacle for text modeling in XML and therefore needs little explaining. Discontinuous text is usually illustrated with a quotation that is interrupted by an aside or by the narrator’s voice (cf. Sperberg-McQueen and Huitfeldt 2008). Non-linear structures, finally, can be found on the level of the source text as well as the markup, e.g., in-text revisions on a manuscript or an abbrevation and its expansion supplied by the editor in markup. The common denominator is a temporary break in the linearity of the text that can be conceptualized as a split of the text stream in two or more substreams or branches.

In this section we show how (or to what extent) these features are modeled in TAG. We take our examples from three pages of Love and Freindship, a short epistolary novel by Jane Austen, written between 1790-1793. The novel is written in a notebook entitled “Volume the Second” and is part of her Juvenalia manuscripts.^[5]

For each text feature, we first describe our philological interpretation of the feature in question. We then show the syntactical serialization of the feature in TAGML and in XML, combined with visualizations of the underlying data models. This will facilitate the comparison between TAGML and XML and, we hope, increase the readers’ awareness of the relationship between the conceptual model and its logical implementation(s).

Figure 4: Jane Austen, Love and Freindship, p.6

Page 6 of the manuscript of Love and Freindship, written by Jane Austen between 1790-1793. Source: Austen 2010, https://janeausten.ac.uk/manuscripts/blvolsecond/6.html.

Figure 5: Jane Austen, Love and Freindship, p.7

Page 7 of the manuscript of Love and Freindship, written by Jane Austen between 1790-1793. Source: Austen 2010, https://janeausten.ac.uk/manuscripts/blvolsecond/7.html.

Figure 6: Jane Austen, Love and Freindship, p.8

Page 8 of the manuscript of Love and Freindship, written by Jane Austen between 1790-1793. Source: Austen 2010, https://janeausten.ac.uk/manuscripts/blvolsecond/8.html.

Workflow

After having encoded the text in TAGML, the document can be uploaded to the Alexandria repository. Alexandria is operated on the command line and is set up to work similar to git, the version control software often used by programmers and digital humanists to work collaboratively on code. Via command line commands, users can “check in” and “check out” TAGML documents to the Alexandria repository. In addition to a TAGML document, users can also upload one or more views in which they can define which (groups of) markup nodes can be filtered out. We created the view functionality because the TAG document in Alexandria can potentially contain a large amount of information, and we assume that not all users will always be interested in all information. A view is expressed in JSON, for example: {"includeGroups":["D"]} {"includeMarkup:["del"]"}. In this example, the view says to include the markup nodes that are grouped with the identifier “D” and all markup nodes labelled “del” when the TAGML document is checked out of the Alexandria repository.

An Alexandria check-in means that the TAGML document is parsed, verified for well-formedness, and stored as a MCT in the repository. A check-out in combination with a view generates a new TAGML document that contains the selected markup nodes, while the master TAGML document remains intact in the repository. The checked-out document can be edited and checked-in again. Alternatively, users can indicate upon checkout whether they want the master document to be exported to another format: currently, we support SVG, XML, DOT, or PNG. Figure 14 depicts this workflow:

Figure 14: Editorial workflow in Alexandria

Visualization of the editorial workflow in Alexandria. The upper row represents the user actions, the second row the commands given to Alexandria. The third row represents what happens in the user’s workspace and the final row shows what takes place in the localized repository of Alexandria. Note that this workflow is depicted as linear; in reality it will rather be iterative, with steps being repeated as often as needed.

Let’s zoom in on the last step in this workflow diagram: the export to XML. The Alexandria command export-xml seems easy enough and to some extent it is: technically, converting a MCT to a single tree is fairly simple. Like XML, TAGML has markup tags with names, attributes, and values. And although TAGML supports different data types for attribute values (in addition to a string, TAGML attribute values can be integers, floats, strings, lists, or Booleans), the TAGML attribute values can be converted to string type attribute values in XML. Still, a graph with multiple concurrent hierarchies contains more information than a mono-hierarchical tree, so a graph-to-tree conversion implies that we have to decide how to express that information. This depends in part on how the user plans to use the exported document. Representing overlapping hierarchies in a single tree, for example, will require additional tagging. Users may therefore want to scale down the amount of information in the XML document and select only the information that is relevant for their purpose. This means deciding whether the exported XML document should have a leading hierarchy and if so, which markup nodes should be part of it. After addressing the algorithmic side of the TAGML-to-XML conversion in section section “The code”, we will discuss the editorial side.

The code

The steps taken during the TAGML to XML conversion are as follows:

Figure 15 (Figure 15) shows a flowchart of this process. The flowchart starts after the user has created a TAGML document and a view and uploaded them in the Alexandria repository.

Figure 15: Flowchart of the steps taken during the TAGML-to-XML conversion

Visualization of the TAGML-to-XML conversion process.

The code base of Alexandria is written in Kotlin. The code fragment below shows the class definitions of Node, MCT, and Event. Edges are stored in two directions: incoming and outgoing. Target nodes are stored in LinkedHashMaps to preserve the order of the nodes.

                        
sealed class Node
data class Markup(val label: String, val colors: List<String>, val id: Long = System.currentTimeMillis()) : Node()
data class Text(val content: String, val id: Long = System.currentTimeMillis()) : Node()

class MCT(val rootNode: Markup) {
	val outgoingEdges: MutableMap<Markup, LinkedHashSet<Node>> = HashMap()
	val incomingEdges: MutableMap<Node, LinkedHashSet<Markup>> = HashMap()
}

sealed class Event
data class MarkupOpen(val node: Markup) : Event()
data class MarkupClose(val node: Markup) : Event()
data class TextEvent(val node: Text) : Event()

As described above (section section “Workflow”), a TAGML document that is checked into the Alexandria repository is parsed and stored as a MCT. When the user gives the export-xml command to Alexandria, the TAGTraverser iterates over the MCT and generates a stream of Events. The Kotlin code fragment below describes the traversal algorithm. It traverses over the nodes in topological order and creates TextEvents, MarkupOpen, or MarkupClose events. A text node generates a TextEvent; a markup node generates a MarkupOpen event of that node. In order to link the MarkupClose events to the appropriate MarkupOpen events, we keep track of the markup that is currently open by using markup stacks. There is a global stack as well as one stack for each color in the MCT. Each markup node is added to the global stack and to the relevant color stack. Before we can generate a TextEvent or a MarkupOpen event for a node, we need to check the top of the relevant color stack(s) to see if it’s not a parent of the current node. Those markup nodes generate MarkupClose events and can be removed from both the color stacks and the global stack. After all nodes have been processed in this manner, the markup that is left on the global stack generates the remaining MarkupClose events.

       fun traverseMCT(mct: MCT): List<Event> {
	val nodes = topologicalSort(mct)
	val result = arrayListOf<Event>()
	val colorToStackMap = HashMap<String, Stack<Markup>>()
	val globalStack = LinkedHashSet<Markup>()
	for (node in nodes) {
		val parents = mct.incomingEdges.getOrElse(node) { emptySet<Markup>() }
		val stacksToCheck: List<Stack<Markup>> =
				when (node) {
					is Markup -> colorToStackMap.entries.filter { node.colors.contains(it.key) }.map { it.value }
					is Text -> colorToStackMap.values.toList()
				}
		for (stack in stacksToCheck) {
			while (stack.peek() !in parents) {
				val nodeToPop = stack.pop()
				if (globalStack.remove(nodeToPop)) result.add(MarkupClose(nodeToPop))
			}
		}
		when (node) {
			is Markup -> {
				node.colors.map { colorToStackMap.getOrPut(it) { Stack() } }.forEach { it.push(node) }
				globalStack.add(node)
				result.add(MarkupOpen(node))
			}
			is Text -> result.add(TextEvent(node))
		}
	}
	globalStack.reversed().forEach { node -> result.add(MarkupClose(node)) }
	return result
}

Finally, the algorithm creates the XML document from the MCT. The code below describes how the algorithm loops over the Events and creates XML tags. The XML tags are based on the type of event and whether the node associated with the event is the leading hierarchy or not. Nodes in the leading hierarchy are used to create XML content elements, nodes in the other hierarchies are converted to Trojan Horse elements. Trojan Horse elements are a specific type of elements or “segment-boundary delimeters” with a namespace definition th: (see . Two related milestones are linked by means of matching @start and @end attributes, so the regular XML <s>The sun is yellow</s> becomes <s th:s sID="foo"/>The sun is yellow<s th:s eID="foo"/> in Trojan Horse markup (De Rose 2004, Barnard et al. 1995, Sperberg-McQueen 2018). Additionally, the Trojan Horse markup elements are given an attribute that is generated from their TAGML group identifier, e.g., @th:doc="D" for all markup nodes in a group called “D”.

                        
       fun createXML(mct: MCT, leadingHierarchy: String, writer: Writer) {
	val events = traverseMCT(mct)
	val xml = XMLOutputFactory.newFactory().createXMLStreamWriter(writer)
	for (event in events) {
		when (event) {
			is TextEvent -> xml.writeCharacters(event.node.content)
			is MarkupOpen -> if (event.node.colors.contains(leadingHierarchy)) xml.writeStartElement(event.node.label)
					    else xml.apply {writeEmptyElement(event.node.label)
						writeAttribute("sID", event.node.id.toString()) }
			is MarkupClose -> if (event.node.colors.contains(leadingHierarchy)) xml.writeEndElement()
					    else xml.apply { writeEmptyElement(event.node.label)
						writeAttribute("eID", event.node.id.toString()) }
		}
	}
	writer.close()
}

The user

The user is usually not aware of the algorithmic details of the XML export and — although the code is open source and there for anyone to look at — not required to either. Still, the conversion process is not just a matter of clicking a button: it is designed so that the user can influence it. Let’s take a closer look. Figure 16 (Figure 16) shows the (simplified) TAGML transcription of the entire text of the fourth letter from Love and Freindship.

Figure 16: TAGML encoded document

The TAGML-encoded text of the fourth letter in Love and Freindship, indented for readability.

The overlapping hierarchical structures in TAGML cannot be automatically transformed to XML. This is where the user comes in: they can choose the hierarchy formed by the group of markup nodes labelled “D”, the hierarchy formed by the markup nodes labelled “T”, or no leading hierarchy in the XML output. In case of the latter, it suffices to use the command alexandria export-xml Austen_VtS, which says as much as “Hi Alexandria, please export the document called Austen_VtS to XML”. Subsequently, the TAGTraverser described in the previous section will iterate over the MCT, generate a stream of events, and build an XML tree. All TAGML markup nodes will be transformed into Trojan Horse elements. A short fragment of the Trojan Horse XML output is shown below:

                        
<?xml version="1.0" encoding="UTF-8"?>
<xml xmlns:tag="http://tag.di.huc.knaw.nl/ns/tag"
 xmlns:th="http://www.blackmesatech.com/2017/nss/trojan-horse" th:doc="D P T">
 <text title="Love and Freindship" type="novel" author="Jane Austen" date="1790">
<page n="6" th:doc="D" th:sId="page0"/><head th:doc="D T" th:sId="head1"/><l th:doc="D" th:sId="l2"
 />Letter 4th<l th:doc="D" th:eId="l2"/><head th:doc="D T" th:eId="head1"/><l th:doc="D"
 th:sId="l3"/>Laura to Marianne<l th:doc="D" th:eId="l3"/><s th:doc="T" th:sId="s4"/><l
 th:doc="D" th:sId="l5"/>Our neighbourhood was small, for it consisted <l th:doc="D"
 th:eId="l5"/><l th:doc="D" th:sId="l6"/>only of your Mother. <s th:doc="T" th:eId="s4"
 />
<s th:doc="T" th:sId="s7"/>She may probably have already <l th:doc="D" th:eId="l6"/><l
 th:doc="D" th:sId="l8"/>told you that being left by her Parents in indegent <l
 th:doc="D" th:eId="l8"/><l th:doc="D" th:sId="l9"/>Circumstances she had retired into
 Wales on economi-<l th:doc="D" th:eId="l9"/><page th:doc="D" th:eId="page0"/>
<page n="7" th:doc="D" th:sId="page10"/><l th:doc="D" th:sId="l11"/>cal motives. <s th:doc="T"
 th:eId="s7"/><s th:doc="T" th:sId="s12"/>There it was our freindship first <l th:doc="D"
 th:eId="l11"/><l th:doc="D" th:sId="l13"/>commenced – Isabel was then one and twenty – 
<l th:doc="D" th:eId="l13"/><s th:doc="T" th:eId="s12"/><s th:doc="T" th:sId="s14"/>
<l th:doc="D" th:sId="l15"/>Tho' pleasing in both her Person and Manners 
<l th:doc="D th:eId="l15"/><l th:doc="D" th:sId="l16"/>(between ourselves) she never possessed the 
hun-<l th:doc="D" th:eId="l16"/><l th:doc="D" th:sId="l17"/>dreth part of my Beauty or
Accomplishments.<l th:doc="D" th:eId="l17"/><s th:doc="T" th:eId="s14"/>

        <!-- more text and markup --> 
    <page th:doc="D" th:eId="page10"/>
    </text>
</xml>

The user could also decide to make one of the hierarchical structures leading in the XML output. In that case, they can add a parameter to the Alexandria command with which they indicate which grouped markup nodes should form the leading hierarchy, e.g., alexandria export-xml Austen_VtS -l D. In this example commend, the markup nodes in group “D” are transformed into XML content elements; the markup nodes not belonging to the leading hierarchy will be transformed into Trojan Horse elements. A fragment of the XML output with the markup group “D” as leading structure would look as follows:

                        
<?xml version="1.0" encoding="UTF-8"?>
<xml xmlns:tag="http://tag.di.huc.knaw.nl/ns/tag"
 xmlns:th="http://www.blackmesatech.com/2017/nss/trojan-horse" th:doc="P T">
     <text title="Love and Freindship" type="novel" author="Jane Austen" date="1790">
     <page n="6">
     <head><l>Letter 4th</l></head>
         <l>Laura to Marianne</l>
         <s th:doc="T" th:sId="s0"/><l>Our neighbourhood was small, for it consisted </l>
         <l>only of your Mother. <s th:doc="T" th:eId="s0"/>
         <s th:doc="T" th:sId="s1"/>She may probably have already </l>
         <l>told you that being left by her Parents in indegent </l>
         <l>Circumstances she had retired into Wales on economi-</l>
     </page>
     <page n="7">
         <l>cal motives. <s th:doc="T" th:eId="s1"/>
         <s th:doc="T" th:sId="s2"/>There it was our freindship first </l>
         <l>commenced – Isabel was then one and twenty – </l> <s th:doc="T" th:eId="s2"/>
         <s th:doc="T" th:sId="s3"/><l>Tho&apos; pleasing in both her Person and Manners </l>
         <l>(between ourselves) she never possessed the hun-</l>
         <l>dreth part of my Beauty or Accomplishments.</l> <s th:doc="T" th:eId="s3"/>
             <!-- more text and markup --> 
         <s th:doc="T" th:sId="s9"/> <q tag:n="1" th:doc="P" th:sId="q10"/>
         <l>"Beware my Laura<q th:doc="P" th:eId="q10"/> (she would often say) </l>
         <q tag:n="1" th:doc="P" th:sId="q11"/>
         <l>Beware of the insipid Vanities and idle Dissipations </l>
         <l>of the Metropolis of England; Beware of the </l>
         <l>unmeaning Luxuries of Bath &amp; of the Stink-</l>
         <l>ing fish of Southampton."</l>
         <q th:doc="P" th:eId="q11"/> <s th:doc="T" th:eId="s9"/>

    </page>    
    </text>
</xml>

Note how the discontinuous text, marked with the suspend and resume signs in TAGML, is coverted to XML. With the attribute @tag:n="1" the discontinued parts of the quotation are linked together. This approach is also suggested by Wendell Piez in Piez 2008. Finally, the header of both fragments contains information about which group of markup nodes are represented in Trojan Horse (th:doc="T D P" versus th:doc="T P").

In order to illustrate the variety of options created by the XML export function of Alexandria, the final paragraphs of this section explore two hypothetical yet highly likely user scenarios. Let’s say that, after modeling the text in TAGML and uploading the TAGML document in Alexandria, a user wants to (1) publish it online, so they need an HTML version of the encoded text; (2) send it for approval to another scholarly editor who prefers to work in Word (e.g., using LibreOffice of Microsoft Office). We can update the workflow diagram by adding these two steps:

Figure 17: Editorial workflow in Alexandria and beyond

Visualization of the editorial workflow with two additional transformation steps.

As the diagram illustrates, the transformation steps in the editorial workflow do not take place in the context of Alexandria, but in the user’s own workspace. This is a conscious choice: we assume that most users prefer their own, possibly customized, tools, transformation scenario’s, and work environments. Accordingly, we provide TAGML users with the opportunity to create workflows and pipelines in which TAGML is seamlessly integrated with other tools. For instance, the user can use XSLT in order to create an HTML document from the XML document.

For the transformation to Word, we use the open source software OxGarage, a RESTful web service that was created by members of the TEI community and allows users to manage transformations between various document formats.^[9] Even without any additional information, the XML document sampled above (with the page structure as leading hierarchy) is easily transformed into a clearly readable Word document:

Figure 18: Word document created with OxGarage

The Word document created using OxGarage.

Note that the text of the addition, marked with add tags in both the source TAGML and the XML output, is automatically surrounded by diacritical marks in the Word output, and that the text marked as deleted in the TAGML and XML sources is represented as crossed out text between square brackets.