Balisage Paper: Serving IIIF and DTS APIs specifications from TEI data via XQuery with support from a SPARQL Endpoint

Architecture: not all is in one TEI file

What 'texts' to serve? The 'diplomatic' transcription of the manuscript? The edition? Of which version? Some relations among textual units in a corpus of ancient literature cut across several languages, periods and interests making the organization efforts of a 'library' clash with reality. Ancient translations, versions of the same 'work' which had their own independent history, collections of smaller units varying in size and scope, summaries, etc. The ancient idea of text was just very different from our own. It is not possible to order ancient texts by title or by author. It is not possible to organise them in a fixed hierarchies without forcing them. What is available instead is a network of relations. Establishing explicit and individual relationships between variously identified units allows to organise ancient literature the way we know it without adding on it the hierarchies which are instead more typical of printed literature. And if we do not know and cannot say anything, we can just do so and let an unidentified text be such without needing to enter a group of other 'similar' items.

For example, a textual unit, that is to say a work, in a more generic sense,^[14] may form part of several other textual units, like in the following example, retrieved searching for records with two <relation> elements with a @name='saws:formsPartOf'

                        
<listRelation>
<relation name="saws:formsPartOf" active="LIT4615Salam" passive="LIT1968Mashaf"/>
<relation name="saws:formsPartOf" active="LIT4615Salam" passive="LIT3575Mashaf"/>
</listRelation>

In an HTML view as well as in another type of output, we can play with this and present the information, relying on the accessibility for our scripts of the information linked. The script will know what to do with 'LIT1968Mashaf'. Also the TEI consumer will be able to resolve with @xml:base to https://betamasaheft.eu/LIT1968Mashaf and will be redirected to the landing page for that item. But the script knows more and given knowledge of what to expect in that context, namely the identifier of a textual unit may be able to fetch its canonical title for example.

Figure 2: Forms part of...

The XQuery doing this knows a great deal about the database structure, its indexes, the desired output. What about the poor client who does not have all these modules and does not want to scrap the HTML to rebuild this information? We could enrich our RDFa support, that is one thing. However, passing on this type of information in the response to a standard request to an API, allows the machine as well as the human to follow those links and allow them to structure the navigation of the collection. This is what the DTS Collection Specification also supports. We have rather some code to expose that in the same way in which the above HTML view is produced, to serve those clients.

Because these XML tags are passed on to the RDF as described above, the graph of relations can be queried from the XQuery building the response to the API request specified by the DTS API, and they can be included in the output json-LD. A function (here) will loop through a list of relations and query for them the triplestore.

declare function dts:sparqls($id, $property){
let $querytext := $config:sparqlPrefixes ||  "SELECT ?x
WHERE {bm:" || $id || ' '|| $property||" ?x }"
let $query := dts:callfuseki($querytext)
return
$query//sr:binding[@*:name='x']/sr:*/text()
};

The dts:callfuseki() function here will simply send an HTTP request to the port where Apache Jena Fuseki is running with the SPARQL query constructed, and iteratively populate a map, for example:

                        
"dts:extensions" : {
      "saws:formsPartOf" : "https://betamasaheft.eu/LIT2384Taamme"
    }

If this extension is exposed by the client of the API, The browsing of the relations between textual units, which are listed by the collection API as a flat list, becomes an actual way to pass from one to the other textual unit without any additional organizational filters. We stick to a plain list and this relations to give it depth and structure, so that the machine client may have at least the same ways to distinguish a link from another link as the human has.

Multiple hierarchies of citation

As well as multiple relations and hierarchies of texts, there are also multiple 'canonical' citations, multiple editions, and multiple citations structure which, depending on the type of context will be resolved by a human reader to a portion of a given text. The example above, of a transcription of a manuscript involves only one such structure, articulated with <pb>, <cb>, and <lb>, but consider the following example.

                        
<div type="edition" xml:lang="gr" resp="#frisk">
<head>
<surplus>ΑΡΡΙΑΝΟΥ</surplus>
ΠΕΡΙΠΛΟΥΣ ΤΗΣ ΕΡΥΘΡΑΣ ΘΑΛΑΣΣΗΣ
</head>
<div type="textpart" subtype="chapter" n="1" xml:id="chapter1">
<ab>
<pb n="1" corresp="#frisk"/>
<pb n="51" corresp="#casson"/>
<pb n="257" corresp="#mueller"/>
<pb n="9r" corresp="#L"/>
<pb n="40v" corresp="#P" facs="https://digi.ub.uni-heidelberg.de/diglit/iiif/cpgraec398/canvas/0084.json"/>
Τῶν ἀποδεδειγμένων ὅρμων τῆς
<placeName ref="pleiades:39290">Ἐρυθρᾶς θαλάσσης</placeName>
καὶ τῶν περὶ αὐτὴν ἐμπορίων πρῶτός ἐστιν λιμὴν
...
</ab>
</div>

Here I have encoded several page breaks of editions (the @corresp points at the @xml:id of a <bibl>). I may certainly refer to this as 'chapter 1' or '1', but I may also refer to page 51 of Casson's edition of page 257 of Muller's to refer to the same passage. Another similar issue is that of the way in which a given part of a work is referred to. The first paragraphs of a liturgical text may contain a commemoration for a certain Saint to be done on a given day of a month. I may refer to this as the commemoration for that Saint, that for the Day or as the first paragraph, depending on the audience or simply on convenience.

                        
<div type="textpart" subtype="chapter" n="1" xml:id="Maskaram">
  ...
  <div type="textpart" subtype="chapter" xml:id="MaskIntroduction">
    ...
  </div>
  <div type="textpart" subtype="chapter" xml:id="Mask1">
    <div type="textpart" xml:id="Mask1Beginning">
      ...
    </div>
    <div type="textpart" subtype="commemoration" xml:id="Mask1Job">
      ...
    </div>
    <div type="textpart" subtype="commemoration" xml:id="Mask1Bartolomewos" corresp="NAR0017SBartalomewos">
      ...
    </div>
    ...
  </div>
  ...
</div>

a reference to 'chapter 1', should resolve to an XPath like div[@n='1'] a reference to 'the second commermoration of the first day of Maskaram' should also resolve to div[@type='commemoration'][2]/parent::div[@xml:id="Mask1"]/ancestor::div[xml:id="Maskaram"] in the same way as an hypothetical reference. The resolver of this reference needs to be able to map it to the correct piece with as few assumptions as possible about what the editor will have decided to be the relevant structures and names for its text and what the client will have picked among those to provide a pointer. Here we have again a place where the code needs to be savvier then we would like it to be if we assume code needs to be as generic as possible. A regex is stored (could be actually kept within the TEI with the existing appropriate elements)^[15] and a function does its best to match, given a priority order what it could refer to piece by piece, building XPaths which are able to match any potential structures. The following is an example from the module of such constructor for a part of an XPath

"/(t:div|t:cb|t:pb|t:lb|t:l)[(@xml:id|@n)='"||
  $r/text()||"' or contains(@corresp,'"||$r/text()||"')]"

The reference used could be the @corresp to the Narrative Unit identifying the commemoration of Bartolomewos without identifying a specific text, as this may occur within differnt types of texts, in different forms, sometimes as edited text, sometimes only as transcription. This is indeed a legitimate, if not the most common reference used, as will also be shown below in the Collatex example Figure 8.^[16]

The XQuery is able, within the exist-db setup for the redirection and parsing of the requested URI, to get a variety of possibly encoded structures and return the correct text. If we had a <citeStructure> in our TEI data, we could achieve the same from an explicit declaration (Cayless et al. 2021), but it is still often the case that we only have parts of text and that the encoding is in slow but constant progress so that such a declaration is not yet possible. The DTS Navigation API, will however only at the moment present one of them. The Collection API request for a textual unit like the one identified by LIT4032SenkessarS,^[17] can be retrieved starting from the HTML view at https://betamasaheft.eu/works/LIT4032SenkessarS/main clicking on the VoID (Vocabulary of Interconnected Datasets) and then selecting the value of void:uriLookupEndpoint, that is https://betamasaheft.eu/api/dts/collections?id=https://betamasaheft.eu/LIT4032SenkessarS. Here the DTS Collection API tells us of the structure of the text, namely that there are textparts, which contain chapters which can contain subscriptio, commemoration and supplication. also this names are given by the encoder and selected by the script according to a series of alternative priorities for each element used in the encoding.

                        
  "dts:citeStructure" : [ {
    "dts:citeStructure" : [ {
      "dts:citeStructure" : null,
      "dts:citeType" : "chapter"
    }, {
      "dts:citeType" : "subscriptio"
    }, {
      "dts:citeType" : "commemoration"
    }, {
      "dts:citeType" : "supplication"
    } ],
    "dts:citeType" : "textpart"
  } ] 

Here we learn of what we may expect, not yet of the actual stuff available. Moving on to the DTS Navigation API with https://betamasaheft.eu/api/dts/navigation?id=https://betamasaheft.eu/LIT4032SenkessarS I am told what is available to begin with, as top structure

                        
  {
    "dts:citeType" : "unit",
    "dts:dublincore" : {
      "dc:source" : [ {
        "@type" : "sc:Range",
        "@id" : "https://betamasaheft.eu/api/iiif/BNFabb66A/manifest"
      }, {
        "@type" : "sc:Range",
        "@id" : "https://betamasaheft.eu/api/iiif/BNFabb66B/manifest"
      }, {
        "@type" : "sc:Range",
        "@id" : "https://betamasaheft.eu/api/iiif/BNFet677/manifest"
      } ],
      "dc:title" : "First half of the year"
    },
    "dts:ref" : "FirstHalf"
  }  

Here, where you can see how the IIIF manifests are also related to the reference (also in the next example, with reference to the correct Ranges), I can then use levels, and I know from dts:citeDepth how many are available, to navigate different levels of structure or I can point to one of the available dts:ref to see what it contains. Since I know already that the commemorations are the fourth level, I can request https://betamasaheft.eu/api/dts/navigation?id=https://betamasaheft.eu/LIT4032SenkessarS&level=4 and get the following among the members of this level of citation

                        
  {
    "dts:citeType" : "commemoration",
    "dts:dublincore" : {
      "dc:source" : [ {
        "@type" : "sc:Range",
        "@id" : "https://betamasaheft.eu/api/iiif/BNFabb66A/range/ms_i1.1.1.3"
      }, {
        "@type" : "sc:Range",
        "@id" : "https://betamasaheft.eu/api/iiif/BNFabb66A/range/ms_i1.1.1.4"
      }, {
        "@type" : "sc:Range",
        "@id" : "https://betamasaheft.eu/api/iiif/BNFabb66B/manifest"
      }, {
        "@type" : "sc:Range",
        "@id" : "https://betamasaheft.eu/api/iiif/BNFet677/manifest"
      } ],
      "dc:title" : "Sǝnkǝssār commemoration of Bartalomewos"
    },
    "dts:ref" : "FirstHalf.1.Mask1.Mask1Bartolomewos"
  }   

Using the dts:ref I can then request the passage with https://betamasaheft.eu/api/dts/navigation?id=https://betamasaheft.eu/LIT4032SenkessarS&ref=FirstHalf.1.Mask1.Mask1Bartolomewos and its text from the DTS Document API, which, if nothing different is stated and supported, should default to return the above XML in a <dts:fragment>. Eventually, other formats of reference known and inferable by the encoder, or user, e.g. https://betamasaheft.eu/api/dts/navigation?id=https://betamasaheft.eu/LIT4032SenkessarS&ref=NAR0017SBartalomewos using the @corresp instead of the @xml:id, would have worked just the same, but the API does not need to declare all the available, it tells its favourite as one option among many valid ones, while the code need to be gentle in accepting any format of request.

This is all very experimental and new, it works only to an extent in this implementation and makes far too many assumptions in the code, some of which may just be plain wrong, but I hope that the features and usefulness of this to match the actual encoding of texts in a varied and complex collection, still allowing yet a higher level of interoperability, passing on to the consumer of the API as much complexity as possible. Paradoxically, the presence of multiple texts and translation of a single identified unit is in this context the simplest bit. The result is needless to say, shaky. Too many uncertainties, too many assumptions in the code. As we prefer mindless scribes because less prone to make changes, also mindless code should be a nicer acquaintance for the codicologist and philologist.

Beta maṣāḥǝft text navigation

Needless to say, we use the DTS API to navigate the transcriptions of manuscripts and the editions of texts which we have. Even if they are partial, as in most cases. Let me say, especially if they are partial or incomplete, we know is a bit of generously encoded structure of the text, e.g. an incipit. This is the norm and not the exception for us, although the collaborative set up as allowed many contributors to share what they have with the rest of the interested parties. Let me first discuss the navigation of a manuscript, which is more standard, as it is defined by the object itself and not by the structure of the intellectual content. We extract transcriptions using Transkribus, as noted above. The following example thus does not contain a guaranteed correct transcription in Ethiopic. The structure of the text, between folia, columns and lines should instead be correct and has been double checked before exporting the results of the Hand-written Text Recognition step. In the sequence of images we see the navigation from folio, to side, to column, to line in the column for the first line of the second folio, recto of Bǝḥerāwi Kǝllǝlāwi Mangǝśti Tǝgrāy, ʿUrā Qirqos, UM-035.^[19]

Figure 3: All text of the transcription of the manuscript

Figure 4: Navigating to Folio 2, https://betamasaheft.eu/ESum035.2

Figure 5: Folio 2, recto, https://betamasaheft.eu/ESum035.2r

Figure 6: Folio 2, recto, column a, https://betamasaheft.eu/ESum035.2ra

Figure 7: Folio 2, recto, column a, line 1, https://betamasaheft.eu/ESum035.2ra1

The navigation is relative to the first available upper level and should allow to navigate within the text. This is rather expensive and time consuming but as a prototype allows us for example to point to exact pieces of text with a simply URL structure and return the correct text. This is also helpful to discover relevant passages. This is useful for example when comparing different manuscript witnesses of a same text. As seen above we can reach the correct places in the description of the content, these will contain the placement of the information and that we can use to resolve to the exact piece of text requested. If a given passage where the text in question occurs is known, then it can be pointed to using this same syntax, even if the description of the manuscript is not complete yet with that information. Alternatively a user could propose a change for the entry, clicking the EDIT button which redirects to GitHub, and add that bit of XML.

Figure 8: Collatex used to collate passages of text retrieved from DTS script, jumping the API

In this case the application does not consume the DTS API, it jumps directly to the XQuery functions fetching the correct portions of text from the parsing of the reference, transforms them into comparable strings, strips of some punctuation and then presents the results provided by passing those normalised strings to Collatex.^[20]

How nice, if that could be done starting from a DTS endpoint so that one could collate passages from data stored in different source databases, eventually in different ways and formats. A functionality similar to that implemented for Alpheios, described below, would allow to collate transcriptions of manuscripts hosted by diverse institutions in the same way in which the Mirador viewer allows its users to see side by side sets of images coming from different providers. And if that collation and the linked IIIF parts could go hand in hand, we could have a generic editor for images and texts, based on solid jsonLD-based standards. But both in this case and the previous actually our application has no reasons to consume its own API, it uses instead the same XQuery functions which are used by the API directly.

Web Annotation indexes

What is an entry into an index of places or persons if not a list of occurrences formatted as a, possibly resolvable, reference to the main structure of the text? Annotations exist in the text for personal names and toponyms marked up in TEI, and querying them together with their context is a fairly easy job for an XQuery script. So, if we can fetch from DTS our table of contents, the contents in the correct order with correct references flexibly adjusted according to the text itself and its author and encoder, we may in the same way fetch a list of marked terms of a given kind, e.g. toponym and produce our indexes by providing an ordered list of these and the corresponding pointer to their context in terms of navigation. Eventually, a script-based index allows us to alter the context and produce indexes from the same annotations for outputs or views which are different. The Distributed Text Services does not have a specification for this,^[21] but I suggested that this would complete, with IIIF, the resources needed to be able to produce API based book products for example, dynamically generated from the collaboratively edited data. Similarly a citation of a passage in another text, encoded like in the following example,

                        
<ref cRef="betmas:LIT2000Mazmur.101.7">
ተመሰልኩ፡
<app>
<lem wit="#D #E #F">ጰልቃን፡</lem>
<rdg wit="#M #V">ጳልቃን፡</rdg>
</app>
ዘገዳም።
</ref>

could be unpacked into a series of meaningful informations for a client which is navigating the index, using the Web Annotation standards, like in the example below, which is extracted from the GitHub issue linked above.

  {
    "body": [
      {
        "text": "ተመሰልኩ፡ ጰልቃን፡ ጳልቃን፡ ዘገዳም።",
        "@lang": "gez",
        "role": "ref",
        "id": "https://betamasaheft.eu/LIT2000Mazmur.101.7"
      }
    ],
    "@context": "http://www.w3.org/ns/anno.jsonld",
    "target": {
      "source": {
        "dts:citeType": "sentence",
        "dts:level": "1",
        "dts:citeDepth": 2,
        "dts:passage": "/api/dts/document?id=https://betamasaheft.eu/LIT4916PhysB&ref=4.2",
        "type": "Resource",
        "link": "https://betamasaheft.eu/LIT4916PhysB.4.2",
        "id": "https://betamasaheft.eu/LIT4916PhysB",
        "dts:ref": "4.2",
        "dts:references": "/api/dts/navigation?id=https://betamasaheft.eu/LIT4916PhysB&ref=4.2"
      },
      "selector": {
        "@value": "TEI/text/body/div[2]/div/div[2]/ab/ref",
        "type": "XPath"
      }
    },
    "type": "Annotation"
  }

In the body of the annotation, I can count on the ability of the server to unpack the @id, from https://betamasaheft.eu/LIT2000Mazmur.101.7 to https://betamasaheft.eu/works/LIT2000Mazmur/text?ref=101.7 which uses the same parameter of the DTS specification to pass on to the client requests related to the passage and the collection. Also the target of the annotation is enriched with a series of information which provide navigation assistance, including the XPath selector, with one among the many possible Xpath to the node which constitutes the annotation (<ref> in this case). Both if I am looking at a wider collection of texts or I am focusing on a portion of an identified text, I can produce on the fly indexes of declared features. For example in the image below, the index of places is loaded besides the navigation bar for the Chronicle of Bakāffā^[22]

Figure 9: The index of places beside the navigation of the text structure in the Chronicle of Bakāffā.

Alpheios Alignment

I would like to conclude with an example of an external client for the DTS services, the Alpheios Alignment tool (Version 2), which has just been developed by The Alpheios Project.^[23] The first version of the Alpheios tool for the alignment of texts in different languages was improved so that texts may be uploaded from source, with copy paste or from DTS API. Available implementations include Alpheios' own collection of Greek and Latin texts and the Beta maṣāḥǝft implementation. In the following images you can see how the interface leads through different requests to the API to select the text which is then loaded and prepared for alignment.

Figure 10: Loading a text in Alpheios from a DTS API.

Figure 11: Navigating the Collection API

Figure 12: Available passages (folia) from the Navigation API

Eventually a DTS 'Search' functionality could be added to load a collection which is a subset of entries coming from a query, so not all resources with some text, but all resources with some text and a match for a given string query. This would need a parameter for this query string to be added to the specification, which has not yet been proposed.

The json export of alignment will be then transformed to TEI for inclusion in the BM data repository and eventually the annotation thus created fed into a set of existing morphological annotations and texts.