Balisage Paper: An XML engine to model and query multimodal concurrent linguistic annotations

Introduction

In this paper, our intention is to present an XML engine defined to model and query multimodal concurrent annotated data. This work stands in the context of the OTIM (Tools for Multimodal Annotation) project which aims at developing conventions and tools for multimodal annotation of a large conversational French speech corpus (http://aune.lpl.univ-aix.fr/~otim/) [Blache 2010b]. This interdisciplinary project is funded by the French ANR agency; it groups together Social Science and Computer Science researchers. Within OTIM, our objective is to provide linguists with a unique framework to encode and manipulate numerous linguistic domains: morpho-syntax, prosody, phonetics, disfluencies, discourse, gesture and posture [Blache 2010]. For that, it has to be possible to bring together and align all the different pieces of information (called annotations) associated to a corpus.

Since some years, several works have studied concurrent markups/annotations associated to the same data, in particular in the context of XML documents. These documents are usually called multistructured. Indeed, XML documents are mainly hierarchical. The hierarchy, captured in a tree-like structure corresponds to one level of analysis of the data contained in the document. Concurrent markup opens a way for dynamically linking and aggregating documents with different structures associated to the same data. The CONCUR feature of SGML [Goldfarb 1990] first pointed out this need in the nineties in context of document-centric encoding where some applications needed to consider more than one hierarchy over the same text.

The main problem with concurrent structures is that merging every hierarchy in an unique XML document implies overlapping: structures cannot be merged in order to get a well-formed XML document without using a flat representation or hyperlinks that make the structure difficult to query with standards XML languages like XPath (http://www.w3.org/TR/xpath/), XQuery (http://www.w3.org/TR/xquery/) or XSLT (http://www.w3.org/TR/xslt/). See [Witt 2004][DeRose 2004] for a review about the problem of overlapping and about multiple hierarchies. Another approach is to keep one document by hierarchy and to solve the problem of the concurrent querying. We stand our work in this last approach: we want to keep each structure safe to use available XML tools and languages for its manipulation.

Main proposals are usually classified in three categories [Bruno 2006][Portier 2010].

Historical solutions which propose syntactic solutions for the representation of multiple hierarchies in the same text, like the CONCUR feature [Hilbert 2005] of SGML or TEI specifications [TEI P4] (e.g. a flat representation or a main (hierarchical) structure and the use references (ID/IDREF) for the description of the other structures). These solutions make impossible querying by means of standards XML languages.

Proprietary graph based-model (and possible alternative syntax to XML) like LNML [Tennison 2002], TexMecs[http://decentius.aksis.uib.no/mlcd/2003/Papers/texmecs.html], XCONCUR [http://www.xconcur.org/], Annotation graphs [Bird 2001] (coming from the linguistic domain coupled to specific extension of XPath) or MVDM [Djemal 2008]. They allow the concurrent markup relying on a granule of data common to all the structures. Nevertheless, hierarchical structures are easier to exploit compared to the graph structure which remain complex when a large number of structures exist. XML syntax for serialization or import/export XML syntax do not make easy their querying by XML standard languages.

Finally, the third category is XML compatible contributions; they generally propose extensions of XML data model (XDM) (http://www.w3.org/XML/Datamodel.html) to consider a set of XML trees [Le Maitre 2006][Jagadish et al. 2004] or several trees sharing their leaves [Dekhtyar 2005][Bruno 2007][Bruno 2007b] [Chatti 2007]. For querying, these proposals define extensions of XPath or XQuery in order to navigate between different structures either by extending the notion of step in XPath [Jagadish et al. 2004] or by adding new axis [Dekhtyar 2005]. Another solution is to extend the semantics of the XQuery filter [Bruno 2006] to use as much as possible an unchanged XQuery structure. Adding Allen’s relations [Allen 1991] by means of function definitions enable to deal with overlapping [Bruno 2006][Chatti 2007].

Besides, some linguistic projects have a similar objective than OTIM, for instance NITE (http://groups.inf.ed.ac.uk/nxt/), AGTK (http://weblex.ens-lsh.fr/projects/xitools/logiciels/AGTK/agtk.htm), ATLAS [Bird et al. 2000], PAULA (http://www.sfb632.uni-potsdam.de/~d1/paula/doc/), XStandoff [Sperberg-McQueen 2000]. These projects rely on graph-based model. They generally propose toolkits for multi-level annotation by means of libraries of data and annotation management. The multiplication of annotation schemes and coding formats is a severe limitation for interoperability. One solution consists in developing higher level approaches (e.g. GrAF [Ide 2007]) or annotation graphs based formats on top of which conversion routines between tools can be developed (see the Atlas Interchange Format [Sthrenberg 2009]). However, these experiments still remain very programmatic.

Interoperability of linguistic annotated resources requires over all to be independent from the coding format. This means to specify and organize the information to be encoded independently from the constraints or restriction of the format (or the annotation tool), then to encode the information into an standard XML format, readable whatever the edition or annotation system. Moreover we have made the choice not to provide new anootation tools for linguists as they have a long experience and they are efficient with their own tools.

This paper is organised as follows: the context and motivations are presented, then we describe the project objectives and its functional architecture. A visual UML representation of the linguistic knowledge is proposed before the OTIM XML representation and Java framework are explicited. The end of the work shows the methods for querying and managing linguistic annotations within the XML engine and the implementation of the whole corpus construction and management before a conclusion.

Context and motivations

The OTIM project can be summarized in two main steps.

The first step concerns the multimodal annotation of a conversational speech between two persons. It is under the responsibility of linguists; annotation is done according to different levels of linguistic analysis. Each expert has to annotate the same data flow according to its knowledge domain and the nature of the signal on which he annotates (signal transcription or signal). Experts generally use dedicated tools like PRAAT http://www.fon.hum.uva.nl/praat/, ANVIL http://www.anvil-software.de/ or ELAN http://www.lat-mpi.eu/tools/elan/. The qualifier multimodal is due to the nature of the studied corpus which is composed of text, sound, video.

Within the project OTIM, linguists propose an encoding for annotating spoken language data, with the acoustic signal as well as its orthographic transcription. They have chosen to use Typed Feature Structures [Carpenter 1992][Copestake 2003] (TFS) to represent in an unified view the knowledge and the information they need for annotation. TFS representation is usual for linguists: it aims at normalizing, sharing and exchanging annotation schemas between experts. Linguistic annotation tools rely on native and not often open formats which are not directly interoperable. TFS provides an abstract description using a high level formalism independent from coding languages and tools.

The second step concerns the representation and manipulation of multimodal annotation. To analyze and find correlations between annotated linguistic domains, it is necessary to consider them grouped together: it implies the definition of a formal model for describing and manipulating them in a concurrent way. The main difficulty in defining a data model comes from the heterogeneity and the distribution of the resources. Concurrent manipulation consists in querying annotations belonging to two or more modalities or in querying the relationships between modalities. For instance, we need to be able to express queries over gestures and intonation contours (what kind of intonation contour does the speaker use when he looks at the listener?) and to query temporal relationships (in terms of anticipation, synchronization or delay) between both gesture strokes and lexical affiliates. The results of queries could be useful to help in constructing new annotations or to extend existing ones.

The paper focuses on the second step. It describes an XML engine based on an architecture dedicated to the construction and the exploitation of multimodal annotated linguistic corpus. Our theoretical standpoint being to share data and resources, we will use open standards from the XML universe (see http://www.w3.org).

Linguistic knowledge is captured by means of three types of information:

Figure 1: TFS of the prosodic domain

TFS description of the prosodic domain within the OTIM project.

Objectives and functional architecture

TFS is well suited to take into account the heterogeneous characteristics of annotated data. TFS provides an abstract description using a high level formalism independent from coding languages and tools. Nevertheless, due to its theoretical nature, such a representation cannot be used within an applicative framework and has to be implemented into other formalisms.

In this context, our contributions are the following:

In summary, from the organization of annotations in terms of TFS, we automatically generate a XML schema [http://www.w3.org/XML/Schema] for each linguistic domain. All the annotations are then encoded following this schema and data are represented in standard XML document. This representation provides a high level of modularity for applicative requirements and enables to modify only the structure / schema needed and it can be queried by our XQuery extension which supports Allen’s relations. Notice that the TEI consortium has proposed guidelines for implementing feature structures in XML (http://www.tei-c.org/Guidelines/P5/) however, we have decided not to encode TFS this way because type hierarchies and inheritance are not easily and directly represented and no querying support has be defined. This engine, from TFS to XML data, based on formalisms independent from linguistic coding languages and tools is an element of answer to the question of interoperability.

Figure 2: The OTIM Framework

The OTIM project applicative framework.

Our approach guarantees that no information is lost when translating one format into the target formalism. However, the Java / XML framework does not provide linguists with a visual representation equivalent to the graphical representation of TFS. Therefore, we have chosen to make a preliminary work within the representation process that leads to XML / Java. This work is the representations of TFS by means of UML diagrams. This representation has two advantages: first it is standard, secondly there is many friendly and ergonomic tools that enable to build UML diagrams like Omondo (http://www.omondo.com/) or Poseidon (http://www.gentleware.com/).

UML Representation

As we have already said, TEI consortium proposes guidelines for implementing TFS in XML (http://www.tei-c.org/Guidelines/P5/). However, OTIM consortium has decided not to encode TFS this way for several reasons [Blache 2010]. First, there is a need of representation for type hierarchies and inheritance. Secondly, it is not realistic, or even possible, to encode all information by means of a TFS, that rapidly becomes huge and intractable. This last argument is important in the perspective of interoperability: annotation tools mainly focus on properties annotation (the encoding of object characteristics). Only some of them propose solution for constituency representation (e.g. in terms of primary/secondary tracks). None implement typing machinery. We think then preferable a decentralized representation in which objects are represented separately, their organization in hierarchy being encoded independently from their properties. Such an encoding offers the advantage to be close to the traditional way of encoding annotations without losing the richness of TFS representation.

OTIM UML representation relies on two different views. Static one describes the static structure in terms of objects, attributes, operations and relationships. This view includes class diagrams. Dynamic view emphasizes the dynamic behavior of a system. One of its advantages is that the language includes a set of graphical notation techniques to create and share intuitive visual models. We can represent a TFS description by a set of UML class diagrams by means of the following mapping:

Figure 3: The phonetics domain

UML representation of phonetics domain.

XML / Java representation

Querying and managing annotations in XML

The OTIM project aims to provide a fully annotated corpus represented in XML which can be queried by the standard XQuery language. Annotations can be used in various ways: simple viewing, domain crossing or reinsertion within standard tools. Every need of access to annotations relies on a query on the corpus. We first present general queries on the corpus and then the OTIM specific queries related to time and temporal relations between annotations. These last queries need an extention of XQuery with temporal capabilities.

for $phoneme in fn:doc("phonetics.xml")/SyntacticPhrase/words/word/phonemes/phoneme[@label="A"]
return $phoneme

for $phoneme in fn:doc("phonetics.xml")/SyntacticPhrase/words/word/phonemes/phoneme
where ($phoneme/TimeInterval/start/@time > 250) and ($phoneme/TimeInterval/end/@time < 500) 
return $phoneme

for $phoneme in fn:doc("phonetics.xml")/SyntacticPhrase/words/word/phonemes/phoneme
where ($phoneme/TimeInterval/start/@time > 250) and ($phoneme/TimeInterval/end/@time < 500) 
return <A duration='{$phoneme/TimeInterval/end/@time - $phoneme/TimeInterval/start/@time}'/>
   

for $ap in fn:doc("prosody-IP_AP.xml")/TurnConversationalUnit/phrases/ip/constituents/ap
where count($ap/syllables/syllable) > 6
return 
  <phoneme id='{$ap/@id}'>{$ap/label}
  {for $phoneme in fn:doc("phonemes.xml")/phonemes/phoneme
   where $phoneme/TimeInterval/start/@time = $ap/TimeInterval/start/@time return $phoneme}
</phoneme>
   

<xs:complexType name="timeInterval">
  <xs:sequence>
    <xs:element name="start" type="timePoint" minOccurs="0"/>
    <xs:element name="end" type="timePoint" minOccurs="0"/>
  </xs:sequence>
</xs:complexType>

<xs:complexType name="timePoint">
  <xs:sequence/>
    <xs:attribute name="time" type="xs:double" use="required"/>
</xs:complexType>
	
<xs:complexType name="Object">
  <xs:sequence>
    <xs:element name="index" type="xs:integer"/>
    <xs:choice minOccurs="0">
      <xs:element ref="TimeInterval"/>
      <xs:element ref="TimePoint"/>
    </xs:choice>
  </xs:sequence>
</xs:complexType>

Implementation

The OTIM framework is implemented within a Java / XML framework. The JAXB technology provides a strong and consistent link between XML Schema and Java representations. The XML management and querying is hosted within one eXist (http://exist.sourceforge.net/) server for web application or within an embedded eXist module for a standalone utilization. At this time, 15 minutes of the corpus have been annotated for 4 domains (phonetics, prosody, syntax, disfluency). The size of an XML document representing the 15 minutes can vary from 150 up to 500 Mo.

Conclusion

In this paper, our intention was to describe an XML engine dedicated to the representation and the concurrent querying of multimodal linguistic annotations implying to analyze and to find correlations between annotated linguistic domains.

From the signal and the signal transcription of a conversational speech between two persons, experts annotate data according to different linguistics domains like prosody, phonetics, morpho-syntax, discourse or gesture/posture. From the TFS provided by linguists that describe the knowledge of the studied domains, from legacy data and annotations, we automatically build an XML corpus which can be queried by means of an extension of XQuery, in a concurrent way. A java framework has been provided and enables to interface with both linguistic tools and OTIM XML information system. With this framework, the problem of the interoperability is solved for all the involved tools (PRAAT, ANVIL, ELAN) and domains within the OTIM project. Moreover, our proposal is based on XML standards.

Nevertheless, we can notice that our XML solution cannot capture the semantics of linguistic knowledge. Moreover, TFS expressivity is limited, for example for temporal relations. Object anchoring is absolute: interval boundaries are represented by the features start and end, with temporal value (an object can also be situated by means of a point). It would be useful to have a relative anchoring. Another limit is due to the underlying model of TFS which is a Directed Acyclic Graph (DAG). When linguists need to annotate co-references or disfluencies which are organized around objects, it would be useful to have an object anchoring which is conflicting with the acyclic graph.

We currently work to propose a knowledge representation formalism which be an alternative to TFS: an ontological approach based on Description Logics [Baader et al. 2003] (DL) and on semantic web technologies for the development of a linguistic Knowledge-based Information System. We have already defined a linguistic ontology from the TFS provided by linguists for prosody, phonetics, lexical and disfluency domains. An applicative framework is under development, it is based on semantic web proposals such as OWL DL (Ontology Web Language5) for the representation of this ontology and SPARQL [Prud’hommeaux 2007] the querying language of semantic web for its manipulation.