Balisage Paper: Introducing The UK National Archives Digital Records Metadata Vocabulary

Background

DRI and the Open Archival Information System

During the 1990s, the Consultative Committee for Space Data Systems (CCSDS) began working with the International Organisation for Standardization (ISO) to develop a standardised framework for the long term preservation of digital data. The initial driver was the need to provide long term storage of data gathered from space missions, but it quickly became apparent that the resulting framework would have a much broader application. At a CCSDS workshop in 1995 a proposal was put forward to establish a reference model for an Open Archival Information System (OAIS). This reference model would set out the functional components, internal and external interfaces, domain objects, and terminology needed to implement an archival information system. Drafting of the reference model was done in an open forum and was completed in 2003 when it was approved as international standard ISO 14721:2003[8].

OAIS stipulates six mandatory principles that an organisation must follows in order to operate a compliant archive. These are as follows [9]:-

Within The National Archives, the responsibility for much of this work is undertaken by the Digital Preservation team. It is their task to negotiate with the information producers (usually a Government Department) and obtain delivery and control of the records via a formal transfer process. Data normally arrives on an encrypted hard disk but can also come via secure file transfer [10]. The data is then loaded into a secure holding area and prepared for ingestion into DRI.

As well as general principles, OAIS provides a model of the information objects managed by an archive. This model describes an information package which is made up of the digital objects for preservation (the documents, images, videos or other files) together with metadata about these objects. There are three significant variants of this model which are as follows:

The information requirements at each stage are different. For example, the quantity and detail of information at each stage will vary significantly. The SIP will contain everything that has been received for archiving at any one time. This could be several gigabytes worth of files or it could be just a handful of files. When loaded into the archive this information package may represent the whole set of related data or it may be part of a much larger collection of data which is accumulating over time. When information is extracted from the archive it could be just a few records that someone has requested to see, for example via a Freedom of Information request, or it could be a large chunk of data that is perhaps destined for display on Discovery [10], the portal which provides online access to TNA's records.

Whatever the purpose, the information package must contain appropriate metadata to make the context of the information clear. Metadata is fundamentally important for making sense of the archived records and is key to satisfying the 4th, 6th, and to some extent 5th, mandatory principle laid down by OAIS.

DRI Metadata

When a block of digital files are transferred from an organisation to the Archives it is a requirement that a CSV file is provided with the data containing some metadata about each file or folder. The exact contents of this file will vary but there are currently six mandatory metadata fields for each item as follows [11]:

Additional fields may include technical information about the files, transcription information and any access restrictions that apply.

On top of this, additional metadata is generated by the ingestion process itself. The SDB software used by TNA for archive management uses a proprietary XML Schema for metadata known as the XML Information Package (XIP) which is based on the OAIS information package model. Tessella provides a tool called SIP Creator which can be used to generate a SIP suitable for ingestion into the archive, together with some basic metadata. The SIP creator takes a small number of parameters that provide some context for the information package, such as the name and identifier of the collection it will become a part of (e.g. War Office: Home Guard records, Second World War), and an accumulation reference which is used if some records from this collection have already been ingested into the archive. This information is placed into the XIP together with information about the folders and files that make up the digital records in the package. As the SIP is ingested into the archive various transformations occur to the XIP as part of the ingest workflow. These transformations perform many functions, most of which are beyond the scope of this paper, which all either enhance or re-arrange the content of the XIP file in some way so that it is ready for long-term storage with metadata describing, as far as possible, the structure, format, content, context, provenance, and sensitivity of the records it accompanies.

There is however one key transformation of the XIP file which is relevant to the topic of this paper and that is the step that adds the metadata from the CSV file provided with the records into the XIP. One element of the XIP schema, simply called Metadata, is deliberately left undefined in order that the users can define their own custom metadata. The guidance provided for the content of this element is as follows:

This element therefore is the natural home for the CSV content. First however it must be transformed into XML. This is achieved through a simple XSLT conversion which converts each row to the following format:

             <row>
                <elem name="identifier">file:/T:/LEV_3/content/Guidance/Letter.doc</elem>
                <elem name="file_name">Letter.doc</elem>
                <elem name="folder">file</elem>
                <elem name="date_last_modified">2013-05-13T14:26:56</elem>
                <elem name="checksum">1c933406517b2b3a4ca8b4fa61db8452</elem>
                <elem name="rights_copyright">Crown copyright</elem>
             </row>
        

Once this file is available it is a matter of a further transformation to convert this to the desired XML Schema. But what is the desired XML Schema? To begin with, a custom schema was created which is descibed in the next section. We will refer to this as XIP Metadata Version 1.0

            <dcterms:identifier>WO/409/27/1/1</dcterms:identifier>    
        

            <dcterms:identifier xsi:type=”tnacat:itemIdentifier”>
                <departmentCode>WO</departmentCode>
                <seriesCode>409</seriesCode>
                <pieceCode>27/1</pieceCode>
                <itemCode>1</itemCode>
            </dcterms:identifier>    
        

            <rdf:Description rdf:about=”http://example.org/book/1234”>
	           <ex:title>A Good Book</ex:title>
            </rdf:Description>    
        

            <rdf:Description rdf:about=”http://example.org/book/1234” ex:title=”A Good Book”/>
        

            <xs:complexType name="metadataType">
                <xs:sequence>
                    <xs:element ref="rdf:RDF"/>
                    <xs:element ref="c:closure" minOccurs="0" maxOccurs="1"/>
                </xs:sequence>
            </xs:complexType>    
        

What's in a Digital Archive?

Having resolved that we would have one vocabulary and one schema, and knowing that we had lots of concise metadata terms that had previously been defined in version 1.0, the next question was what exactly was it we were describing?

TNA uses the term deliverable unit to refer to something that can be retrieved from an archive. In the paper world this could refer to a single sheet of paper such as a letter, or it could refer to a notebook or it could even refer to a cardboard box containing a number of notebooks. Whatever it is, it is something that an archivist can reasonably hand over to a researcher. It would not be reasonable to expect the archivist to hand over a single sheet of paper torn from the notebook. You get to see the whole notebook, in its entirety, or none of it.

A deliverable unit also has what is referred to as a manifestation. The deliverable unit represents the idea of something that can be handed over, whereas in fact what you receive is a manifestation. This may sound confusing at first but if you consider that a notebook may have been copied then it follows that you may be get to see a copy of the notebook and not the original. This may be because the original is considered too fragile to hand over, or that the archive never received the original, or perhaps there just happens to be lots of copies! This is not so strange when you think that even ancient manuscripts were frequently copied [25].

To complicate matters further, what represents a deliverable unit in the physical world is not necessarily the same thing in the digital world. Although the notion of computer files and folders seems analogous at first to the idea of physical documents and folders, this is not always such a clear line to draw. For example, consider a notebook that has been scanned, page by page. Whereas in the physical world you have one clear deliverable unit, in the digital world you may have twenty image files, one for each page. It is no longer clear what is the deliverable unit, so a more pertinent question becomes, what is the record? In other words, what is the most sensible interpretation of a record, bearing in mind the document creator's apparent intention when creating the document or documents? It is this record that needs cataloguing, describing and generally enriching with metadata.

Through talking to the digital preservation specialists at TNA, reviewing the existing documentation and analysing the content planned for accession into the archive, it became apparent that there are four clearly defined types of digital record. Each of these record types has different requirements for the way it is archived and described. These types are as follows:

With these record types established, it became possible to create a basic object model onto which we could attach our metadata terms.

Version 1.0 of the TNA metadata guidelines established a number of XML Schemas for different metadata domains and, although this notion was dropped for convenience and maintainability in version 2.0, it was still considered useful by the digital preservation team to group these terms in some way. The hierarchical structure of XML allows for grouping simply by nesting elements and this convenience is carried over into RDF/XML. There is a further implication of this within RDF/XML however, as it is representation of RDF and therefore follows the Subject – Predicate - Object pattern of the triple. In RDF/XML a nesting within a subject indicates a predicate and the predicate indicates the object, which can be a literal value, such as a date or string as shown in the following example:

            <tna:BornDigitalRecord rdf:about="http://example.org/66/LEV/2/D4SL/Z">
                <tna:legalStatus>Public record</tna:legalStatus>
            </tna:BornDigitalRecord>    
        

Alternatively the object could be another resource with it's own predicates, in which case a URI is used to represent that resource as shown in the next example:

            <tna:BornDigitalRecord rdf:about="http://example.org/66/LEV/2/D4SL/Z">
                <tna:legalStatus rdf:resource="http://dbpedia.org/resource/Public_record"/>
            </tna:BornDigitalRecord>    
        

Another possibility with RDF is to use blank nodes [26]. A blank node indicates that something exists but does away with the need for a URI to identify it. It is an anonymous object if you like, but it can still have a specific type. In RDF/XML a blank node can be created simply by nesting an element representing the object inside an element representing the predicate as in the following example:

            <tna:BornDigitalRecord rdf:about="http://example.org/66/LEV/2/D4SL/Z">
                <tna:cataloguing>
                    <tna:Cataloguing>
                        <dcterms:title>Telegraph Media Group Ltd Submission</dcterms:title>
                    </tna:Cataloguing>
                </tna:cataloguing>
            </tna:BornDigitalRecord>    
        

By using blank nodes and creating classes within our ontology for these objects it might be possible to create a rich, descriptive and readable RDF/XML structure for the XIP metadata files. To see whether this was true, we first started by modelling the objects we had defined in our ontology.

fig. 1: Record Model

fig 1. OntoGraf [22] diagram of the record model.

The diagram above, displaying part of the vocabulary model, shows that a Record is a type of OWL Thing (every object in an OWL vocabulary is, by definition, a subclass of OWL Thing) and has four sub-types, as previously described. A Record can have a number of properties which are themselves objects (resources) that have their own properties. So a Record can have a Transcription object which itself will have a number of properties to do with transcription. The full current list of classes, object properties and data properties is detailed in Appendix B.

Once a model had been established using the ontology it was possible to create some prototype RDF/XML that made use of these terms. The results can be seen in Appendix C.

When presented to the Digital Preservation team, these prototype RDF/XML models were well received and found to be highly readable. Furthermore, this type of RDF/XML structure can be XML Schema validated. After all, it is only XML and TNA have tight control over the terms and structure of the RDF/XML they are creating. They are not trying to validate just any RDF/XML and so it poses no special validation challenges, once an appropriate schema is available.

Only minor alterations were proposed to the layout and terminology before taking this forward to production development. One of these amendments was to do with the creation of individuals within the vocabulary and there is more detail on this in Appendix D.

The Future

One of the significant challenges that currently faces the National Archives when handling metadata for digital records is that metadata changes. This change may be required because there was some error in the metadata originally, or perhaps some new metadata becomes available, or perhaps some fact asserted in the metadata changes. On the other hand though, it is not desirable for the tapes in the dark archive to be updated every time some minor modification is requested. This reading and writing to tape causes wear and tear which, if it happens frequently, could reduce the viable lifespan of the tapes and increase the risk of corruption. It would be very useful therefore to have somewhere to keep the metadata, or at least a copy of the metadata, where it could be accessed and edited without touching the archive. Perhaps these edits could be written to the tape at a low frequency when a sufficient quantity of them had built up to make the tape edits worthwhile (although whether this is the right thing to do, from an archival point of view, has yet to be decided).

Fortunately, much of the computer architecture for this is already in place at TNA. One of the bonuses of creating RDF/XML is that these snippets of RDF/XML that are being inserted into the XIP file could easily be copied elsewhere. The DRI project already has a Linked Data catalogue for maintaining processing and inventory information for DRI, based on Apache Jena Fuseki and the Jena TDB triple-store (the DRI Catalogue), so it would be quite simple to post these new RDF/XML snippets to the Linked Data catalogue. A better alternative may be to replicate this architecture and have a separate place to keep this rich metadata for easy access. With a SPARQL endpoint set up it would be a matter of submitting SPAQRL update queries to modify the information held. Probably the only significant components that would need to be developed from scratch are a web service to send SPARQL Protocol queries to the endpoint and a GUI for information management teams to add, delete and edit the metadata.

The RDF/XML metadata that is being placed into the XIP is usually the most interesting metadata of all. It is the human-entered information that can really tell you something interesting about the record, rather than the somewhat dry machine-generated technical metadata. If this metadata was placed in an editable repository using Semantic Web technologies, technologies that allow such things as context-aware searching, inferencing and entity recognition, many new facts would likely be unearthed.

Furthermore, having this data available in a Semantic Web format means that should TNA wish to do so it, it could publish this metadata on the Internet as Linked Data. Doing this would enable individuals and organisations in the wider world to search for and link to items buried deep within the archive. Having the technology to make connections between data held in different archives (and other institutions) has the potential to lead to valuable new insights into our past.

To make this process easier, The National Archives have decided to make their vocabulary publicly available. It is hoped that the experiences at TNA will be of benefit to other archival institutions and that they will make use of these terms, either by associating them with their own terms, or even using them as preferred terms.

The vocabulary and supporting XML Schemas are available on github at https://github.com/digital-preservation/dri-vocabulary.

Conclusion

The Semantic Web can trace it's roots back to ancient Greece. The field now known as ontology was first described by Plato (429-347BC) and later his pupil Aristotle (384-322BC) when they talked about modelling the world [28]. It is also firmly rooted in natural language and logic and as such, it is reasonable to assume that its constructs will make sense to future generations who discover it buried deep in a digital archive.

That and the fact that Semantic Web technologies such as RDF/XML and OWL can help to solve the practical problems faced by archivists today, namely creating concise, readable metadata that can be easily generated and automatically validated, make them an excellent choice for this situation.

Add to that the potential to open the archive to far more meaningful, context-aware searches and the ability to make connections between pieces of information held in different repositories, maybe in different countries, and suddenly a good idea sounds like a very exciting one with enormous potential for the future.

An organisation like the National Archives is dedicated to preserving the national memory of the United Kingdom. Like human memory, a national memory is important for many reasons. There will be good memories and bad. Some memories make great stories and some will be painful to recall. All memories though serve the fundamentally important tasks of reminding us who we are and how we have learnt to do the things we do. If anything can be done to sharpen our national memory, it can only be for the good of us all.