Balisage Paper: Agile Business Objects Management Application for Electronic Records Archive Transfer Process

Archival Business Objects

Before the transfer of any set of records, the Producer (usually a government agency) has to submit two business documents, namely Record Schedule and Transfer Request to NARA. The Record Schedule contains instructions the general disposition and maintenance of various types of records, such as permanent versus temporary records, and the retention period. Over the lifecycle of the Record Schedules, Transfer Requests will be created for every physical transfer to specify:

Transfer mode can be electronic wiring or via physical media such as audio cassette, microfilm, CD, DVD, video cassette, parchment, or photographic print. Each business object goes through multiple business processes and negotiation of disposition between NARA and the producer before records are transferred from the contributing entity to NARA. Access restrictions may be applied to records that contain privacy data.

Within the ERA system, Record Schedule, Transfer Request and other archival business objects (ABO) are implemented as e-Forms. Currently, ERA e-forms are encoded using JSP (Java Server Pages) pages One main advantage of e-Forms over traditional paper documents is that e-Forms facilitate deterministic processing by the system thanks to:

The ERA users manage these ABOs via the Archival Business Object Management Application defined by:

With respect fo flexibility and extensibility, the design should allow adding or modifying a field to an ABO form to be done at low development cost. Moreover, changes to business rules that govern the forms or the fields within a form should be easily accomodated without requiring a lot of coding effort.

Authority Lists

Authority Lists, also known as code lists consist of values used to establish normalized values for certain key fields in a business object. The goal of Authority Lists is to institute a controlled vocabulary to be used in e-Forms and archival descriptions which is part of the NARA metadata. Examples of authority lists are:

50 states with 2-letter abbreviations.

Federal agency names.

On the access side, Authority Lists play an important role in building queries from various forms of search terms. For instance, with the use of Authority Lists, searching for NARA is the same as National Archives and Records Administration. Due to its importance and sometimes fluidity of the data, the ERA system is required to allow privileged business users the ability to manage the Authority Lists. Any update to the Authority Lists should take effect immediately and be made available to future creation and updates of business objects and forms.

Functional Requirements

Notably, the design for the application should exhibit low cost for development and maintenance. Moreover, since ERA is an Service Oriented Architecture (SOA) system with Business Process Management (BPM) and Enterprise Service Bus (ESB), such design should show an ease of integration with the workflow. The functional requirements for ABO can be summarized as follows:

The following diagram depicts the components and services of the ABO Application from the SOA layer pattern perspective. The applications for Business Object Management, Business Object Review, and Business Object Approval can be linked together by a Business Process, which can be expressed in (Business Process Management Notation (BPMN) 10 and executed by a BPM engine.

Figure 1

Archival Business Object Management

Overview

XForms 2 is a W3C specification for implementing an XML-based Web forms. In some sense, XForms can be viewed as a next-generation of HTML Forms. While HTML Forms is a mix of markup for presentation and form data. XForms takes the approach of separating data from presentation. Data in XForms can conform to an XML schema which makes data validation against the schema pretty straightforward. For Ajax server-side XForms 4, real-time validation would provide immediate feedback in case of error in user input. Therefore, the user doesn't have to finish a lengthy form to know that an error has occurred. Note that in the case of an ABO, a form may span more than one scrolled page. Unlike the HTML form, the output of an XForms is an XML instance ready to be stored in an XML-aware repository. On the other hand, XForms has constructs to control user input events; but, these controls are not dependent on the particular presentation modality. Therefore, it is possible to create a form that will take input from various input devices, such as keyboard and voice. In the paper Multimodal Interaction with XForms 6, Mikko Honkala et al. proposed XForms Multi-modality (XFormsMM) to facilitate concurrent support of multiple modalities using XForms. The XFormsMM architecture comprises of the following components:

The application of the multi-modality is to enable a web form to support various UI devices such as desktop, wireless handset, IVR applications, etc. In the case of our application, this interesting feature might be exploited to develop user interface providing access to all users, impaired and non-impaired.

Advantages

The advantages of XForms have been identified in the literature and past conferences 4 5:

Overview

Genericode 3 defines a standardized model to manage code lists using a defined XML schema. Essentially, the idea is for a code to have a code key, and multiple code values. Every XML project has controlled lists that need to be supported in the XML application. There are two schools of thought for controlling code lists. The Universal Business Language (UBL) 19 is an OASIS-Open standards effort to describe business documents, e.g. invoice, purchase orders, etc. in XML. In version 1.0 of the standard, all the code lists for codes such as country codes, currency codes, etc. were embedded in the XML schemas. As individual countries started adopting UBL, it became apparent that placing large code sets in the schemas was a problematic approach. Some of the codes change rapidly and during implementation this required modification of the "standard" schemas which theoretically meant they were no longer UBL compliant. The definition of the codes were in the documentation portion of schema and not readily available to the application. Modifying schema to support changing codes became a configuration problem.

Members of the UBL technical committee developed the Genericode concept and it was adopted by UBL and other organizations. Genericode has now its own OASIS-Open technical committeer20 , and is currently at version 1.0.

Why is Genericode Valuable to NARA Enterprise

Besides the codes that are located in the business objects and metadata descriptions of records, NARA receives records from agencies in the U.S. Federal government, as well as some private collections. Every record set has its own set of codes. The codes themselves can have the same meaning but different codes across multiple records. For example, if we look at the records of ship manifests from the Irish Potato Famine, a code as simple as the age of the person can have several codes representing the age and/or event depending on individual ship manifests.

The logical expectation is that a person"s age is pretty straightforward. However looking at the table below, which contains actual values of codes from the passenger list maintained by NARA r21 it is readily apparent and a person"s age in a record isn't as cut and dry as some would expect. For example you wouldn't expect to see a value of 900 to represent a person's age.

The table below shows a representation of how the Genericode is organized. The advantage of the Genericode approach is that multiple codes can represent a single concept.

The Genericode standard has 3 major sections:

If we look at the table Table IV above, the Genericode representation would be:

<CodeList xmlns="http://docs.oasis-open.org/codelist/ns/genericode/1.0/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <Identification>
        <ShortName xml:lang="en">Age Code List</ShortName>
        <LongName xml:lang="en">Age Code List Irish Famine, period 1/12/1846 - 12/31/1851</LongName>
        <Version>0.3</Version>
        <CanonicalUri/>
        <Agency>
            <LongName xml:lang="en">
                United States National Archives Records Administration
            </LongName>
            <Identifier>1</Identifier>
        </Agency>
    </Identification>
    <ColumnSet>
        <Column Id="Description" Use="Required">
            <ShortName>Description</ShortName>
            <Data Type="xsd:string" xml:lang="en"/>
        </Column>
        <Column Id="Ship1" Use="Optional">
            <ShortName>Ship 1</ShortName>
            <Data Type="xsd:string" xml:lang="en"/>
        </Column>
        <Column Id="Ship2" Use="Optional">
            <ShortName>Ship 2</ShortName>
            <Data Type="xsd:string" xml:lang="en"/>
        </Column>
        <Column Id="Ship3" Use="Optional">
            <ShortName>Ship 3</ShortName>
            <Data Type="xsd:string" xml:lang="en"/>
        </Column>
    </ColumnSet>
    <SimpleCodeList>
        . . .
        <Row>
            <Value ColumnRef="Description">
                <SimpleValue>age 03</SimpleValue>
            </Value>
            <Value ColumnRef="Ship1">
                <SimpleValue>3</SimpleValue>
            </Value>
            <Value ColumnRef="Ship2">
                <SimpleValue>003</SimpleValue>
            </Value>
            <Value ColumnRef="Ship1">
                <SimpleValue>3</SimpleValue>
            </Value>
        </Row>
        . . .
    </SimpleCodeList>
</CodeList>

Advantages

As stated above, enumerated data can be problematic in XML documents. One approach is to have allowed values for a data element be enumerated in an XML schema, so that associated XML documents and their enumerated values can easily be validated by a standard XML parser. However, if there is a need to add, remove, or change values to the enumeration lists, then the schema has to be modified. In 12, the author proposed different solutions to extend the enumeration lists in an XML schema, by using XML mechanisms such as <xsd:union>, <xsd:pattern>, or <xsd:annotation>. The author argued that some of these solutions are advantageous because they require only one pass validation, thus avoiding performance penalty.

On the contrary, Genericode offers an approach that allows the management of the enumerations to be independent from the XML schema. Although this would imply a separate validation for the code list values, it does have some advantages 13:

Another key advantage for NARA to maintain all their code lists in a "generic" common format is the ability to create a standard NARA-wide authoring environment for developing and maintaining code lists across the enterprise. NARA can ultimately have thousands of code lists when preserving and describing electronic records for the entire federal government.

Advantages

By combining XForms and Genericode, our approach can benefit the advantages of both XForms and Genericode. Indeed, the introduction of Genericode into XForms provides a separation of concerns in the software and data development:

Development Process

The development process for XForms-based approach consists of the following steps:

Implementation

Our implementation of XForms/Genericode uses the Apache Tomcat application server as the infrastructure. The Orbeon XForms Server is used as a platform for managing the forms. We chose Orbeon over other XForms applications for the following reasons:

We chose to use a standard XML database as the repository for storing XML components. Initially we used eXist Open Source repository then moved to MarkLogic (commercial software) for the prototype that integrates XForms and BPM section “XForms and Business Process Management (BPM)”:

The figure below shows the interaction of the forms to the Genericode.

Figure 2

XForms+Genericode Architecture

Maintaining the code list external to the form provides the ability for the use of the same code list in multiple forms or multiple applications. When a code list is updated, all the forms that consume the code list will be automatically updated without any recompilation of code.

Populating Fields from Code List Lookups

Certain fields can be automatically populated based on the selection of a single code. There are several places when this becomes important for NARA. A good example is the "Record Group ". NARA classifies all records by a number which represents a title of the record group. A record group number can represent an agency or a collection records. For example the record group 21 represents" Records of District Courts of the United States ". Once a record group is assigned it never changes. Every federal agency is associated with one or more record groups (usually just one).

In an ABO form, the user selecting his/her agency will only be presented with the record group(s) associated with his/her agency. Once they select the record group, the record group title is automatically populated in the XML. Below is the XForms code which provides this functionality:

    <xforms:variable
      name="AgencyName"
      select="../era:OrganizationInformation/era:AgencyInformation/era:AgencyName"/>
    <xforms:select1
      ref="instance('TransferRequest-instance')//era:RecordGroupNumber"
      appearance="full">
        <xforms:itemset
          nodeset="instance('AgencyRecordGroup-instance')/Row[AgencyName
          = $AgencyName]/RecordGroup">
            <xforms:label ref="."/>
            <xforms:value ref="."/>
        </xforms:itemset>
    </xforms:select1>
</xforms:group>

The example above shows that the pull-down list is being populated from the AgencyRecordGroup-instance XML codelist. This codelist is set in the XForms model section using the <xforms:instance> element shown below.

<xforms:instance
  id="AgencyRecordGroup-instance"
  src="http://localhost:8080/exist/rest/db/home/era/CodeLists/Agency-RecordGroup-Form.xml"/>

The codelist is being pulled from an eXist XML repository. An major advantage to having the codelist maintained in a external XML file is that if an agency name changes (and they do quite often), the form does not require modification. Once the code list is updated, all forms using the code list are automatically updated. The record title gets populated by using an XForms <xforms:bind> function using an XPath statement. The attribute calculate attribute is used to set the value.

<xforms:bind
  nodeset="instance('TransferRequest-instance')/era:IdentificationInformation/era:RecordGroupTitle"
  calculate="instance('RecordGroupTitle-instance')/RecordGroupInfo[RecordGroupNumber
  = $RecordGroupNumber]/RecordGroupTitle"/>

The XPath statment in the calculate statement above basically says get the value of the record group title from the code list instance called "RecordGroupTitle-instance" where the <RecordGroupNumber> in the list matches the "RecordGroupNumber" variable. The $RecordGroupNumber has been set previously in the form.

Business Rules

The ABO forms must follow business rules that dictate the dependencies between the fields within a form, and also between the fields in one ABO to another. For example, if the "Required" indicator of a group of fields is checked, then valid values must be supplied to all fields within that group.

Some of the rules for the interaction of code lists in forms can be quite complex. The use of XForms and XML code lists allow the ability to define these rules using XPath statements.

Figure 3 show a pull-down selection bar for access restrictions. The form components change based on the selection in he pull-down list. The values of the pull-down list are populated by a code list for access restrictions. There are business rules associated with what information needs to be completed based on the value of the access restriction. For instance, if the user selects " Presidential Records Act (p)(3) Statute " the user must select a statutory citation. (See Figure 4)

Figure 3

Example Pull-down List

Figure 4

Statutory Citation pops up.

The XForms construct that controls whether the field is displayed based on the selection is below:

<xforms:group ref="era:AccessRestriction[contains(., 'FOIA(b)(3) Statute')] |
era:AccessRestriction[contains(., 'Presidential Records Act (p)(1)
  ')]">
. . .
</xforms:group>

The XPath above states that if the access restriction contains "FOIA (b)(3) Statute" or "'Presidential Records Act (p)(1)" then display the field.

Figure 5 shows the user selecting "PRMPA- Personal Privacy (D)". You will notice that in Figure 6, nothing is provided the user.

Figure 5

PRMPA - Personal Privacy (D) Selection

Figure 6

PRMPA- Personal Privacy (D) Selection Result Screen

Asset Catalog Entry

An ACE is represented by an XML document that conforms to a well-defined XML schema. At the high level, the structure of an ACE is compliant with the various kinds of metadata as described in the OAIS information reference model 14. The following aspects have been considered in the design of the ACE structure:

Archival Management Taxonomy

The design of ACE structure should allow the classification of digital objects in ERA according to different taxonomies. Archival Management Taxonomy is a special taxonomy that is mostly used by archivists, or record managers from federal agencies. A public user would be most likely interested in other taxonomies associated with the domain of the content of the data. In order to support the Archival Management Taxonomy, an ACE must include information extracted from the ABOs (Record Schedule and Transfer Request) that were created before the digital object represented by this ACE got ingested. With this organization, we can develop an application that allows business users to browse all digital objects grouped under a set of related ABOs.