Managing electronic records business objects using XForms and Genericode at the National Archives and Records Administration

Balisage 2009

Quyen L. Nguyen
National Archives and Records Administration
quyen.nguyen@nara.gov

Betty Harvey
Electronic Commerce Connection, Inc.
harvey@eccnet.com

Presentation Date: August 11-14, 2009

Introduction

Motivation
- Business Requirements
- System Architecture
XForms and Genericode Approach
- Benefits
- Design and Implementation

Section 1: Motivation

Introduction

Business and Technical Perspectives:

ERA: fulfill NARA mission in the information technology age
Provide long term preservation and access to digital objects
Archivists and Agencies Record Managers will have a modernized system with automatic workflow that can streamline the digital archive business process.
SOA Framework and XML in ERA System

System Challenges

Section 1: Motivation

Evolution Relativity

Variety of domains and data types.
Rapid volume growth.

ERA Non-Functional Requirements

Section 1: Motivation

Evolvability
- New record types, data types, and services could be added
- New technologies in software and hardware could be inserted
Scalability. Ability to adapt to the growth of record volume and user community.
User-friendly
Secure
Highly available
Open standards

Archival Business Object (ABO) Requirements

Section 1: Motivation

Elimination of free text form.
Structured fields that conform to a pre-defined data model.
System validation of input data.
Elaborate validation based on embedded business rules during creation/update of business objects.
Presentation of ABO.
- Browser View: W3C HTML standard browser.
- Print View: XML --> XSL-FO --> PDF
CRUDVS (Create, Retrieve, Update, Delete, Versioning, Search) operations will be supported.
Storage model: XML Format

Authority Lists

Section 1: Motivation

Controlled vocabulary
Controlled enumerated values
Authority List Management
- Maintainability
- Responsible organization
Building queries from various forms of search terms
There can more than 20 authority lists in one ABO form

Simple Authority List

Section 1: Motivation

Name-value pairs:

Name	Value/Definition
NARA	National Archives and Records Administration
USPTO	United States Patent and Trademark Office
NOAA	National Oceanic and Atmospheric Administration
...	...

Complex Authority List

Section 1: Motivation

More than a single name-value pair

This table can be considered a complex code list because it contains values for the same information Container Name.

Container Code	Description	Depth	Width	Height	TotalCubicFoot
ACF-S	Archival Quality Cubic Foot Container, Standard	16.500	12.500	10.750	2217.188
AER-A	Aerial Film Can, Standard	5.500	5.500	10.000	302.500
AER-B	Aerial Film Can, 500	7.250	7.250	9.500	499.3438

Business Rules

Section 1: Motivation

Required or Optional data.
- GAO Concurrent Status
Inter-dependency between fields
- Agency Manual Indicator=yes, then: Manual Name, Manual Version, Item ID within Manual.
- Authority List filtered based on another field
- Calculation performed based on some fields being selected
Editable field based on state

Workflow

Section 1: Motivation

TR Workflow

BPM workflow
System orchestration

SOA Framework

Section 1: Motivation

XML at:
- Business level
- Communication level
- Storage model

In Search of an Approach for e-Forms

Section 1: Motivation

How to ensure data integrity?
- Validate input data
How to facilitate data exchange between components and services?
- SOA framework
- Data conversion
Performance: response time and user experience
Consistency
- Uniform and consistent error handling and error messages
Modularity and reuse
Low cost maintenance
Modality of User-interface
Storage Model
Standard support

In Search of an Approach for e-Forms and Code Lists

Section 1: Motivation

Separate evolvability
Separate modularity
Separation of control

Alternative Approaches for e-Forms

Section 1: Motivation


		
   <html>
   <body>
   <form action="createTR.php" method="post">
   Transfer Request ID: <input type="text" name="tr-id" />
   Agency: <input type="text" name="agency" />
   <input type="submit" />
   </form>
   </body>
   </html>


   <form action="createTR.jsp" method="post">


   <form action="createTR.asp" method="post">

Alternative Approaches: Discussion

Section 1: Motivation

Benefits

Can design with MVC
Leverage J2EE, .NET
Leverage power of procedural language for business logic

Aternative Approaches: Issues

Section 1: Motivation

How to add or change a field?
- Changes required throughout all layers of MVC
- Development cost
- Turn-around time
How to format field or section to make it attractive?
- Screen refreshes upon value selection in authority list
How to represent values in a code list?
Relation with XML data
- Marshalling of XML data
- Modification to schema requires code recompilation

Aternative Approach for Authority Lists

Section 1: Motivation

Use xsd:enumeration

Code list changes, schema changes
Schema version
How to implement complex code lists?

Use relational database

SQL in applications
Maintenance issue
Data exchange requires conversion

Problem Summary

Section 1: Motivation

Evolvable design
- Extensible
- Adaptive to changing business rules
- Cost effective
Fit into ERA SOA architecture
XML standard based

Section 2: Brief Introduction to XForms

Introduction

This section will provide a brief introduction into the XForms specification and the benefits of using XForms for creating business documents.

XForms Specification

Section 2: Brief Introduction to XForms

XForms is a W3C Specification
- Version 3 - dated 29 October 2007
- Version 2 - dated14 March 2006
- Version 1 - dated 14 October 2003

Benefits of XForms

Section 2: Brief Introduction to XForms

Described as ext generation of webforms
Standard Vocabulary
Reusable modules (XInclude)
Native XML Results
Use of MVC (Model-view-controller) - separation of business logic and user interface.
Used with other vocabularies, usually XHTML but could be custom vocabulary.
Behaves as an HTML form. Complexity is hidden from user.

XForms Controls

Section 2: Brief Introduction to XForms

Very similar to HTML forms but with more functionality.

XForms Control	Definition	Example
<xforms:input>	Free form text.	Link to Example
<xforms:secret>	Used to capture data while being hidden. Can be used to capture password information.	Link to Example
<xforms:textarea>	Free formed block of text in a multiline format.	Link to Example
<xforms:output>	Display information to the user.	Link to Example
<xforms:upload>	Provides the ability for a user to upload an external file.	Link to Example
<xforms:range>	Provides the ability for a person to upload a value from a range of values.	Link to Example
<xforms:trigger>	Provides a user triggered action	Link to Example
<xforms:select>	Provides the ability to allow the user to select multiple values from a list.	Link to Example
<xforms:select1>	Provides the ability to allow the user to select a single value from a list.	Link to Example
<xforms:submit>	Provides the ability submit data to a processor. Works in conjunction with the <xforms:submission> element.	Link to Example

Section 3: Genericode Introduction

What is Genericode

Section 3: Genericode Introduction

OASIS-Open Specifcation
http://docs.oasis-open.org/codelist/cs-genericode-1.0/doc/oasis-code-list-representation-genericode.pdf
Specification for encoding code lists in XML
XML W3C Schema

Section 4: Genericode and XForms

Genericode Structure

Section 4: Genericode and XForms

Three major sections:

Identification Section
Column Setup Section
The body of the code list.

CodeList Structure

Sample Genericode

Section 4: Genericode and XForms


 		
<CodeList xmlns="http://docs.oasis-open.org/codelist/ns/genericode/1.0/" 
 	xmlns:era="http://archives.gov/era/schema" 
 	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
 	xsi:schemaLocation="http://docs.oasis-open.org/codelist/ns/genericode/1.0/ 
                  ile:/G:/Projects/standards/Genericode/genericode.xsd">
    <Identification xmlns="">
         <LongName xml:lang="en">Holdings_Measurements</LongName>
      <Version>1</Version>
      <Agency>
            <LongName xml:lang="en">US National Archives Records Administration</LongName>
         <Identifier>1</Identifier>
       </Agency>
</Identification>

    <ColumnSet xmlns="">
         <Column Id="ContainerCode" Use="required">
         <ShortName>Container Code</ShortName>
         <Data Type="xsd:normalizedString" xml:lang="en"/>
      </Column>
      <Column Id="ContainerDescription" Use="Required">
         <ShortName>Container Description</ShortName>
         <Data Type="xsd:string" xml:lang="en"/>
      </Column>
      <Column Id="Depth" Use="Optional">
         <ShortName>Depth</ShortName>
         <Data Type="xsd:integer" xml:lang="en"/>
      </Column>
      <Column Id="Width" Use="Optional">
         <ShortName>Width</ShortName>
         <Data Type="xsd:integer" xml:lang="en"/>
      </Column>
      <Column Id="Height" Use="Optional">
         <ShortName>Height</ShortName>
         <Data Type="xsd:integer" xml:lang="en"/>
      </Column>
      <Column Id="TotalCubicFoot" Use="Optional">
         <ShortName>Total Cubic Footage</ShortName>
         <Data Type="xsd:integer" xml:lang="en"/>
      </Column>
    </ColumnSet>
    <SimpleCodeList xmlns="">
        <Row>
            <Value ColumnRef="ContainerCode">
            <SimpleValue>ACF-S</SimpleValue>
         </Value>
         <Value ColumnRef="ContainerDescription">
            <SimpleValue>Archival Quality Cubic Foot Container, Standard</SimpleValue>
         </Value>
         <Value ColumnRef="Depth">
            <SimpleValue>16.500</SimpleValue>
         </Value>
         <Value ColumnRef="Width">
            <SimpleValue>12.500</SimpleValue>
         </Value>
         <Value ColumnRef="Height">
            <SimpleValue>10.750</SimpleValue>
         </Value>
         <Value ColumnRef="TotalCubicFoot">
            <SimpleValue>2217.188</SimpleValue>
         </Value>
      </Row>
      <Row>
            <Value ColumnRef="ContainerCode">
            <SimpleValue>AER-A</SimpleValue>
         </Value>
         <Value ColumnRef="ContainerDescription">
            <SimpleValue>Aerial Film Can, Standard</SimpleValue>
         </Value>
         <Value ColumnRef="Depth">
            <SimpleValue>5.500</SimpleValue>
         </Value>
         <Value ColumnRef="Width">
            <SimpleValue>5.500</SimpleValue>
         </Value>
         <Value ColumnRef="Height">
            <SimpleValue>10.000</SimpleValue>
         </Value>
         <Value ColumnRef="TotalCubicFoot">
            <SimpleValue>302.500</SimpleValue>
         </Value>
      </Row>
      <Row>
            <Value ColumnRef="ContainerCode">
            <SimpleValue>AER-B</SimpleValue>
         </Value>
         <Value ColumnRef="ContainerDescription">
            <SimpleValue>Aerial Film Can, 500</SimpleValue>
         </Value>
         <Value ColumnRef="Depth">
            <SimpleValue>7.250</SimpleValue>
         </Value>
         <Value ColumnRef="Width">
            <SimpleValue>7.250</SimpleValue>
         </Value>
         <Value ColumnRef="Height">
            <SimpleValue>9.500</SimpleValue>
         </Value>
         <Value ColumnRef="TotalCubicFoot">
            <SimpleValue>499.343</SimpleValue>
         </Value>
      </Row>
...
</SimpleCodeList>
</CodeList>

Genericode and XML Fat

Section 4: Genericode and XForms

What is XML fat?

Layers unnecessary of XML hierarchy for the application layer?
Unnecessary is in the eye of the beholder.
- Genericode provides a 'standard' format to create, maintain and update code lists.
There are arguments for and against direct use of Genericode in the application layer.
- Agree not to argue amonst ourselves about the BMI (Body Mass Index) of Genericode
- Let's Move Forward

Reduced Fat Code List Approach

Section 4: Genericode and XForms

In many cases the fat content of the genericode is not a concern. However when organizations have multiple codelists and hundreds of codes for a single code list cutting the fat is essential for streamlined forms.

Developed a XSLT
Convert Genericode to 'fat free' version
Fat-free version less than 25% size of Genericode. (Container example 87 lines vs. 22 lines)
Expedites processing of code list.
Code names located in elements vs. attributes.

Conversion to Form Code List

Section 4: Genericode and XForms

Conversion To Form Code List

Fat Free Version of Container Code List Example

Section 4: Genericode and XForms


	 	
<FormCodeList codeListName="Holdings_Measurements" version="1">
      <Row>
        <ContainerCode>ACF-S</ContainerCode>
         <ContainerDescription>Archival Quality Cubic Foot Container, Standard</ContainerDescription>
         <Depth>16.500</Depth>
         <Width>12.500</Width>
         <Height>10.750</Height>
         <TotalCubicFoot>10.750</TotalCubicFoot>
      </Row>
      <Row>
        <ContainerCode>AER-A</ContainerCode>
         <ContainerDescription>Aerial Film Can, Standard</ContainerDescription>
         <Depth>5.500</Depth>
         <Width>5.500</Width>
         <Height>10.000</Height>
         <TotalCubicFoot>10.000</TotalCubicFoot>
      </Row>
      <Row>
        <ContainerCode>AER-B</ContainerCode>
         <ContainerDescription>Aerial Film Can, 500</ContainerDescription>
         <Depth>7.250</Depth>
         <Width>7.250</Width>
         <Height>9.500</Height>
         <TotalCubicFoot>9.500</TotalCubicFoot>
      </Row>
...
</FormCodeList>

Incorporating Form Codelist into XForm

Section 4: Genericode and XForms

XForms <xforms:instance> element can point to a file.
Can point to a fully-qualified URL
Point to the local file system

Fully Qualified URL for Code List

Section 4: Genericode and XForms

The two examples below are both fully-qualified URL's. The first example points to the eXist XML repository that contains the codelists. The second example points to a fully qualified URL.

In the example below the <xforms:instance> element points to the filesystem. The directory root is the directory where the XForms is located.

Full URL


	 
<xforms:instance id="Container-instance" 
      src="http://localhost:8080/exist/rest/db/home/era/CodeLists/ContainerFormCodeList.xml"/>
 
 <xforms:instance id="Container-instance" 
      src="http://www.myurl.com/codelists/ContainerFormCodeList.xml"/>

Code List Located on the File System

Section 4: Genericode and XForms

A code list can reside on the file system. The XForms engine will require access to the file system.

Pointing to the File System

	 	
<xforms:instance id="AccessRestriction-instance" 
     src="CodeLists/ContainerFormCodeList.xml"/>

Using the Code List Select1

Section 4: Genericode and XForms

Create an <xforms:select1> control. The <xforms:select1> control creates a radio button selection.
The 'nodeset' attribute is the fully-qualified XPath statement to retrieve a list of values from the codelist.
Notice in the sample below, the <forms:label> and <xforms:value> are the same. The label displays the value of the ContainerType.

Example of Select1 for Container Type


<xforms:select1 ref="era:ContainerType">
   <xforms:item>
      <xforms:label>Choose One...</xforms:label>
       <xforms:value/>
   </xforms:item>
   <xforms:itemset nodeset="instance('Container-instance')/Row/ContainerDescription">
      <xforms:label ref="."/>
      <xforms:value ref="."/>
   </xforms:itemset>
</xforms:select1>

Select1 Display

NOTE: Only 1 value can be saves.

Repeating Codes

Section 4: Genericode and XForms

If you have repeating values, you should use <xforms:repeat> to wrap each code into XML.


<xforms:repeat nodeset="era:ContainerInformation/era:Container" id="repeatContainer">
  <xforms:select1 ref="era:ContainerType">
       <xforms:item>
        <xforms:label>Choose One...</xforms:label>
        <xforms:value/>
       </xforms:item>
       <xforms:itemset nodeset="instance('Container-instance')/Row/ContainerDescription">
        <xforms:label ref="."/>
        <xforms:value ref="."/>
       </xforms:itemset>
      </xforms:select1>
</xforms:repeat>

<xforms:trigger>
     <xforms:label>Repeat Container</xforms:label>
     <xforms:insert nodeset="Paper"
            at="index('repeatContainer')"
            position="after" ev:event="DOMActivate" />
</xforms:trigger>

Display of Repeating Codes

Populating Values From Single Code

Section 4: Genericode and XForms

The <xforms:bind> element is uses to bind values from the codelist instance to the resulting XML document.

From the code <ContainerCode> we want to populate three values. We will need 1 <xforms:variable> and three <xforms:bind> elements.

Container Code List


<?xml version="1.0" encoding="UTF-8"?>
<FormCodeList codeListName="Holdings_Measurements" version="1">
   <Row>
       <ContainerCode>ACF-S</ContainerCode>
       <ContainerDescription>Archival Quality Cubic Foot Container, Standard</ContainerDescription>
       <Depth>16.500</Depth>
       <Width>12.500</Width>
       <Height>10.750</Height>
       <TotalCubicFoot>10.750</TotalCubicFoot>
   </Row>
        . . .
</FormCodeList>

Setting the <xforms:variable> element

Section 4: Genericode and XForms

The <xforms:variable> element is used to provide the capability to compare the instance value (form) against the codelist value. The codelist can reside in the database or on the filesystem.

name: The name you give the variable. This will be the name you use to reference the name in a calculation.
select: The XPath location to location value you want to set the variable too.

Set XForms Variable

         
 <xforms:variable name="containerType" 
                  select="era:ContainerInformation/era:Container/era:ContainerType"/>
    <xforms:setvalue ref="era:ContainerDimensions/era:Height" 
                     ev:event="DOMActivate"/>

Setting <xforms:bind>

Section 4: Genericode and XForms

The <xforms:bind> element must be used for every value in the codelist that gets set.

nodeset: The XPath statement for the element and/or attribute being set.
calculate: The XPath for determining the value that should be populated. Notice a variable is used to compare against. [ContainerDescription = $containerType]

XForms Binds


 <xforms:bind nodeset="descendant::era:Height"  id="bindHeight"
   calculate="instance('Container-instance')/Row[ContainerDescription = $containerType]/Height"/>
  <xforms:bind nodeset="descendant::era:Width"  id="bindWidth"
 calculate="instance('Container-instance')/Row[ContainerDescription = $containerType]/Width"/>
. . .

XForms-Genericode Demo

Section 4: Genericode and XForms

Demo of XForms-Genericode http://www.eccnet.com:8010/XForms/TransferRequestRedesign

In the words of Andy Kaufman

Section 4: Genericode and XForms

Thank You Very Much!

The End

Introduction

Section 1: Motivation

Introduction

System Challenges

Evolution Relativity

ERA Non-Functional Requirements

Archival Business Object (ABO) Requirements

Authority Lists

Simple Authority List

Complex Authority List

Business Rules

Workflow

TR Workflow

SOA Framework

In Search of an Approach for e-Forms

In Search of an Approach for e-Forms and Code Lists

Alternative Approaches for e-Forms

Alternative Approaches: Discussion

Benefits

Aternative Approaches: Issues

Aternative Approach for Authority Lists

Problem Summary

Section 2: Brief Introduction to XForms

Introduction

XForms Specification

Benefits of XForms

XForms Controls

Section 3: Genericode Introduction

What is Genericode

Section 4: Genericode and XForms

Genericode Structure

CodeList Structure

Sample Genericode

Genericode and XML Fat

Reduced Fat Code List Approach

Conversion to Form Code List

Conversion To Form Code List

Fat Free Version of Container Code List Example

Incorporating Form Codelist into XForm

Fully Qualified URL for Code List

Full URL

Code List Located on the File System

Pointing to the File System

Using the Code List Select1

Example of Select1 for Container Type

Select1 Display

Repeating Codes

Display of Repeating Codes

Populating Values From Single Code

Container Code List

Setting the <xforms:variable> element

<xforms:variable> attributes

Set XForms Variable

Setting <xforms:bind>

<xforms:bind> attributes

XForms Binds

XForms-Genericode Demo

In the words of Andy Kaufman