Balisage Paper: Message Format Persistence in Large Enterprise Systems

Focus on cardinality

Typical enterprise models focus on cardinality. In markup such as XML and JSON, as well as in RDF, cardinality is not typically important. XML representing someone with one address may look like this:

<person>
   <name>bob jones</name>
   <address> 123 Wall St, New York, NY </address>
<person>
            
        

and a relational model can also represent the same data:

The XML model is easily amended to hold an arbitrary number of addresses:

<person>
   <name>bob jones</name>
   <address> 123 Wall St, New York, NY</address>
   <address> 76 Seaside Terrace, The Hamptons, NY</address>
<person>            
        

Where in a relational model, changing cardinalities from 1:1 to 1:N requires additional tables and a join key to be in third normal form.

Clearly, cardinality is of critical importance to relational modeling as getting that wrong requires a disruptive change, and cardinality drives third normal form modeling, E-R diagramming and related practices. In contrast, cardinality is less important for XML or JSON message modeling.

Other modeling approaches such as UML are similarly infected with a concern for cardinality, as cardinality determines the data structure (Array vs. single value) needed in object-oriented code, so it must be called out on models that drive functions, classes and method implementation.

Readability and intelligibility

Documents are human readable, where E-R diagrams and related models are machine readable. This is because of hierarchy and the need to follow pointers or links.

The evidence for this is informal but compelling. Documents emerged organically from human societies – insurance applications, messages, contracts, bills of material, articles and books (including this one) have been organized hierarchically for centuries, and such organization suits the human mind well.

Pointers and tables, however, are confusing and difficult to follow for a human. One must often literally follow the lines on a complex E-R diagram to understand what is related to what. Many readers of this paper will recall the experience of walking up to a huge E-R diagram on plotter paper or taped together from many smaller sheets and attempting to trace the various connections to the tables and entities in an attempt to understand what is related to what.

Often, the maintainer of the diagram will try (with limited success) to group related tables into particular areas of the diagram, thus imposing one level of hierarchical organization on a spaghetti bowl diagram of inter-connected tables or entities. This minimal hierarchy is helpful, but illustrates that the fundamental approach is suited to relational modeling or object-oriented programming rather than document modeling.

Beware conventional modeling tools

We include this discussion here to suggest that we, as an industry, need to re-examination our typical modeling practices, and that document modeling should be elevated as the primary goal of modeling in a large enterprise, tools such as schema editors, UML, or new approaches should be considered that ease the development of document models, and each practitioner should be clear if they are building a “good model” informed by decades of relational modeling, or a “good model” based on how the document model and derived Message Models will function.

Processing

The process of calling the DSH and processing a request is conceived of as having four “legs.”

As one may imagine, there are other complexities such as multi-step processes where a message is split into many sub-requests which are combined using internal workflow and state persistence, deferred due to outage windows, routed to a multi-day, human-in-the-loop review processes, translated to REST, X12 or other formats and the like, but these four legs constitute the essential workflow of every call in the system.

How DSH stores message data directly

As noted above, messages flowing through DSH are stored with minimal modification as persistent data. Each of these four legs transmits and processes different data, but all are XML message formats and all a record of all messages for all legs are stored by the DSH.

If this data were stored, say, in relational form, a complex XML to relational mapping process would be needed for each and every message. Some designers would even introduce a third model – an object oriented model – in a Java tier in between the messages and relational persistence to parse the XML to POJOs only to then transform the data to tabular.

This additional mapping and modeling would have been cost and time prohibitive because of the modeling, the mapping, and the overhead and delay needed for every message or API change. Any change to an API or message format would require a new relational model, new mappings, new DDL to provision the model, and upgrade scripts and coordination to change existing data if the system was already in production.

Instead, saving the message with minimal modification has proven to be efficient and powerful. The business benefits include: enabling historic auditing and data lineage, monitoring the system for issues in real time, support of forensic investigations into data or system problems, and helping correct any consumer issues that may be found.

The portion of DSH that stores message traffic in this manner was designed and implemented in the final few months of the DSH development. The component of DSH is called the Event Management Framework, from the notion that receiving, processing and sending events are “events” within the DSH system. The bulk of system development prior to this feature also used documents as the primary persistent mechanism (for state management, workflow and other functions) but is beyond the scope of this paper.

Message format persistence

As described above, the persistent format leveraged and closely resembled the message model, and a NoSQL storage product was used that natively stores and transforms XML. The benefit was to use as much of the message model as possible, specifying only differences – how was this accomplished?

The design goal was to use existing structures to the extent possible, and specify only additional data and removed data relative to the message model. This is somewhat in the spirit of object-oriented modeling, where the differences between a subclass and superclass are specified in the subclass, and common structure and functionality is “inherited.”

<message-event xmlns=”http://cms.gov/dsh/EMF”>
  <metadata>
    <timestamp>2013-10-02T18:22:33.444</timestamp>
    <service>minimal insurance coverage confirmation</service>
    <success>successful</success>
	<stateOrTerritory>Texas</stateOrTerritory>
  </metadata>
  <result>
    <opm:insuranceCheck xmlns:opm=”http://opm.gov/insure>
      <opm:insuranceExists>true</opm:insuranceExists>
      <opm:typecode>44b</opm:typecode>
      <opm:ssn>REDACT FOR DSH<opm:ssn>
      <opm:insuredStartDate>
      [. . .] 
                

Rapid application development and flexibility

A “dashboard” capability was developed within the DSH atop this model to aggregate and filter system activity based on event types, destinations, times, success and error codes, and other information (primarily the <metadata> information illustrated above). These dashboards can answer questions such as: how many enrollment request to Kaiser Health were rejected (based on metadata holding error codes) for the state of Texas during the disk failover period from 1:15am to 1:37am on December 17th? Were they retried?

<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

 <xsl:param 
    name="removeElementsNamed" 
    select="'givenName|familyName|ssn|phone|addressLine1'"/>

 <xsl:template match="node()|@*" name="identity">
  <xsl:copy>
   <xsl:apply-templates select="node()|@*"/>
  </xsl:copy>
 </xsl:template>

 <xsl:template match=
 "*[name() = tokenize($removeElementsNamed, '\|')]"/>
</xsl:stylesheet>
        

Storing data “as-is”

Because the message formats did (and as agued above, should) exist prior to the persistence need, we can call this storing data “as-is” which emphasizes the benefit of aligning message and persistent models and storing documents.

DSH as enterprise integration using a message model

Conceptually, a portion of the US Federal Government spanning multiple agencies is functioning as a single enterprise by virtue of their integration with the DSH. Messages built on a uniform data model integrate all these systems, with the primary data consumer being the Department of Health and Human Services’ HealthCare.gov program.

This is therefore architecturally an Enterprise Integration effort based on design and use of a coherent message model, despite the systems involved being maintained and operated by different Federal and state agencies as well as private companies.

This is a version of enterprise integration where existing systems are integrated by a new model, but not an example of a system where a new model is pushed down by governance to existing systems, to be used internally across the enterprise. That is, the overall enterprise is integrated by a common message model, and the new analytics/tracking component then leveraged this message model for persistence. However legacy and externally-managed systems continued to use their own internal object and persistence models, and in fact continued to use their own message models, which were converted to a common model within DSH.

We now turn to an example where the message model is built a priori and used more extensively to support interoperability across the enterprise.

What is a Data Hub?

A Data Hub is a centralized persistent store that gathers data from numerous (disparate) systems of record or data silos, on a periodic basis or in real-time, and stores all data in one place and one form.

The Data Hub then becomes the preferred source for various kinds of data, and likely the only source for integrated, cross-line-of-business data. The Data Hub becomes the ideal place to access data due to its scalability, convenience and de-coupling from transactional systems, and upstream transactional systems typically remain the systems of record and handle new data capture, workflow, and end-user interactions.

Note that the prior example was the CMS “Data Services Hub” which is a specific computer system that loosely conforms to the Data Hub pattern; but here we have an insurance provider using a more generic “Data Hub” architectural pattern to build a central repository. The insurance provider Data Hub stores but the core business data for the insurer using message formats for efficiency (both computational and software delivery efficiency). In contrast the DSH stored a record of messages flowing into and through the DSH for analytics and monitoring.

The primary difference is that the DSH is for downstream analysis and forensics, where this Data Hub implementation provides real-time APIs that expose consolidated data to the rest of the enterprise to support service-oriented architecture (SOA) patterns and transactional workloads. This real-time, transactional support of a SOA is in contrast to a Data Lake or Data Warehouse which are typically analytic, and cannot support real-time,, transactional loads.

High-level architecture

Figure 3

Data Hub interaction with systems of record, systems of interaction, and one MDM system

The diagram above illustrates the insurance company’s Data Hub enterprise ecosystem, including four systems of record – one for Providers (doctors, pharmacists, dentists and labs), two for claims processing, and one for membership. In addition to these systems of record, it also includes a legacy Master Data Management (MDM) system that manages the merging and splitting of identities, and two interaction systems that consume data but do not store much of their own data. The interaction systems need access to consolidated data to provide accurate, up-to-date information, and the systems of record also have a (lesser) need for enterprise data. For example, the membership system may need to evaluate a member’s risk or claims history by accessing a member’s consolidated claims history.

This pattern allows for both operational isolation and increased functionality in the Data Hub. E.g. the Doctor Finder application includes geospatial and text searching, where the providers system of record does not. Some of the systems of record did not support online transaction loads that are supported by the more robust Data Hub infrastructure and software.

Not shown are bulk exports to a downstream analytic Data Lake based on Hadoop that performs most reporting and analytic functions. Since the data is exported to the Data Hub and the data hub has larger processing capacity than the systems of record, the Data Hub then exports data to the Data Lake without incurring an additional bulk read from the various systems of record.

Data Hubs are distinguished from ESBs and virtual databases in that they store data and index it in one place, rather than coordinating many queries and transforms in real time, provide unique indexing and analytics, and allow operational separation from source systems via separate disks and servers.

Core business objects used in message formats

The insurer’s Data Hub provides services using a consistent message format within well-defined APIs to the entire the enterprise. These messages may combine data from multiple data domains (such as membership, claims and payment). The messages have simple request/response headers wrapping a core Business Entity model, which is coherently designed – meaning naming conventions and overall style are consistent across all business entities, regardless of the services or business domain for a particular message.

Here is a notional response message which we see has some elements near the top that are generic to all requests returning lists of items (<total-results>, <page> and <page-size>), and the <result-item> element then holds a Business Entity or list of Entities – in this case medical claims.

<claim-history-response xmlns="http://insurer.com/response">
  <total-results>240</total-results>
  <page>1</page>
  <page-size>20</page-size>
  <result-list>
    <result-item>
      <claim:medical-claim xmlns:claim="http://insurer.com/claims">
        <patient-id>124799819</patient-id>
        <claim-date>2015-02-02</claim-date>
        <primary-problem code="icd-9">250.01</primary-problem>
        <provider-id>9981772</provider-id>
        <claim-line> 
          <procedure>
            <code system=HCPCS”>J1815</code>
            <description>insulin injection per 5 units</description>
          </procedure>
          <claim-amount unit=”USD”>253.28</claim-amount>
        </claim-line>
      </claim>
    </result-item>
  </result-list>
</claim-history-response>   
            
        

Physically, this Data Hub runs on a clustered, distributed database, is easier to scale up with massively-parallel or elastic infrastructure, has less frequent and more predictable outage windows, is fully highly-available (HA) and has other advantages as a system explicitly built for this kind of high-volume, consolidated data access.

As with the HealthCare.gov DSH, the insurer Data Hub runs on an enterprise-class NoSQL distributed database comprising multiple primary nodes and a similar number of standby nodes at a remote disaster recovery site, synchronized by database replication. This level of disaster recovery and clustering are not available for all the upstream, transactional systems, making the Data Hub more robust and available and also more easy to administer than a federation of the disparate upstream products and systems.

Common Messaging Formats

To provide common services using common messaging formats, the insurer’s Data Hub implementation also stores data in the messaging formats themselves. The above data structure has a clear delineation between message management data (in the http://insurer.com/response namespace) and business data (in per-domain namespaces such as http://insurer.com/claims).

However, there are two minor changes to the overall messaging format used to persist it, with specific motivations for each.

<claim:medical-claim xmlns:claim="http://insurer.com/claims">
<patient-id>124799819</patient-id>
<claim-date>2015-02-02</claim-date>
<primary-problem code="icd-9">250.01</primary-problem>
<provider-id>9981772</provider-id>
<payments>
    <claim-payment>
        <paid-date>2015-03-05</paid-date>
        <payee>
            <payee-type>group practice</payee-type>
            <payee-name>Yakenflaster Health Group</payee-name>
            <payee-id>787877211</payee-id>
        </payee>
    </claim-payment>
</payments>
<claim-line> 
    <procedure>
        <code system=HCPCS”>J1815</code>
            <description>insulin injection per 5 units</description>
    </procedure>
    <claim-amount unit=”USD”>253.28</claim-amount>
</claim-line>
</claim>
                    

<envelope>
    <header xmlns="http://insurance.com/db-header">
        <identifiers>
            <id>9981772</id>
            <id>787877211</id>
        </identifiers>
        <snomed-codes>
            <code src="250.1">44054006</code> 
            <code src="HCPCS">237599002</code>
        </snomed-codes>
        <triples> ... RDF data here ... </triples>
    </header>
    <business-entity>
        <claim:medical-claim xmlns:claim="http://insurer.com/claims">
            <patient-id>124799819</patient-id>
            <claim-date>2015-02-02</claim-date>
            <primary-problem code="icd-9">250.01</primary-problem>
            <provider-id>9981772</provider-id>
            <payments>
                <claim-payment>
                    <paid-date>2015-03-05</paid-date>
                    <payee>
                        <payee-type>group practice</payee-type>
                        <payee-name>Yakenflaster Health Group</payee-name>
                        <payee-id>787877211</payee-id>
                    </payee>
                </claim-payment>
            </payments>
            <claim-line> 
                <procedure>
                    <code system=HCPCS”>J1815</code>
                        <description>insulin injection per 5 units</description>
                </procedure>
                <claim-amount unit=”USD”>253.28</claim-amount>
            </claim-line>
            </claim>
    </business-entity>
</envelope>
                

Data Chunking for re-use

The data chunk approach provided substantial, but not completely arbitrary, flexibility in building different data extents. E.g. for the insurance company, the data chunks can be re-combined to support both a small Claim Summary message and also a larger, detailed Claim description message.

Data Chunks vs Basic Types

Determining the size of a data chunk is a key design choice. In Chessell15 atomic, re-usable entities are called “Basic Types.” Basic Types, in Chessell, et al, are exactly the un-changed smaller pieces of information that do not change from place to place in the model. They may be addresses, effective date ranges, telephone contact information and the like.

However, when substantively re-using a Message Model throughout the technology stack and persisting it (nearly) as-is, much larger chunks can be re-used and standardized than envisioned by Chessell et al. This is because in traditional systems, the models must be radically re-modeled (because traditional systems use a relational persistent model, a hierarchical XML messaging model and a graph-shaped Object Model – all of which are incompatible to some extent). The Data Hub approach uses the same fundamental model – a hierarchical document model – at all layers, therefore re-use can be far higher, with hierarchy being preserved across layers.

Re-use of the preeminent or pre-existing Message Model allows the chunk size and granularity of re-use to be much larger. Certainly tens or hundreds of data fields is acceptable in many cases for a chunk, rather than only a few. For the insurer, Data Chunks represented a “core” claim with vital claim information together with additional data chunks for conceptually separate claim processing information, claim lines, provider information, historic information, claim review information and so on.

This diagram illustrates that the persistent model data chunks are recombined to form different message formats, allowing re-use at the chunk level, but also flexibility in message formats.

Figure 4

This diagram illustrates that the persistent model data chunks are recombined to form different message formats, allowing re-use at the chunk level, but also flexibility in message formats.

These chunks are not the persistent documents, but rather are the building blocks of both persistent and message documents (the insurer used XML, so the messages and persisted entities were “documents.” If JSON were used they would be called objects or entities). This insurer was using XML Schema for modeling, so the chunks were defined as types and various master schemas imported these types to include them in the overall schemas.

Ideally, many messages will correspond exactly to the persistent form, with no additional chunks added, or extraneous chunks removed. That is, the persistent form should be chosen to include a set of chunks that is the most natural and commonly served message form. This way, message formation at runtime is as fast and simple as possible, and coding of transforms is minimized.

This is easiest when there is one well-known and expected Message format – e.g. healthcare systems use HL7 CCD documents to transmit patient summaries. Other message formats will require some re-combination of the chunks and will incur at least minimal performance overhead, though still far less than a wholesale transformation from relational to message formats, or even field-by-field transformation between disparate document formats.

Data Chunks as dynamic message construction

One typical form of combination is to create summary records, such as a Member summaries. The Member summary may contain some Member data chunks, and also have non-member data such as payment and recent claim activity. This cross-domain message is intuitively not ideal for persistence because it crosses domains and is very particular to a summarization use case.

By gathering the chunks of related entities into a summary message, the system is effectively summarizing and providing related information (e.g. last 20 claims) with core information (e.g. basic member information).

Relationships among data chunks can be represented with RDF (semantic triples) or other graph technologies. Documents + graphs combine to form a powerful data representation approach as documents represent trees well (but not graphs) and graphs model relationships naturally (but don’t highlight natural hierarchies as found in trees).

This approach is actually quite similar to Dynamic Semantic Publishing (DSP -such as implemented at the BBC and described in Rayfield12 http://www.bbc.co.uk/blogs/bbcinternet/2012/04/sports_dynamic_semantic.html) where documents are statically stored as XML or HTML and then dynamically assembled based on semantic rules and ontologies. Here we are dynamically “publishing” the message format needed rather than an article, e-book or web page. While a detailed analysis of similarities between dynamic message construction and DSP are beyond the scope of this paper, note that the DSP approach often uses RDF and ontologies to assemble information as needed.

Architecture implications

Using data chunks as the building blocks of documents provides re-use between the persistent model and the message models in an enterprise. Only the differences need be additionally modeled once the enterprise message model is complete.

To summarize how differences are specified in examples above, these differences include:

The key advantage being that the actual domain models comprising all business data did not need to be re-modeled or mapped in either system.

This architecture allowed development efficiencies where an additional model was not created, and computational efficiencies where messages in the Insurance Provider case were largely de-normalized and ready to serve via APIs and services.

This is a substantial conceptual shift from the traditional approach. Typically, and for decades, message models have been XML, JSON, EDI or similar, and persisted data has been relational as defined by an entity-relation (E-R) model. Often an object model is added to the mix (as though someone thought that when two models are good, three must be better). These models are all fundamentally incompatible and enterprise modeling techniques (such as Model-driven Architecture and logical modeling) formed to ease the mapping and impedance mismatch problems inherent in the formerly-ubiquitous underlying technologies.

In the DSH and insurance provider systems described above, where message models drive the architecture the goal was no longer to work around the model incompatibilities, but rather to manage the re-packaging and chunking of re-usable data. Arguably, this focus is more critical to the business and directly calls out areas of commonality and re-use, rather than having architecture and data modeling focused on remediating mapping problems.

Efficiencies from persisting message model components

The efficiency and agility of persisting message data comes from the two shifts implicit in the approaches described above.

First, there is no need to re-model the bulk of the data. Since integration is the primary concern in combining a number of systems into a coherent enterprise, the message formats can come first. The advantages of a hub-and-spoke approach dictate that a single common model is developed (modulo data chunks where different messages include more or less data as composed of different data chunks). Avoiding re-modeling the data is a large savings vis the typical approach, and also speeds up application development, mitigates risk and reduces complexity.

Second, the insurer’s approach, in particular, allows architecture to re-focus on the concerns of identifying re-usable data chunks and packaging and re-packaging them into a variety of messages. This packaging and re-packaging is more fundamental to the business than simple translations among incompatible formats, and is amenable to semantic technology that describes links and collections among chunks.

NoSQL technology alters the overall architecture

A key observation from this work is that NoSQL technology does not “plug in” to an existing architecture as a persistence alternative in place of a relational database. Instead, the overall nature of the architecture - starting with data architecture and system architecture but extending also to enterprise architecture – must change when NoSQL technologies are used.

The author’s experience is that this shift is challenging to navigate, and ideally these examples illustrate the different concerns and approaches that become paramount when the data models can naturally and easily cross “layer cake” boundaries – that is, when XML or JSON are used for messaging and can also be persisted (approximately) as-is. “Standard” techniques such as logical models and complex transforms or mapping technologies among relational, object and message formats no longer need to be the focus of architecture, and data re-use and re-packaging of existing XML or JSON structures rise to the fore. ^[1]