Balisage Paper: XML Solutions for Swedish Farmers

The Swedish Federation of Farmers

Quoted from the LRF website at www.lrf.se (also see id-lrf-presentation):

The members' operations range from small-scale farming to large operations handling crop, livestock, and more. They frequently rely on state and EU funding to subsidise their operations; however, to receive funding, they need to comply with any relevant regulations, requiring inspection both by themselves and by appointed officials.

Checklists

To ease the compliance-related tasks for its members, LRF provides a website (id-miljohusesyn-nu) with regulatory audit schemes, checklists with questions intended to highlight every relevant regulation, each of them accompanied by help texts that offer more detail. Today, around 11,000 LRF members have registered on the website to use the services. Their registration data is used to filter the checklists and the associated facts to only include the relevant contents; for example, a dairy farmer should not have to answer questions about beekeeping.

Both the checklists and their associated facts are output in PDF files that are either filtered for the individual member, excluding irrelevant questions and facts, or output as a total checklist document, a 130+ page monster that includes everything.

Figure 1: Checklist Questions in the PDF...

For every checklist question, there are three possible answers: Yes and N/A indicate compliance and non-applicability, respectively, while no identify areas where actions are required before approval. For these, members can add comments, notes, expected completion dates and other information.

The questions always include help texts to aid the members in completing the checklists. These help texts, called facts in LRF parlance, appear on the web as pop-ups, and in the PDF as separate sections.

Figure 2: ...And Their Matching Help Texts

System Overview

The web checklist answers, including the member partial or complete checklists and their personal and company information, are handled by and stored in an SQL database. The checklists themselves, as well as their associated facts and other auxiliary information appearing in both the web and the PDFs, are fed from an eXist-DB XML database that converts XML contents to the required output formats and supplies links to these using an extended XLink linkbase when getting a request from the SQL database.

Figure 3: System Overview

The XML, split into roughly a thousand modules, is written in a DocBook-based authoring environment in oXygen Author and stored in a work area in the eXist database. Once new texts are ready, they are copied to a publishing chain area in eXist, where they become part of the content output to the web and to the PDFs.

The website also includes a large bank of reference information ranging from legal texts to links, help texts for the site, news and more, all of which is related in some way to the compliance process driving the checklists but that is not included in the published checklists. This information is written in and fed by a Wordpress CMS that seamlessly coexists with the SQL-DB and eXist DB feeds.

eXist Layout

The eXist database has the following collection layout:

Figure 4: eXist-db Collection Layout

The eXist layout is split into several parts:

The Publishing Chain

The principle of the publishing chain is simple enough. There is an export map that contains the raw data to be published. The data is stored in subcollections according to information type: Checklists, facts and standard texts copied from the work area are stored in their own subcollections, and the user data (metadata and their checklist answers) imported from the SQL database are stored as combined files per user in users.

Each user is identified using a unique user ID generated by the SQL database. A convention used everywhere in the system is that uid0 designates the total checklist with all of the questions and facts, while uidXYZ identifies the subset for some user with the uid XYZ.

In the older oXygen/eXist system as well as this one, the raw data in export is then converted by a chain of steps, each converting a specific type of information for a specific user, basically either the total, uid0, or uidXYZ, and the results stored in corresponding subcollections in pub. This is the basic process:

As mentioned, the uidXYZ process runs on demand, when a user logged onto the checklist website orders a PDF version of his answers, while the uid0 total checklist document is produced nightly and published on the website for those members that have not yet registered for the online checklist services.

Each step above consists of an XQuery wrapper, called by the SQL-DB as an http request during the synchronisation and publishing processes, that in turn invokes XSLT stylesheets that do the actual conversion. Frequently, the XQuery will prepare the indata in some way, perhaps adding base URIs or filtering the indata according to some criteria based on the regulatory checklist branch (mhs or rhs; see above).

The facts and the standard texts are also converted to HTML for use by the SQL database for the web versions of the checklists, where they are used as context-sensitive help texts. The publishing process is similar to the above, using one XQuery per step and information type, storing the results in their respective subcollection inside the html collection. Since the facts and standard texts are written using a DocBook derivative, the XSLT used is a very slightly updated DocBook XSLT 2.0 package^[6].

Whenever the website requires an HTML fragment, the SQL database sends an http request to eXist, invoking an XQuery that reads a linkbase listing every resource in the system (see section “Node IDs and the Linkbase”), and gets a link to the converted HTML in return.

Node IDs and the Linkbase

Early on, it occurred to us that an extended XLink linkbase that listed the resources in eXist would be useful. For example, to use a fact HTML file as a help text on the web requires only the link to the HTML in eXist, so looking that link up in one place made sense. Similarly, when authoring a checklist question in oXygen, locating the associated fact's URI in a single linkbase file rather than searching a number of collections inside the work area seemed like a sensible alternative.

So, the most interesting resources (and relations between them) for lookup by the SQL database or others are the checklists and their associated facts. , using @node-id values as keys.

<?xml-model href="http://localhost:8080/exist/rest/db/lrf/system/common/schemas/lrfbook/lrfbook.rnc" type="application/relax-ng-compact-syntax"?>
<section 
    xmlns="http://docbook.org/ns/docbook" 
    text-type="checklist" 
    version="lrfbook" 
    operation="G">
    <titleabbrev>G</titleabbrev>
    <title>Allmänna Gårdskrav</title>
    <qandaset 
        text-type="group" 
        xml:id="d1e17" 
        node-id="node-id-G1-2015-03-26-0100">
        <title>Anmälnings- och tillståndsplikt</title>
        <para>Reglerna nedan berör enbart anmälnings- och tillståndspliktiga företag. Syftet är att
            minska risken för att miljön förore­nas eller att andra betydande olägenheter för
            människors hälsa eller miljön uppstår. </para>
        <qandaentry 
            text-type="question" 
            xml:id="d1e31" 
            node-id="node-id-G1-1-2015-03-26-0100"
            cc="true">
            <question>
                <para>Om verksamheten är anmälningspliktig, har anmälan gjorts till kommunen?
                    (Tvärvillkor att anmäla djurhållning)</para>
            </question>
        </qandaentry>
        <qandaentry 
            text-type="question" 
            xml:id="d1e50" 
            node-id="node-id-G1-2-2015-03-26-0100"
            cc="true">
            <question>
                <para>Om verksamheten är tillståndspliktig, har tillstånd lämnats av länsstyrelsen?
                    (Tvär villkor att anmäla djurhållning)</para>
            </question>
        </qandaentry>
        ...
    </qandaset> ... 
    <qandaset 
        text-type="group" 
        xml:id="d1e1058"
        node-id="node-id-G10-2015-03-26-0100">
        ...
    </qandaset>
</section>

<?xml-model href="http://localhost:8080/exist/rest/db/lrf/system/common/schemas/lrfbook/lrfbook.rnc" type="application/relax-ng-compact-syntax"?>
<sect1 
    xmlns="http://docbook.org/ns/docbook" 
    cc="true" 
    node-id="node-id-G1-1-2015-03-26-0100" 
    text-type="fact" 
    version="lrfbook" 
    xml:id="d729e2">
    <title>Anmälningspliktig verksamhet</title>
    <para>(Tvärvillkor)</para>
    <para>Djurhållning över 100 djurenheter kräver en anmälan till kommunen. Anmälan om t.ex. utökad
        djurhållning ska göras minst 6 veckor före planerad start. Verksamheten bör invänta
        kommunens beslut i ärendet, då beslutet kan innehålla särskilda villkor. Kommunens beslut
        kan även överklagas av t.ex. närboende.</para>
    ...
</sect1>

Linkbase

An extended XLink linkbase at its simplest is basically a list of points of interest, pointers to resources called locators. They are pointed out using a URI and possibly some kind of fragment identifier or pointer inside the resource, frequently to ID attributes. Optionally, the linkbase may then define relationships between these locators, called arcs, and there are also a number of other features having to do with giving the locators and arcs roles as well as other information about their expected behaviour. For a far better introduction to XLink, please read the specification (see id-xlink).

The linkbases (there is one for each regulatory branch of checklists) look like this (the actual file in this case contains some 3500 locator entries):

<linkbases>
    <linkbase 
        xmlns:xlink="http://www.w3.org/1999/xlink" 
        xlink:type="extended">
        <locator 
            xlink:type="locator"
            xlink:href="http://localhost:8080/exist/rest/db/lrf/mhs/export/facts/A10-10-Fyrhjuling-och--node-id-A10-10-2015-04-16-0200.xml"
            node-id="node-id-A10-10-2015-04-16-0200" 
            res-type="fact"
            xlink:label="A10-10-2015-04-16-0200-fact"/>
        <locator 
            xlink:type="locator"
            xlink:href="http://localhost:8080/exist/rest/db/lrf/mhs/export/facts/A10-9-Rekommendation--node-id-A10-9-2015-04-16-0200.xml"
            node-id="node-id-A10-9-2015-04-16-0200" 
            res-type="fact"
            xlink:label="A10-9-2015-04-16-0200-fact"/>
        <locator 
            xlink:type="locator"
            xlink:href="http://localhost:8080/exist/rest/db/lrf/mhs/export/facts/A10-8-Kontroll-av-dra-node-id-A10-8-2015-04-16-0200.xml"
            node-id="node-id-A10-8-2015-04-16-0200" 
            res-type="fact"
            xlink:label="A10-8-2015-04-16-0200-fact"/>
        ...
        
        
        <locator 
            xlink:type="locator" 
            xlink:href="http://localhost:8080/exist/rest/db/lrf/mhs/export/checklists/checklist-A-uid0.xml#d1e5485" 
            node-id="node-id-A10-8-2015-04-16-0200" 
            res-type="checklist" 
            xlink:label="A10-8-2015-04-16-0200-question"/>
        <locator 
            xlink:type="locator" 
            xlink:href="http://localhost:8080/exist/rest/db/lrf/mhs/export/checklists/checklist-A-uid0.xml#d1e5493" 
            node-id="node-id-A10-9-2015-04-16-0200" 
            res-type="checklist" 
            xlink:label="A10-9-2015-04-16-0200-question"/>
        <locator 
            xlink:type="locator" 
            xlink:href="http://localhost:8080/exist/rest/db/lrf/mhs/export/checklists/checklist-A-uid0.xml#d1e5502" 
            node-id="node-id-A10-10-2015-04-16-0200" 
            res-type="checklist" 
            xlink:label="A10-10-2015-04-16-0200-question"/>
        ...
        
    </linkbase>
</linkbases>

The example lists three checklist questions and their associated facts. Note that the checklist locators include fragment identifiers to pinpoint the question inside the checklist. Also note the @res-type attribute that identifies the type of resource^[7] that is listed.

Similarly, the linkbase lists every resource in the work area.

The linkbase does not currently include the converted versions of the resources in html and its subcollections. Instead, it relies on the nightly synchronising process to do the actual conversion and simply rewrites the link that is returned to the SQL database to point at the HTML equivalent, with the query returning a 404 if there is no HTML version at the target location.

A query made by the SQL database for the HTML version of the fact file A10-8 (@node-id="node-id-A10-8-2015-04-16-0200"), highlighted above, includes the domain (regulatory checklist branch), type and media, like this:

http://localhost:8080/exist/rest/db/lrf/system/common/linkbase/xquery/get-resource.xq?domain=mhs&node-id=node-id-A10-8-2015-04-16-0200&type=fact&media=html

This would simply return the HTML version's URL:

<data>http://localhost:8080/exist/rest/db/lrf/mhs/html/facts/A10-8-Kontroll-av-dra-node-id-A10-8-2015-04-16-0200.htm</data>

This would then be used when publishing the pop-up help text on the website:

Figure 5: An HTML Fact Published As Help Text

The linkbase is also used to find matching resources when editing a resource in oXygen. See section “Finding Matching Resources ”.

Note

It would, of course, be possible to implement the query functions in eXist without gathering the @node-ids and their associated resources inside a linkbase, and instead look for them directly in eXist, but we saw several advantages with a linkbase:

Firstly, a linkbase does not have to list every resource with a @node-id in eXist, only those that we consider to be needed. That selection can easily be changed by rewriting the linkbase generation code.

Second, the linkbase offers a single point of contact for outside systems; it provides in itself a level of indirection. A query function that returns content from it easier than something that deals with the entire database.

Work

The work area replicates the general collection structure found in the publishing chain, split into two regulatory branches:

Figure 6: The Work Area

The information is placed strictly after information type: checklists in checklists, facts in facts, and so on. This again makes it very easy to manipulate and move the data in well-defined steps.

Finding Matching Resources

When authoring content in oXygen Author, the linkbase is used to locate corresponding resource(s) for the one being edited. The function is available as a context-sensitive menu item:

Figure 7: Find Matching

So, when editing, say, the checklist question A10-8, a query to find the corresponding question (implemented as an oXygen action in the LRFBook framework) would locate the A10-8 fact and open the query answers as XInclude links in a wrapper document^[9]:

<?xml version="1.0" encoding="UTF-8"?>
<db:article xmlns:db="http://docbook.org/ns/docbook" version="lrfbook">
   <db:title>Matchande resurser</db:title>
   <db:para>Klicka på XInclude-länkarna nedan för att öppna de matchande resurserna.</db:para>
   <xi:include 
      xmlns:xi="http://www.w3.org/2001/XInclude"
      href="http://31.216.36.169:8080/exist/rest/db/lrf/work/mhs/facts/A10-8-Kontroll-av-dra-node-id-A10-8-2015-04-16-0200.xml"/>
</db:article>

The fact that oXygen will automatically expand XIncludes makes this approach even more useful:

Figure 8: matched-res-a10-8.jpg

If the matching fact needs to be edited, the author only has to click on the XInclude link to open the file.

Users, Selections and Profiling

The user registration information (name, address, telephone, email, etc) is created and maintained in the SQL database, as is any information concerning their respective operations. These include two somewhat generic areas with information on the farm and the work environment, but also two more specific operational areas, livestock and crop. All four are grouped in far more detail, however, and many of them are interrelated across the operational areas. For example, a small farm with only a few hens will have fewer regulatory requirements on the farm and the work environment than a large, 400-cow plant. There are also geographical dependencies, with some farms situated near protected water bodies and groundwater stocks and therefore falling under other regulatory requirements.

The many relationships between the operational areas, the user information and the user checklist answers was deemed to require a relational database, even in the new system.

If left unfiltered, the sheer number of checklist questions (MHS alone contains around 650 of them) has caused many members to shy away from the checklists and understandably so. For this reason, the new system adds profiling capabilities when authoring checklists.

The idea is as simple as it is common. A group or question is profiled using one or more labels that describe when the group or question is applicable. Profile values might include >400-hens or a county name, stating that the question is only applicable if the member has more than 400 hens or operates in the specified county. In the XML, the profiles are whitespace-separated attribute values:

<qandaentry profiles="value1 value2">...</qandaentry>

These questions, of course, become part of the total checklist that is exported to the SQL database and used as the online checklist web form:

Figure 9: Checklist Questions on the Web

The profiles for each question are tracked by the SQL database, but additionally, the SQL database also defines relations between separate profiles. The >400-hens value, for example, might relate to a question regarding the farm's size and facilities, meaning that answering yes to the question about the number of hens being more than 400 might automatically cause other questions to be included and yet others to be excluded. In any case, the member is directed only to relevant questions.

The profile values are created and edited in the SQL database^[10], but converted to XML and exported to system/common/profiles/xml/profiles.xml in eXist when needed. This file is simply a list of profile values:

<profiles xmlns="http://cos.se/ns">
    <section>
        <title>Verksamheter
        </title>
        <profile>Biodling</profile>
        <profile>Djurhållning</profile>
        <profile>Livsmedelsförädling</profile>
        <profile>Odling</profile>
        <profile>Trädgårdsföretag_Potatisodlare</profile>
        <profile>Övrigt</profile>
    </section>
    ...
    <section>
        <title>Odling
        </title>
        <profile>Ekologisk_odling_EU</profile>
        <profile>Ekologisk_uppfödning_KRAV</profile>
    </section>
    ...
    <section>
        <title>Län
        </title>
        <profile>Blekinge_län</profile>
        <profile>Dalarnas_län</profile>
        <profile>Gotlands_län</profile>
        <profile>Gävleborgs_län</profile>
        <profile>Hallands_län</profile>
        <profile>Jämtlands_län</profile>
        <profile>Jönköpings_län</profile>
        <profile>Kalmar_län</profile>
        <profile>Kronobergs_län</profile>
        <profile>Norrbottens_län</profile>
        <profile>Skåne_län</profile>
        <profile>Stockholms_län</profile>
        <profile>Södermanlands_län</profile>
        <profile>Uppsala_län</profile>
        <profile>Värmlands_län</profile>
        <profile>Västerbottens_län</profile>
        <profile>Västernorrlands_län</profile>
        <profile>Västmanlands_län</profile>
        <profile>Västra_Götalands_län</profile>
        <profile>Örebro_län</profile>
        <profile>Östergötlands_län</profile>
    </section>
</profiles>

This file is then read by the Profiles pop-up dialog in oXygen, used by the author to edit the profiles of a group or question:

Figure 10: Profiles Pop-up Dialog in oXygen

Here, the author selects any profile values that apply to the group or question. If it is about beekeeping, for example, the author might probably want to check the beekeeping profile value, as well as any physical locations that are applicable.

The pop-up is implemented using oXygen's proprietary CSS extensions, making adding a simple dialog such as this easy, using only a few lines:

qandaset:before, qandaentry:after {
    content: "Profiler: "
        oxy_editor(
            type, popupSelection,
            edit, "@profiles",
            values, oxy_xpath('string-join(doc("http://profile:profile@31.216.36.169:8080/exist/webdav/db/lrf/system/common/profiles/xml/profiles.xml")//cos:profile/text(), ",")'),
            resultSeparator, " ",
            columns, 10,
            selectionMode, "multiple",
            fontInherit, true);
    font-size:12px;
}

The values property does most of the job fetching the profiles and edit defines the attribute (@profiles).

System Collection Layout

The system collection layout reflects the same philosophy that is used for the contents as described in the previous sections:

Figure 11: The System Layout

Here, the organisation is based on the differences between the regulatory checklist branches, using a common collection for shared resources and dividing the rest into branch-specific parts. Currently, the mhs branch is more complex, with more than double the amount of data than rhs and including PDF publishing in addition to the web.

This might change but if so, this layout makes it easy to implement the change. It is also easy to add a branch, something that is being discussed.

Schemas

The eXist/oXygen system uses a number of schemas, two of which deserve mentioning here.

Versioning and Rollbacks

None of the XML content is currently versioned in any way. While it has always been an option to enable eXist's versioning module and to add similar content management to the SQL database, so far the customer has been unwilling to pay for that particular feature.

This does not mean that there is no way to go back to an earlier stage if something goes wrong. eXist is backed up nightly, and the backups for the last week (and, further back, once every month) are stored in case errors manage to find their way to the published content. In that case, rollbacks will always be made to work rather than the publishing chain and then republished, as this ensures that the checklists will always be associated with the correct facts.

Nevertheless it is usually easier and faster to fix the problem in oXygen and then republish, as the backup is handled by developers rather than the writers; to date, a rollback of one or more documents since the old system went live has happened on a handful of occasions, and once when the server itself failed. Nothing of the kind has yet happened with the new version.