Balisage Paper: The Dream of a CMS

XML First

That all changed with XProc 1.0. As long as I had something in the DMS calling an XProc engine and a pipeline of my choice, adding a new publishing format or an import function was a matter of writing the stylesheets and an associated pipeline. It was brilliant!

And it was to get even better — not only was I able to write an XProc pipeline for the processing but I could also generate a matching user interface in XForms using XProc! [Using XML to Implement XML]

Some time later, I helped design another content management and (web-based) publishing system to produce regulatory checklists for farmers seeking national and EU funding, this time using oXygen and eXist-db [XML Solutions for Swedish Farmers: A Case Study]. Importantly, beyond a few web technologies and design choices outside of my control, it was all implemented using XML technologies — XProc, XSL, XLink, XQuery, and a few other things beginning with X — so rather than having to deal with binary blobs in a SQL database, I could query and process the XML directly.

My next step was to test the waters by combining the aforementioned XProc and XForms publishing approach with eXist-db [ProXist]. It was a bit clunky but it worked; I was able to output a UI, published by an XProc pipeline and associated framework, based on the type of output pipeline I had.

I still missed version management and full traceability — what I had in that SQL server-based system — but implemented in an XML database using XML technologies. I came up with VML [Multilevel Versioning], an XML vocabulary for version management, and an outline suggesting how to implement it in eXist-db.

A Question and a Devious Plan

The XML first content management approach stayed with me. I would occasionally suggest solutions along those lines for my clients while keeping busy with migration pipelines, publishing stylesheets, and DITA customisations, but surprisingly no-one wanted to sponsor my XML-first system. I did write another eXist-db and XForms project, a registration app for Balisage's sister conference, Markup UK [Eating Your Own Dog Food], but that was about as close I got.

But then, in 2022, a client asked me how I would go about implementing a portal for publishing DITA and S1000D content. The portal would publish the service documentation of a car manufacturer about to launch their very first model, but also fit into my client's larger strategy of providing an entire service lifecycle management chain, from 3D CAD data to service and end user documentation, parts catalogues, and so on.

I proposed a system where the XML content is stored in an XML database as-is, without pre-converting anything, filtered and queried as-is, and finally published to HTML on the fly. I had a devious plan.

Content

DITA [DITA Specification] and S1000D [S1000D Specifications], while very different on markup level, have similar approaches to content: firstly, there are the topics, that is, the reusable blocks of information where the actual content lives, and secondly, there are the publications or structure descriptions (maps in DITA, publication modules in S1000D) that combine the topics into actual documents with chapter and section hierarchies.^[1]

The portal had to be able to display both — individual topics would be enough for, say, a disassembly task or a function description for a single component, while the structures would be needed to browse through the full publications.

A structure view helps illustrate the overall document structure and functions as a table of contents, but it also helps highlight the reuse of common components — for example, see the arrows pointing to Warning 2 in Figure 1.

Figure 1: Reusing Topics

Both vocabularies employ what is known as conditional processing (some vocabularies, including DITA, call it profiling), basically declaring applicabilities (which is really S1000D terminology) for the content: this topic applies to products A and B, this applies to regions APAC and EU, and so on. By profiling content from individual paragraphs to entire topics, reusing becomes easier; a single topic can be reused in multiple contexts, spanning multiple products, product variants, audiences, and so on.

The filtering is, of course, also useful when searching for content, but also for publishing on the fly; an end user can select the exact model and variant to only include relevant information when publishing to HTML.

Browsing and Filtering

In the portal, we store the source content as-is, and that content needs to drive everything. For example, we only list the topic types and profile values actually in use in the database, not everything that is possible. The UI should always reflect the actual content.

While XML databases use XML as their primary format, we still want to generate basic resource lists only once, when initialising, rather than every time those resources are queried.^[2] This makes most operations far quicker, especially with a large enough database.

The initialise operation generates several lists, all in XML format:

The resource list is especially interesting. It is currently limited to content and structure XML only, so while there are plenty of other files, XML and binary both, they are not used directly by the main UI. For each listed XML, we add a basic information such as a database URI and a title (extracted from the file contents), but also profiling and topic/resource type information. The result is a (very long) list of file elements:

<file xmlns:exist="http://exist.sourceforge.net/NS/exist" uid="d97e1" depth="-1" selected=""
    type="topic" uri="/db/test/content/dita-examples/01/my_first_portal_topic.dita"
    name="my_first_portal_topic.dita" created="2023-04-09T19:29:19.867+02:00"
    last-modified="2023-04-09T19:29:19.867+02:00" id="my_first_portal_topic"
    dita-content-type="content" product="A" audience="C D E"
    root-profiles="product(A) audience(C D E)" outputclass="" excluded="false">
    <title xmlns:ditac="http://cadituk.com/xquery/ditac">My First Portal Topic</title>
</file>
<file xmlns:exist="http://exist.sourceforge.net/NS/exist" uid="d98e1" depth="-1" selected=""
    type="topic" uri="/db/test/content/dita-examples/01/second_portal_topic.dita"
    name="second_portal_topic.dita" created="2023-04-09T19:29:19.9+02:00"
    last-modified="2023-04-09T19:29:19.9+02:00" id="my_second_portal_topic"
    dita-content-type="content" product="A B" audience="D E"
    root-profiles="product(A B) audience(D E)" outputclass="" excluded="false">
    <title xmlns:ditac="http://cadituk.com/xquery/ditac">Second Portal Topic</title>
</file>

The file list is read by an XForm and presented as a file browser with various controls:

Figure 2: File List

For structures (maps) we add nested file elements representing linked resources. Here's a DITA map structure:

<file xmlns:exist="http://exist.sourceforge.net/NS/exist" uid="d89e1" depth="-1" selected=""
    type="map" uri="/db/test/content/dita-examples/02/section_d.ditamap"
    name="section_d.ditamap" created="2023-04-09T19:29:19.791+02:00"
    last-modified="2023-04-09T19:29:19.791+02:00" id="map_bb1_hd1_3vb"
    dita-content-type="structure" outputclass="" excluded="false" expanded="false">
    <title xmlns:ditac="http://cadituk.com/xquery/ditac">Section D</title>
    <file uid="d90e1" depth="0" selected="" type="topic"
        uri="/db/test/content/dita-examples/02/topic_8.dita" name="topic_8.dita"
        created="2023-04-09T19:29:19.519+02:00" last-modified="2023-04-09T19:29:19.519+02:00"
        id="topic_8" dita-content-type="content" outputclass="CLS2" excluded="false">
        <title xmlns:ditac="http://cadituk.com/xquery/ditac">Topic 8</title>
    </file>
</file>

The View button starts a publishing process that converts the selected topic to HTML and opens it in a separate tab. For DITA maps, that button becomes a Browse button and presents the structure as a tree, with expandable subtrees and, of course, viewable topics:

Figure 3: Map View

The UI is generated from the file list XML using an initial XQuery script that calls some XSLT, allowing us to not only localise the UI (all UI labels and text are stored in language- and country-specific XML files) but also configure how the UI is presented, using a configuration XML file to define what features are being used. For example, one setting allow us to select one or more resources for later processing or filtering:

Figure 4: Select/Unselect Buttons

Additional controls allow us to filter the file list by applying profiling information:

Figure 5: Profiling Controls

When applying filters, we simply add the currently selected values to the file list XML:

<profiles>
    <product>
        <value>B</value>
    </product>
    <platform/>
    <audience/>
</profiles>

This is used by the XForm controls, of course, but also for publishing. When a topic is published (see section “Publishing”), the profiles element is converted to a DITAVAL filter [TBA ref] and sent to the publishing process as a parameter.

Most of the file browsing and filtering features are implemented with XForm controls that either act on the file list XML directly or call XQuery functions.

A configuration XML file controls most aspects of the UI, including styling, scripts, and so on, so changing the appearance of the UI is a matter of tweaking the CSS stylesheet(s) and updating the UI configuration:

Figure 6: Alternative UI Layout

Finally, note that the file list XML acts as an XForms instance, so in addition to the resource metadata, we frequently set attribute values for XForms-related processing. For example, in the nested file elements shown above, there is an @expanded attribute to expand and collapse the tree representation. Other processing includes the resource type and the DITA outputclass, both of which are useful when filtering and publishing.

Publishing

The first portal version ended up being a DITA implementation because that's what the customer, an automotive manufacturer, is using. DITA isn't without its advantages; most aspects of DITA, from authoring to publishing, are well supported today, meaning that we wouldn't have to write publishing stylesheets from scratch. Rather than using the DITA Open Toolkit, a first choice for many implementers, we chose XMLMind's DITAC framework [XMLmind DITA Converter] because it was far better suited to being integrated with eXist-db.

Standard DITA functionality, from profiling using DITAVAL filters to conref links, is provided by DITAC out-of-the-box, which is a huge help: DITAC includes default stylesheets for HTML, PDF, and a few other formats. This means that a DITAC stylesheet to output a specific layout, known as a plugin, is really just an extension and therefore much faster to write.

The portal DITAC implementation brings the functionality to eXist-db and deals mostly with database URIs and packaging, but also JavaScript code to handle 3D^[3]:

Figure 7: 3D in an Illustrated Parts Catalogue

Most aspects of the portal are XML technologies that the author feels quite comfortable with; this is not one of them.

Authoring

Adding authoring is easy, and easiest by far is to connect oXygen XML Editor to eXist-db^[8], with the connector available out-of-the-box. Other editors require more work. If all you want is an integration to the (XML) database without versioning, then we're done.

Actually, even with versioning, I'd argue we're only ever going to edit the latest version of any document. If you check out an earlier one, you still don't actually edit that version, you create a fork and edit that instead.^[9]

In a proper version management with check-out and check-in, your editor integration will require additions to do that.^[10] I'd argue that the check-out function is a matter of locking the file and reasonably easy to achieve. There are probably also a number of convenience functions, but for a bare-bones authoring environment, this should be enough.

A reviewer rightly pointed out that the editor is frequently a stumbling block for non-markup people. While I do agree, addressing that particular problem must come later. oXygen, for example, makes an excellent effort towards user-friendliness for many types of authors^[11], with or without an associated CMS, and so I would argue that a well-designed CMS (which this one aims to be) can only help.

Document Management

And we arrive at the heart of the matter. You'll note that we already have browsing and filtering capabilities; I'd argue that the portal's browsing UI isn't needed for basic editing when using oXygen (see section “Authoring”) since we'll always edit the latest version and the whole versioning business is handled in the CMS.

Other editors might have to either implement proper in-editor integration — which to me sounds a fairly difficult thing to do — or add a UI trigger to open a selected resource in the XForm, much like the buttons shown in Figure 4. That function, of course, would be a little something written in XQuery, perhaps made slightly more complex with check-out/check-in functionality.

The check-out/check-in functionality requires a suitable flag, of course, but it's is easy enough to add one to the file list XML as an attribute (with the flag controlled by XQuery functions behind the scenes):

<file checkout="true">...</file>

Other flags (and associated XQuery functions) are easily implemented, of course.

urn:x-example:thing:123456:<version>:<xmllang>

urn:x-example:thing:123456

<file urn="urn:x-example:thing:123456">...</file>

<file urn="urn:x-example:thing:123456" version="15">...</file>

<file urn="urn:x-example:thing:123456" version="15">...</file>
<file urn="urn:x-example:thing:123456" version="14">...</file>
<file urn="urn:x-example:thing:123456" version="13">...</file>
...

<file urn="urn:x-example:thing:123456" version="15">...</file>
<file urn="urn:x-example:thing:123456" version="14">...</file>
<file urn="urn:x-example:thing:123456" version="14" xml:lang="sv-SE">...</file>
<file urn="urn:x-example:thing:123456" version="14" xml:lang="fi-FI">...</file>
<file urn="urn:x-example:thing:123456" version="13">...</file>
...

But What About...?

While I've thought about the subject for years^[14], this paper simply cannot reflect everything. Also, I could be missing something obvious. The latter I can't do much about, but here's to addressing at least some of the outstanding questions:

The First Live Version

The portal went live before Christmas 2022 in what the author does regard as an unqualified success. The database contains 100+ GB of data, much of it 3D content but also thousands of XML resources, and the performance is surprisingly good. The XForms UI, in particular, works amazingly well.