Balisage Paper: Framing the Problem

XML-based systems: better than ever but just the same

XML-based data and document-processing solutions have been around long enough now that we can begin to see their outlines and commonalities, despite their vast differences, across domains, between large and small, and down through generations. In some respects, a server farm talking to an XML database back end and producing thousands of arbitrary “views” per minute of dynamically populated content for a range of clients (desktop and mobile) on the Internet, is still just like a 486 processor running Netscape Navigator with a browser plugin on Microsoft Windows 95, with 8MB of RAM. The astonishing differences between then and now can be enumerated using all kinds of metrics. Things now are so much larger, faster, more complex. Perhaps not more cumbersome (the bad old days are hard to forget), but not less so. The similarities may be harder to see; yet it helps to understand the later, more complex form when one considers there has been a straight line of development from the earlier and simpler.^[1] We are not doing more than we were doing back then. Much useful work was done by systems built on these principles, even years ago. We are only doing it at ever greater levels of scale, speed and complexity.

Figure 1

A screenshot of a contemporary online editing interface: this is structured TEI, being edited live in oXygen XML Web author, the presentation configured using stylesheets written by George Bina and others. (What is in view is not styled with XSLT but with CSS+extensions; the application framework can also run XSLT or any other external process.)

This is worth keeping in mind when turning our attention to present opportunities, as experience reinforces our sense of how much we can now do, for example with an ad-hoc (yet strategically chosen) collection of schema(s), XSLT, CSS, Schematron, and glue, whether that be some vendor-specific code, or something patched together, or something very powerful, general and extensible, such as an application coded in XProc (a generalized declarative pipelining language) or a database running XQuery. Even more, we have entire editing environments custom built for purpose, up to and including web-based environments, which purport to make it possible for people who know nothing specifically about text encoding technologies, to use the systems nonetheless, as “experts” (presumably in their own domains ... but then by using it, they also become expert in the system itself). Such frameworks, suites and toolkits are nothing new; but costs for designing, building and supporting these environments and working interfaces have dropped by an order of magnitude every decade since the 1970s – and may (for all we know) continue to do so. This by itself has changed the equations.

Figure 2: XML editors, oldish and newish

Screenshot of SoftQuad XMetaL (perhaps then BlastRadius), taken in 2002, followed by a view of the same text open in oXygen XML Editor, 2016. Of course, this is a stunt, but it is remarkable how easy it was (twenty minutes of CSS) to emulate the formatting being demonstrated by the old editor, in the new one. (A few more minutes would allow for considerable improvements to the design.)

Indeed, in this last observation is a point that is easy to allow to slip by. Up until fairly recently, there were two very distinct sorts of “framework development”, as one might call it, related to XML and XML-based publishing and data processing systems. One was the creation of customized structured editing environments – systems that attracted significant investments over what is now decades (publishing systems stretching back as far as computers do), and which by the late 1990s had evolved into an entire market for software and services. Typically these applications, albeit nominally standards-conformant (generally some flavor of SGML was the starting point), were implemented on proprietary stacks of one form or another, and set out to support processes end-to-end, typically by combining user interfaces (for creation and management) with production capabilities, usually by way of integration with a publishing application.^[2] To build such an application meant cultivating expertise not only in applicable standards (where they existed) but also the software application itself and its formats and interfaces (which were typically proprietary, if only to fill the gap where standards did not exist – because they were being tried for the first time). Notwithstanding the challenges, this was something that was done successfully, and more than once.

Yet while – at least in some vertical markets (where factors of scale and automatability are especially favorable) – applications have been built and sustained on these principles, there has also developed a healthy counter market (one might call it) in free and low-cost tools for document and data processing. The web itself is of course the main difference between now and the 1990s – two decades proving to be long enough for several revolutions in publishing. Indeed one of the most profound revolutions has been in the (relative) universalization of access to information on how to operate and build on the web. In other words, the web itself has been the means for its own dissemination of expertise, and other forms of data processing (off as well as on the web) have benefited from being "transmissable" on the web (as a platform for knowledge transfer). So for the mere cost of a data connection, one has access to libraries and indeed executables that have required years (albeit often by do-it-yourselfers like yourself) to develop, encapsulate and publish. We work now in a kind of ecosystem of software development activity, which changes everything about how we work and how are work can combine with and magnify the work of others. And this is true not only for technologies of web development as such, but of much that goes with it.

Consequently there have appeared alternatives to the high-end editing environments for supporting XML-based processing workflows, approaches that assume a much more do-it-yourself attitude. From the early days, it became possible to build and operate systems by stitching together whatever markup and text processing utilities you could get to work, perhaps Perl or Python, or perhaps DSSSL (when we had that) or next XSLT. These components would then be bolted together using whatever pipelining architecture was available to you. (Maybe it was shell scripting over the file system or through editor buffers.) Typically this meant writing batch or shell scripts that managed files according to custom-made logic over the file system, invoking open-source or otherwise-accessible executables in order to get the actual job done. But sometimes you would have other interfaces to query and even edit your data.

Solutions patched together in this way tended not to be end-to-end document processing or publishing systems, but instead focussed on problems that need to be solved, or that they could usefully or neatly address. While there might have been some front-end component to these systems, that generally wasn't their strength; instead they tended to provide services on the back end or downstream of the editing process. Their operators, in contrast to the first kind of (customized) editing environment, tended to be technologists, at least to the point that they could troubleshoot when problems arose. When orchestration of processes became complex (as they soon enough did), simple interfaces would be contrived using build languages (or other methods). While such systems require considerable expertise in relevant standards and technologies (including libraries and APIs) to build and sometimes to operate, unlike the top-down editors, they have tended to be unencumbered by proprietary barriers to development and reuse, while arguably, in comparison to proprietary alternatives, providing even more open-ended functionality. It turns out that all by themselves, XSLT, Apache Ant, and file utilities can take you a very long way.

Indeed it should be kept in mind that at some point it becomes difficult to draw a boundary between a “customized editing environment” and the kinds of support for processing in general that such an editing environment routinely offers as well. In other words, what makes these systems distinguishable at all is not that they offer any particular sort of user interface – there may be no GUI in sight – but rather that they bundle front and back end services and processes in a way that makes them accessible and controllable by users. Indeed, users should be at the heart of it, and the focus needs to be on providing users, within available means, with something both neat and effective for them, something that produces more value in its outputs than it demands in inputs (of time, resources, attention). The term “workflow” can help us keep things in mind as well: it's not just about where data comes from and how it starts, but what happens to it.

So on the open source side as well, two decades have seen real development. Thirty years after the standardization of SGML and nearly twenty after XML's formal release, developers have a range of choices that (back then) only the most audacious were able to envision. These include not only tooling in support of XML and related standards in all the major languages and platforms, and not only proprietary and semi-proprietary stacks, but open source solutions and specifications as well. In particular (in the context of this discussion), XProc should be mentioned, inasmuch as we now have a pipelining language even apart from our transformation language, something that is indeed useful. As tools and technologies have matured (built, tested, deployed, shaken down, shaken out) and public resources such as the standards themselves (or: public specifications) have stabilized, open source and free-for-use tools have continued to offer a low-cost-of-entry alternative to the big, top-down systems. In turn, this has proven to be an opportunity for companies who offer software and services based on the availability of such tools in addition to its primary beneficaries, the projects and organizations that can use them to solve problems.

Consequently, one might consider XProc running in XML Calabash inside oXygen XML Editor (one product which has played an important role in this evolution^[3]) to be a culmination of both strands of development – structured editors that offer all the functionality of environments that used to cost ten (or a hundred) times as much to build and deploy, yet emerging out of (and taking advantage of) the community-standards-based tagging initiatives and the tools that go with them (these include DITA, TEI, NISO JATS etc.) rather than only from top-down imperatives.

Ironically, part of the value offering of a tool such as oXygen XML Editor is the extent to which – precisely by incorporating directly the “best in category” open source toolkits – it actually insulates its developers from proprietary entanglements, allowing us to work independently to provide whatever local user base we have, with something they need, without having to go further upstream to get the support we need in doing so. Not incidentally, this also lowers the cost of oXygen, since its own "proprietary footprint" over the functionality set, is smaller; due to this lower cost, and especially given its component-based assembly, it also becomes thinkable to use a tool like oXygen not just "end to end" but also for much smaller and discrete sorts of operations. In short: at its lower cost it can serve much more flexibly in the “big mix” (of processes and formats) that characterizes information management today.

What else has changed? The market environment. No longer is this (as it seemed to be 20 years ago) a seemingly Quixotic struggle against the blind and casual use of proprietary word processors and desktop publishing applications to produce, manage and archive information. The alternative today to XML and structured data seems to be, not word-processing-and-unstructured-data, but HTML-and-the-web. Personally I am of the belief that appearances are deceptive, and this is really only the old foe in a new form, albeit not really a foe at all! – and yet paradoxically that an XML mindset – and even an XML toolkit – can still flourish in this environment. In other words, the problem is the same as it was, and XML/XSLT still offers the solution (or at least, a means to provide a solution), and so the key is to continue to use them even while adapting. Even today, and even and indeed especially on the web, XML/XSLT can be a kind of secret super-power.

The problem is knowing when this is. That is, recognizing opportunities: where can applications of transformations, and processing pipelines built out of them, achieve meaningful goals, whether for a project, organization or business? Where can we apply the power of transformations? becomes the key question. And – if they do not always save energy – then how might we use XML-based technologies to produce things (useful or beautiful) that would be impossible or unthinkable without them?

Transformations: Lower and higher-order organization

What hasn't changed is the basic three-cornered transformation architecture: source / transformation / result^[4], a triad that immediately refracts and multiplies once we engage “meta-processes” or control processes, by exerting controls over our components (first, our nominal “source” but soon enough over the transformation itself or components of it, and finally over results as well) using the same methods – transformation – as we use for core processes. Only the simplest of systems reduces to a single source/specification/view (not even an HTML page with a CSS stylesheet in a browser these days typically constitutes a single source with a single display specification); what is much more common is some kind of series or cascade of processes and applications, many of which have, if not active dependencies, then at least implicit dependencies on further processes “up stream” (i.e. at presumed earlier points in a document's lifecycle or processing history). Indeed the real power of these systems is when you become able to manage their complexity by putting different transformations of various kinds and purposes together. So XML's power comes not from the simple or single fact of descriptive encoding (the separation of concerns that comes with any model/view/controller especially one as cleanly separated as this), but also from the fact that XML data can be handled – frequently but not always by design – in aggregate and that XML is itself composable. We can not only display documents, but also process, edit and amend them in any way imaginable, in groups and in groups of groups.

So it becomes possible to apply logical operations to data sets in sequences of steps, between (the details vary) authoring or acquisition, assembly, aggregation, normalization and integration, production and publication. Indeed it should not be hard to see in these steps, analogues and “recapitulations” of activities that publishers already perform and indeed that is their purpose, to perform them – if electronic means and media offer a chance to do anything truly “new” (which is a separate discussion), that is generally as a complement and offshoot of doing something that has already been shown to need doing. Nonetheless it has been demonstrated repeatedly that when it comes to things like formatting and reformatting documents with consistency; marshalling and managing metadata and adjunct resources (such as images or sample data sets belonging to published items or articles); normalizing, regularizing and integrating cross-references and citation data; the creation of indexes and ancillary resources with their descriptions; and numerous other of the typical “chores” that come with publishing, XML-based systems can provide means to lighten the load consistently. The theory and practice of all this is fairly well understood.

Importantly, while these “meta-transformations” include validation,^[5] they may also include other operations that might (within a purely “functional” universe) be considered as side effects or as externally dependent (that is, as reflecting arbitrary state external to the system), such as utilities for reading from file systems, compressing and decompressing sets of resources, copying files in a file system or via FTP over the Internet, and other sorts of tasks. Join these capabilities to the intrinsic capability of XML (which is typically used, when working inside such an architecture, to represent file structures etc.) for querying using the same tools (XPath, XSLT, XQuery) as you use for document handling, and the system has yet more leverage (as has not gone unremarked at Balisage, for example see Rennau 2016). A utility or set of utilities can produce formatted output from controlled source data (that much we accept) but it can also do much more than that, providing services in exposing and editing that data; providing for customized quality checking; providing for help with mundane tasks associated with file management (to a local set of requirements), and so on.

When XSLT is the technology of choice, it becomes natural to describe or characterize these operations as “transformations”, although that term also then comes (by analogy) to indicate any “before/after” relation of applying a logical process, algorithm or filter to a source document, thereby achieving (and perhaps persisting, as an artifact) a result, whether by XSLT or other means. Synonyms for “transformation” in this sense might include “view”, “rendition” or “filter”.

Of course nothing is consumed nor even “transformed” in a transformation. What happens instead is that a new artifact is created next to the old one, sometimes a copy or simulacrum, presumably always some sort of (intended) representation or reading of it. It is as if the butterfly were sitting next to its pupa, which is still alive and able to transform again. Transformation is, really, is not about changing anything but more like “bottling” it, in a different form or package, or as an extract or permuted version of itself in some way. Indeed it is the same as a query, albeit with a different goal (an entirely altered view of a whole, not just an extracted bit) and we can certainly describe our triad as a model-controller-view (source-query-result) if we like – the differences being rather in emphasis.

Considered as abstract operations, transformations are also composable. Chain two or more transformations together as a pipeline, setting aside any intermediate results between them, and you have what can be regarded as a single transformation. (You also need another layer, i.e. a way to represent this multiples-in-one relation. You could do this with a shell, or an XProc, or a wrapper XSLT, etc. Meta-process!) Considered as a black box, any system-level component can be regarded as a transformation – with certain specified inputs and certain expected results therefrom – even if internally, in its implementation, it is quite complex. It doesn't help that as XSLT developers know, transformations are sometimes (frequently) pipelines themselves internally (in whole or in part), as indeed XSLT can be seen as a language for pipelining function calls. (And the same will be true of other transformation technologies in their ways.) So the actual line between “transformation” and “pipeline” is very blurry.

Nonetheless it is convenient to think of transformations as being the atomic units (either because elemental and rudimentary, or because designed to be logical/recomposable, or simply because they are opaque, “black boxes”); “pipelines” are more generally transformations when considered as aggregates not as processes. Additionally, we may wish to stipulate that the model does not insist that a transformation be an automated or automatable process. Manual intervention may be the best or only way to produce it. (What makes a data manipulation or management task a transformation is not whether it is automatable or how difficult it is, but whether it can be described adequately to be performed and assessed.)

Then too, pipelines (considered as transformations) can also be arranged into higher level assemblages, suites of utilities and operations and declaration sets (including declarations of constraints) that work together to achieve specific ends in information processing (both for “publishing” and any others you might think of). These are indeed more difficult to talk about, since while the XSLT at the center hasn't changed all that much (allowances between made for the vast differences in XPath 1.0 and 2.0 data models), the languages in which these assemblages were composed and made available to users, have changed a great deal over time. So a set of tasks that might have been accomplished, in a very rudimentary yet not inelegant way in, say, 1998, with a compiler-build utility such as GNU make – creating a static HTML web site, for example, or generating an index -- these same operations (as noted), we might today be doing using an XProc pipeline and processor invoked from inside an editing application.

Despite not exactly having a name – we might call them frameworks or assemblies or toolkits or any number of things, while remaining somewhat confused as to the boundaries exactly – these things can be distinguished from other productions. Indeed every year at Balisage, a considerable number of them are demonstrated – for example in 2016 we have Caton and Vieira 2016, Clark and Connel 2016, Cuellar and Aiken 2016, Galtman 2016, Graham 2016, Horn, Denzer and Hambuch 2016, Kraetke and Imsieke 2016, Lubell 2016 Sperberg-McQueen 2016. These are “applications” in only a stricter sense than the commonest sense today (“applications” as commoditized piece of software); as meaningful assemblies and configurations (“meaningful” in their contexts of use) of available and handmade parts, they are not software or standards, schemas or libraries, even though all these are greatly mixed up in and with all of these things and typically make use of one or several of them.

Whatever the best term to designate them, first we have to see them for what they are. Typically they are built or deployed on top of some third-party software or architecture, presenting them with wrappers, customizations and extensions that suit them for particular tasks in the project of interest. And once again there is this fractal aspect whereby these creatures resemble, in form, the things out of which they are composed: frameworks (as considered as more or less tightly integrated sets of processes, available to an operator who accomplishes some task) are much like pipelines in their basic outlines, which are much like transformations. They have multiple inputs and outputs, including stylesheets and configurations as well as (so-called) source data;^[6] they can be regarded as black boxes and sometimes (but not always) executed with the push of a button or click of an icon. They typically entail achieving and applying, more or less explicitly, a logical mapping from source format(s) to target(s), in service of some more or less complex information processing objective, such as populating or querying a data store, acquiring or editing documentary information, publishing in multiple formats, integrating with external resources. Each one is unique although they are frequently offered as models to be emulated. Sometimes libraries, applications or languages spin off from them.

However fuzzy the terminology, these system-level assemblages have been maturing; functionalities are tried, tested and demonstrated, and platforms, software and methods all improve. No longer are they always such duct-tape and baling-wire assemblages as they were in the old days. Indeed, even the most peculiar customized local processes over XML can now, often, be sufficiently integrated into interfaces (whether editing and content and workflow management environments) that operators can use them without having to see angle brackets at all, or more generally, without having particularly to care for or about encoding issues in the least.

That someone should able to edit structured text “friction free” (whatever this means for formats behind the scenes) is something of a Holy Grail. My own feeling is that whether and how intricacies of tagging are best hidden away, very much depends on the circumstance. However, one implication of what follows (as I hope the reader agrees) is that the prospect of “tagless” XML (or HTML) editing is ultimately less promising than the way framework architectures can reduce the complexity and tedium, not of encoding as such, but of all else that goes with it. If tagging were the only thing a “content expert” had to think about, in other words, I think she would swiftly learn it. How to tag things is today one of the lesser problems in getting stuff up and out. The more daunting challenges are the ones at the level of deployment architectures and environments. What are you asking users to learn?

Then too, smart framework development can be one way to help bridge the expertise gap. An actual, functioning XML document processing environment can be a learning platform as well as a means of production.

A transformation cost model

One intriguing thing about this progression is the fractal nature of the relation (blurry boundaries between categories, self-similarity across scale) from one level to the next. Another is that structurally, they are isomorphic: any component can (and at some level must) be defined in terms of its putative input(s) and output(s) however complex each of these may be (since there might be aggregation as well as filtering at any point). Thus there is always this tripartite pattern, source/transformation/result, where the middle term is regarded as a black box capable of achieving the mapping from source(s) to result(s), however specified.

In turn this suggests we can apply a single analytic framework to all levels of scale and complexity, whether looking from “far away” or “up close”. Such an analytic framework is offered here. It seeks to interpret any problem of transformation or of document processing, equivalently, as a problem of managing costs of investment (in time but also in expertise)^[7] in view of risks and opportunities of return over time. Note that it is only “pseudo” quantitative, in that its usefulness should be less in whether and how it can be applied to result in some sort of number, but because it may elicit a better understanding between relative factors when considering what makes transformations valuable, when they are valuable. That is, the intent here is to offer not only a way of assessing when and where there may be opportunities, but also some (perhaps counter-intuitive) observations about when these systems work well and why they are sustainable.

The formula is based on the stipulation that any document, instance or occasion of transformation must be considered as a member of a set. This is reasonable to do and indeed inevitable given how we typically and ordinarily develop transformations as rules of implication between types and classes. (We can consider one-offs – transformations that are run once on a single instance of a source document, then thrown away – as singleton sets, so there is no real problem here.)

Viewed as a member of its set, any given transformation will incur the cost of development of that transformation, plus setup for running it (at all, that is the first time, but also in general, for all runs), plus the operations costs (after setup, including marginal costs per item), all divided by the count of successful instances (runs).

Or equivalently (splitting operations cost into an average):

Or (abbreviated)

These all say the same thing. Extracting operations costs for the set and averaging it across members of the set is helpful for weighing the significance of this addend directly against the “development” (more properly D+S) side. This assymetry of factors is the most important thing about this model. The direct implication is that as long as operations overhead is relatively high and in particular if it increases relative to count (most especially, increases linearly – i.e. there are significant marginal costs per item), it will continue to burden transformation costs, even while costs of Development and Setup are amortized (distributed) across the entire set. If instead, operations costs can be brought to zero - then it is possible to achieve very low costs for the transformation as a whole, as Development+Setup costs are distributed, even when the count of documents is relatively low.

“Setup” here includes setting up and running the transformation, that is the costs of running the transformation over and above the costs of handling items in the set (the overhead). With good tooling and/or expertise available, this can drop very low. However (when not) it can also be a prohibitive factor, a barrier to entry.

“Development” is the cost of development of the transformation to the point where it gives acceptable (and/or presumably correct) results. Note that in order for a transformation to run at all, Development costs must typically be incurred early. Yet this is not always the case – while maintenance, similarly, might be considered a type of deferred development. It should also be kept in mind that Development costs are relative to available skills. Not all developers provide the same D “energy” for the same time invested and sometimes the difference will be many multiples (where a skilled developer writes robust functioning code more simply and rapidly than an inexperienced developer facing a problem for the first time). Indeed the relative availability of expertise making D affordable is currently the most important gating factor.

It is useful to consider Development and Setup as independent. An organization may be able to pay for Development (or maybe it can be discounted by acquiring a "stock" solution), but if setup costs are prohibitive, this doesn't matter. In a contrasting case, maybe Setup is low (in the event) but since Development is high, a transformation is not viable. (Perhaps an unskilled operator has access to tools he does not know how to exploit, or perhaps a transformation is underspecified and thus impossible to “get right” even with the best code.)

The average operations costs is where the real problems are hidden. Only when operations costs go down (and especially if they can go down appreciably as count goes up), do transformations have a chance of paying for themselves quickly. Operations costs will include, primarily, all the costs of managing and transmitting files and data resources, including any copying, “grooming”, renaming, editing, marshalling of resources, cross-checking and confirmation. Assume that any time a resource is touched, some minor cost can be associated. This attention may be expended at the level of each individual document, or of the set as a whole. It will be highly variable across operational contexts - sometimes the additional marginal cost per document is extremely low (maybe it can be shared among a large documents at once); at other times, it is not.

A further complication hidden in this value (operations costs) is that in circumstances where not every transformation is successful, the successful ones may have to pay for the unsuccessful ones as well as for themselves. Costs of detection of problems and of their remediation have to be included in operations. And this may also entail new costs associated with overhead, for example if further tasks in file management, project management or task tracking are required when errors or lapses occur. In other words, operations costs will explode in situations where actual throughput is low (count goes down) because transformations are fallible – this is the dark side and the worst problem of automation technologies gone awry: bad transformations (which require corrections or workarounds) may eat up all the benefits of good ones. The viability of any transformation over the scope of actual inputs, and the assurance of a high rate of success (most especially, before all inputs are seen), remain critical determinants in whether a pipeline is actually worth the cost of building and running it.

Several things are suggested by this model:

Finally, it is interesting that this accounting also makes it possible to begin thinking about one of the more vexing distinctions to developers of transformation pipelines, namely the difference between transformations that go “up hill” and those that go “down”. The metaphor invokes gravity: the idea is, it's harder to go up hill. Since our model already seeks to represent “cost” in terms of energy inputs (D, S, operations), it is possible to consider what higher cost (up) vs lower cost (down) should mean in closer detail.

Further implications of the model

In addition to suggesting such a unified view of transformations and the contexts in which they operate, this model also provokes other considerations. In particular, while it can readily be observed that the out-of-pocket costs and commitments in building an XML-based toolkit (whether standards- and community-based or entirely homemade) are much lower than ever before, nonetheless the practical challenges may still be somewhat daunting.

The abstract to this paper suggests some of the practical questions one must ask. Notably, those include specific questions of architecture and environment, which the general model proposed here, deliberately avoids – and they are questions that have always faced developers of systems of this sort, and will presumably always do so. Nonetheless the cost model does offer some perspective, suggesting a few interesting points:

Moreover, we can now see what effects it is likely to have if the costs of S+D have been dropping for everyone due to market externalities. Largely because experts in XML transformation (or for that matter, of document processing technologies) are fairly rare, D in particular is still not exactly low. Yet nevertheless given available tools, for someone who does have facility (or who can learn!), costs of D can be very low over time. In principle, this should bring the technologies within the capabilities of a great many organizations who were formerly, simply, unable to enter this market – if they can cultivate the talent.