Balisage Paper: Using a testbed to assess XML Database Performance

Introduction

We have previously used the term XML Storage Systems to describe a range of systems which are used to store XML (structured) data [Paxton 2024]. This encapsulates processors which implement one-off XML parsing and transformation, systems which store XML files whole (and perhaps allow processing of them), systems which decompose XML documents and store them in another storage system (perhaps a SQL DB) and so-called Native XML Databases, in which collections of XML documents are decomposed and stored within a system which supports efficient retrieval, reconstruction, analysis and update of the documents.

While an XML Processor like Saxon [Saxon Processor] is focused on one-off processing of documents, Native XML Databases such as FusionDB [FusionDB], Elemental [Elemental], BaseX [BaseX], and eXist-db [eXist-db] are designed to handle data which is repeatedly accessed, and potentially updated. As their name suggests, they are much closer to traditional SQL and NoSQL Databases in the concerns they address; they just manage XML structured data.

Native XML Databases and XML Processors share a common, standardized query language for XML in [XQuery 3.1]. The standardization of XQuery offered an obvious basis on which to measure, compare and contrast the performance of XML Storage Systems. The XMark Project [Schmidt 2002] defined a canonical XML data collection and a set of XQuery queries over that collection. The XMark data collection and queries have been widely used to assess the relative performance of multiple XML Storage Systems, for instance by [Lu 2005].

The term database testbed was introduced by [Difallah 2013] in the context of their system OLTP-Bench, which was designed to flexibly control, manage and extract metrics from the benchmarking of OLTP systems. Observing that the core benchmark code was only a small part of a realistic production-quality benchmark, led to the understanding that the infrastructure of dispatch, monitoring and metric extraction and reporting was a problem that could be solved within a testbed which allowed for easier construction of new benchmarks in future.

In the NoSQL space, the YCSB system [YCSB10] predates OLTP-Bench, and is a database testbed in practice if not in name. More recently, NoSQLBench [NoSQLBench], [DataStax 2020] has embraced the testbed ethos, and has united pluggable adapters for database instances with the concept of Virtual Data Sets, a flexible and powerful means of generating large, pseudo-random but repeatable synthetic data with which to drive benchmarks.

The continued progress in benchmarking of NoSQL and OLTP systems exposes the relative lack of progress in benchmarking XML Databases. Much of the literature on the subject of XML benchmarking is not recent, and a review such as [Mlýnková 2008] remains relevant. We have revisited the XML Benchmarking landscape in this paper [Paxton 2024] in detail, and have proposed a way forward by adopting NoSQLBench as a testbed, adapting it with an XML:DB driver to operate with standardized XML Databases, and enhanced it with a Virtual Data Set(VDS) based library for generating synthetic XML database collections.

In the rest of this paper we restate our goals in developing an XML Benchmarking testbed, and describe the recent progress we have made towards them.

Architecture of the XML Benchmark Testbed

The OLTP-Bench paper [Difallah 2013] defines the goals of benchmark testbeds as:

When running benchmarks from a testbed (or otherwise) we recognise 3 distinct phases to the process: We want to bring the benefits of a testbed as described by [Difallah 2013] into the XML benchmark domain, and to do this we have extended NoSQLBench so that it can work effectively with XML Storage Systems.

Running XMark in the Testbed

Having built an XML:DB adapter and an XML generation tool, we can construct a NoSQLBench script to run XMark benchmark queries. NoSQLBench scripts are written in YAML, and are structured at the top level as a number of scenarios. Our scenarios are:

As an example, here is the scenario for XMark benchmark 3: Return the first and current increases of all open auctions whose current increase is at least twice as high as the initial increase.

                     
xmark003:
# Single threaded
warmup:     run driver=elemental tags==block:"xmark003.*",
                cycles===TEMPLATE(warmup-cycles,5)
                threads=1 errors=timer,warn
                endpoint=xmldb://localhost:3808
read:     run driver=elemental tags==block:"xmark003.*",
                cycles===TEMPLATE(read-cycles,10)
                threads=1 errors=timer,warn
                endpoint=xmldb://localhost:3808

and here is the single XMark benchmark 3 operation,

                     
xmark003:
ops:
  query:
    xquery version "3.1"
    for $b in /site/open_auctions/open_auction
    where $b/bidder[1]/increase/text() * 2 <= $b/bidder[last()]/increase/text()
    return <increase
        first="{$b/bidder[1]/increase/text()}"
        last="{$b/bidder[last()]/increase/text()}" />

NoSQLBench is invoked as a command to run a series of scenarios:

                     
java -jar ./nb5/target/nb5.jar ./adapter-existdb/target/classes/activities/xmldb_xmark.yaml
    reset xmark003 auctionfile="auction-xmlgen2-f10.xml"

This runs the scenarios reset and xmark003 in order, and has set the (script defined) auctionfile parameter as the name of the XML database file to install (part of the reset scenario). Metrics from the run can, by default, be found in NoSQLBench log files. The default result_success metric from the read activity is what we need to tell us how the benchmark has performed:

            {activity="read",container="xmark003",instance="default",jobname="nosqlbench",
            name="result_success",
            node="192.168.68.62",
            scenario="xmark003",session="o2988XR",workload="xmldb_xmark"}
            count = 10
        mean rate = 0.00 calls/nanosecond
    1-minute rate = 0.00 calls/nanosecond
    5-minute rate = 0.00 calls/nanosecond
   15-minute rate = 0.00 calls/nanosecond
              min = 10779885568.00 nanoseconds
              max = 12788957183.00 nanoseconds
             mean = 11526261964.80 nanoseconds
           stddev = 587242416.24 nanoseconds
           median = 11340349439.00 nanoseconds
             75% <= 11827937279.00 nanoseconds
             95% <= 12788957183.00 nanoseconds
             98% <= 12788957183.00 nanoseconds
             99% <= 12788957183.00 nanoseconds
           99.9% <= 12788957183.00 nanoseconds

A Comparison of XMark xmlgen and xmlgen2 Auction Data

In order to demonstrate that our new XML generation library is arguably adequate, along with the XML:DB adapter, as an XML benchmark system we have chosen to appeal to the the XMark benchmark. First we present the results of running a selection of XMark benchmarks using auction sites generated both by the original XMark xmlgen and by our new xmlgen2. Time pressure has prevented us completing running all the benchmarks, but we have tried to select a broad illustrative sample. Then, we discuss the results.

The XQuery for each of these benchmarks is in the test YAML script which is avaliable from: https://github.com/evolvedbinary/nosqlbench/blob/exist-driver/adapter-existdb/src/main/resources/activities/xmldb_xmark.yaml.

Looking at the table, we can see that most benchmarks result in a very slightly slower runtime for xmlgen2 compared to that for XMark xmlgen. We speculate that the consistently slightly larger database, which is due to the VDS mechanism we use to generate text yielding larger paragraphs, is the uniform reason for this. Even so, the differences are never more than 5%-10%.

We consider in more detail the queries that differ most significantly:

Conclusions and Future Work

First of all, we have validated that our new xmlgen2 (VDS) generator can generate an auction site sufficiently similar to XMark xmlgen that the same set of queries execute correctly.

Second, we are able to run performance tests against both versions of the auction database, in a range of sizes, and get repeatable results. Our NoSQLBench driver extension is responsible for this.

Third, our results show that the selection of XMark queries we have tested behave consistently across auction databases generated by XMark xmlgen and xmlgen2 (VDS). While results are not identical, they are close enough to show that our system is generating a sufficiently similar auction database, and we have been able to identify and repair flaws in xmlgen2 on the basis of discrepancies in the performance of the XMark benchmarks.

This all means that moving forward, someone who wishes to benchmark their XML Database can confidently adopt our NoSQLBench adapter and xmlgen2 to easily run XMark benchmarks. If they have maintained XMark benchmarks with the original XMark xmlgen they can carry out a one-off comparison of their benchmark results with both systems, and then confidently adopt xmlgen2 and NoSQLBench as a modern testbed with which to build a continuous performance regression test setup.