Balisage Paper: Fat Markup: Trimming the Fat Markup Myth one calorie at a time

Size Matters

The total size of a marked up document matters for some purposes. The larger the size of the document, generally the more memory it consumes when parsed, the more space it takes in storage and the more time it takes to transfer over a network. However, a single measurement of size is misleading.

Size can be measured in bytes, characters, nodes and other metrics - both on storage and in memory. Encoding and compression can affect the byte size given the same character size in storage (and often memory). Choices of particular markup style representing the same document data can affect the number of characters; for example choosing shorter element names can produce a document with less characters. Markup choices can also affect node structure and size; for example using attributes in XML instead of elements for some data produces a different representation of the node tree itself which in turn affects the number of characters and bytes. There are many other considerations such as numeric precision, ignorable and boundary whitespace, defaulted attributes, use of namespaces etc. While some specifics are particular to the markup language the concept is valid in both XML and JSON formats.

Thus given the same document as an abstract object, it is meaningless to attempt to provide a single measurement of its size as expressed in a particular markup format. However, one can measure specific sizes of a particular representation of a document.

But even given a specific metric for the size of a document, does it matter? That would depend on what one cares about. Size is at best an indirect measurement of some other quantity one is interested in such as time to transmit over a network or disk space used. General intuition is that probably smaller is "better". However, to get a more definitive answer then that we need to ask "better for what and whom?" A binary compressed document may be smallest and better at conserving disk space but not better at readability, usability or parsing speed.

Shape Matters

The "Shape" or particular structure of a document matters for some purposes. For example, readability and ease of authoring by humans or machines may be important. The "shape" also affects the design of parsers, data models and programming API's.

A particular shape may be better for some uses then others - an exact mapping to a particular programming language object structure may be useful in that language but cumbersome in another. Even in a single language some models may be better suited for direct access and others for searching.

Shape of a document is hard to quantify, but I suggest that one can use "ease of use" as a proxy. A shape that doesn’t fit your use case is hard to use and could be considered "Fat" in terms of the difficulty of using it.

Speed Matters

When something is considered "Fat" it's generally implied that its "Slow". Speed matters. The time it takes to transfer a document from disk to memory, the time it takes to transfer across a network, the time it takes to parse and query a document - all matter. There is also developer time. How long does it take a developer to learn a language and write code to process a document?

Sometimes speed in the small doesn’t mean speed in the large. The story of the Tortoise and the Hare can provide good reflection on this. As a practical example, imagine a document which is very fast to load into a programming language but the data format produced doesn’t lend itself for some use cases so extra processing, internal transformations to other models or use of complex and slow libraries may be required to access the document. For example it is common practice in JSON to embed HTML snippets as strings. If you want to search, combine or reformat them you may then have to construct an HTML document and parse it with an HTML parser to produce an object model which useful. The leanness of the original markup is lost in the overhead of having to perform multiple layers of parsing.

Looks Matter

Fat is generally considered "Ugly". A "Fat Markup" often implies an ugly one. If a document format is so ugly you can't stand looking it, you will try to avoid it.

Fortunately "Beauty is in the eyes of the Beholder". This is true in markup as well as the humanities. As time, exposure, and fashion change so can the subjective beauty of a markup format.

What's it all mean?

Sweeping statements of technologies like "Fat", "Slow", "Bloated" have imprecise meaning. Worse they are often chosen to pull in emotional associations which may not apply to the specific cases. So even if it were considered true, what does it mean by saying "XML Is Fat"? I suggest it means very little, actually of negative usefulness. A markup itself is not used in isolation; it is used in the context of complex work-flows including editing, distribution and processing. So what should one do to characterize a Markup Language in a meaningful and useful way?

I suggest a path lies in defining tests that measure specific use cases and provide reproducible metrics. These metrics can be used to get a better understanding of the performance of markup in a meaningful way.

In short, An Experiment.

Multiple Document Corpuses

Most of the past publications I have found focus on a single corpus of documents and usually attempt to derive a single metric. These are often very artificial, focused on a single domain or structurally similar. For this experiment I take 6 very different documents and a baseline to attempt to cover a range of document uses 'in the wild'.

Subsets of larger corpus and duplication of smaller datasets were chosen so the resultant XML and JSON sizes ranged from 100 KB to 1 MB. These limits were chosen so the size was large enough to take noticeable time to load, parse and query but not so large as to break most reasonable browsers even on mobile devices.

The Document corpus used is as follows

Document Size and shape

Most research I have found makes the incorrect assumption that there is a single representation of a document in a particular markup. I take a different approach. Starting with seven (7) different documents I produce 2 variants in JSON and 3 variants in XML that all represent the same abstract document. These documents span several data styles including simple tabular, highly structured and document format. Some of the corpus is synthetic and some taken from real world data. In addition I test the support for HTTP gzip compression on these documents.

Methodology and Architecture

The experiment contains a Server, Browser, and Analysis components. The components were created using standard open source software and as much as possible designed to focus on the attributes being tested and eliminating introduction of bias from components which are not related to the test goals.

Client

The Client is a Browser based JavaScript application. The GUI components of the client were developed using GWT GWT. The code which performs the measured parts of the tests are hand written JavaScript, with the exception that jQueryjQuery is used for part of the test as indicated.

Figure 6: Client Application

On start-up the client requests an index document and shows a list of all files in the corpus, their types and sizes. The user can choose to run tests individually or as a whole.

There is one button which when tapped runs all tests then submits the results to the server via an HTTP POST. The time it takes to load, initialize and update the GUI, and to send the results is not included in the results.

Publicly available data available to the client program (browser) is also included in the results. This data is used to identify the browser and OS of the client.

Tests

When the "Run All Tests" button is tapped the following procedure is run. For each of the files in the index the following is performed and individually timed.

• The data file is fetched from the server in raw (not compressed) format and stored as a string. Where possible the actual Content-Length is used to update the data size metric for this file.

• The data file is fetched from the server in compressed (gzip) format and uncompressed and stored as a string. Where possible the actual Content-Length is used to update the compressed size metric for this file.

• The data is parsed into an in memory object. For JSON files this uses the JavaScript eval() method. For XML files standard browser methods are used to parse the XML file.

• The object is "Queried". To simulate a consistent query across the variety of the corpus all nodes of the document are recursively descended and counted. The number of nodes visited is recorded along with the elapsed time.

• The data is again parsed into an in memory object. For both JSON and XML files, jQuery is used to parse the document and produce a jQuery object.

• The object is "Queried". To simulate a consistent query across the variety of the corpus all nodes of the jQuery document are recursively descended and evaluated. The number of nodes visited is recorded along with the elapsed time.

Test Coverage

In addition to a variety of data sources and formats, the experiment attempts to cover a range of devices, operating systems, browsers and networks. This is achieved by "crowd sourcing". The URL to the test was made public and distributed to a range of mailing lists and social media sites including Amazon Mechanical TurkMechanical Turk. Ideally this experiment can continue for a long duration so that trends over time can be measured.

It is expected that performance varies considerably across devices, especially mobile devices, and prior research has largely ignored differences across devices.

Only the browser "User Agent" string is used to distinguish browsers, devices and operating systems. This makes some measurements impossible such as distinguishing between broadband, wifi, 3G, LTE and other networks.

However, even lacking some measurements and precision, seeing the range of performance across platforms is still educational and useful.

At the time of this paper approximately 1200 distinct successful tests results were collected.

Browser Coverage

Table II

name	count
Chrome	423
Firefox	277
Mobile Safari	148
IE	75
Android Webkit	75
Safari	72
Chrome Mobile	57
Opera	15
Chromium	7
Mobile Firefox	4
IceWeasel	4
IE Mobile	3
Opera Mobile	3
Avant Browser	2
SeaMonkey	2
Epiphany	1
Netscape Navigator	1
Rekonq	1
Konqueror	1

Figure 7: Browsers Covered

Number of tests suites run on specific browsers

Operating System Coverage

Table III

OS	count
Windows 7	402
iOS 6	138
OS X 10.8 Mountain Lion	136
Windows XP	70
Linux	69
Android 4.1.x Jelly Bean	48
Linux (Ubuntu)	46
Android 4.2 Jelly Bean	39
Windows 8	38
OS X 10.7 Lion	36
OS X 10.6 Snow Leopard	33
Android 4.0.x Ice Cream Sandwich	32
iOS 5	20
Android 2.3.x Gingerbread	18
Windows Vista	17
Android	8
Android 2.2.x Froyo	5
FreeBSD	3
Windows Phone 8	3
Windows RT	2
OS X 10.5 Leopard	2
iOS	1
Linux (CentOS)	1
Linux (Fedora)	1
iOS 7	1
OS X	1
Solaris	1

Figure 8: Operating Systems Covered

Number of tests suites run on specific Operating Systems

Data Sizes

XML is inherently larger then than JSON. This is known as 'obviously true' due to several factors inherent to XML markup.

As we can see from Figure 9 and Table I, for each data set the XML and JSON file size can vary significantly depending on the formatting choices used. However, if one carefully chooses a formatting style to minimize size then XML and JSON come very close with XML smaller in some cases and JSON in others. Looking at the compressed sizes of the files in Figure 10 we can see that the file sizes are very close regardless of formatting style.

Figure 9: Corpus File Sizes

Compressed and uncompressed file sizes

Figure 10: Compressed File Sizes

Compressed file sizes (excluding twitter data)

Data Transmission Speed

The predominant factors influencing data transmission speed are the size of the data transmitted and the network speed. In addition, processing speed of the client (browser) and server have some influence as well as packet losses and other internet related issues. Mobile networks generally have significantly higher latency so are more affected by packet loss and retransmission.

This experiment focuses on measuring end to end speeds of HTTP from server to browser using uncompressed and HTTP gzip compression across a range of devices and browsers. It does not attempt to examine the root causes of network bandwidth and traffic.

The range of devices and networks produces a large scatter of transmission speeds. While the trend is that uncompressed and compressed data are fairly linear with respect to bytes transferred there are a lot of outliers. Note that each mark represents the same base document requested uncompressed then compressed via HTTP. If network speed was identical for both then the marks would have x and y values equal, forming a tight line. Rather we see significant amount of outliers.

Transmission Speed

Figure 11

Transmission time in bytes/sec for Compressed (x) vs Raw (y)

Parsing Speed

So far we have dealt with markup agnostic metrics. It doesn’t take any more or less time to transfer the same number of JSON bytes as XML bytes. Parsing is a different matter. This experiment tests two variants of parsing for each JSON and XML. The first variant is to use standard low level JavaScript methods for parsing. The other variant is to use a common library (jQuery). The code for parsing was hand written JavaScript and corresponds to what seems "best practice" in web programming today.

          
  eval('(' + responseString + ')');

	            
   $wnd.jQuery.parseJSON( responseString );

function getIEParser() {
    try { return new ActiveXObject("Msxml2.DOMDocument"); } catch (e) { }
    try { return new ActiveXObject("MSXML.DOMDocument"); } catch (e) { }
    try { return new ActiveXObject("MSXML3.DOMDocument"); } catch (e) { }
    try { return new ActiveXObject("Microsoft.XmlDom"); } catch (e) { }
    try { return new ActiveXObject("Microsoft.DOMDocument"); } catch (e) { }
            
    throw new Error("XMLParserImplIE6.createDocumentImpl: Could not find appropriate version of DOMDocument.");
};
if ($wnd.DOMParser){
    parser=new DOMParser();
    xmlDoc=parser.parseFromString(responseString,"text/xml");
}
else // Internet Explorer
{
    xmlDoc=  getIEParser();
    xmlDoc.async=false;
    xmlDoc.loadXML(txt); 
}
return xmlDoc ;

	            
 $wnd.jQuery.parseXML( responseString );

Figure 16: Parsing speed using JavaScript in seconds vs file size

Parses time in seconds for native JavaScript (y) vs size of raw document in bytes (x).

Figure 17: Parsing speed using jQuery in seconds vs file size

Parses time in seconds for jQuery (y) vs size of raw document in bytes (x).

Query Speed

Testing query across markup languages and a diverse corpus is tricky to achieve without bias. For the purposes of this experiment I postulate that the purpose of loading the document (JSON or XML) is to use all of it's data in some form.

Therefore to simulate a fair "query" I perform a recursive decent of the parsed object's document model and count every node. There are certainly many other equally good definitions of "query" but I wanted to have a similar test across all the corpus that has a reasonable relevance to typical use cases.

The following listings show the JavaScript code for a recursive decent query of all nodes in the document. It is very interesting to the author that despite popular opinion the code for JSON and XML are extremely similar. XML has attributes which adds a few lines of code but otherwise it's effectively the same amount of programming to recurse and examine a JSON object and an XML object in JavaScript and jQuery. XML adds about 4 lines of JavaScript to handle attributes in both the JavaScript and jQuery case, but other than that the code is nearly identical.

	            
var n = 0;
var walk = function(o){
  if( o == null || typeof(o) == "undefined" ) 
     return;
  n++;
 
  for(var prop in o){
    n++;
    if(o.hasOwnProperty(prop)){
      var val = o[prop];

      if(typeof val == 'object'){
          walk(val);
      }            
    }
     else
        var val = o ;
  }
};

walk( this );
return n; 

	            
var n = 0;
var walk = function( o )
{
    if( o == null || typeof(o) == "undefined")
        return ;
    n++;
    $wnd.jQuery.each(o, function(key, value) {
      if( value != null && typeof(value) == "object" )
           walk( value )
}
walk(this);
return n;
        
 

	            
var n =0;
var walk = function( o )
{
    if( o == null || typeof(o) == "undefined" )
       return ;  
    n++;
    if( o.attributes != null && typeof(  o.attributes) != "undefined" )
        for( var x = 0; x <  o.attributes.length; x++ ) {
            n++;
        }
    if(  o.childNodes != null && typeof(  o.childNodes) != "undefined" )
        for( var x = 0; x <  o.childNodes.length; x++ ) {
            walk(  o.childNodes[x] ); 
        }
}
walk(this);
return n;
 

	            
var n = 0;
var walk = function( o )
{
    if( o == null || typeof(o) == "undefined" )
       return ;
    n++;
    $wnd.jQuery( o).children().each( function( ) {
        n++ ;
        walk( this );      
    })
    var a = o.attributes;
    if( a != null && typeof(a) != "undefined" )
        $wnd.jQuery(a).each( function() {
         n++;
    })
}
walk(this);
return n;       
 

Taking a look at the query times however, JSON clearly has an advantage over XML in pure query times. jQuery clearly imposes a significant penalty on XML query but it also imposes a huge penalty on JSON query.

Figure 22: Query speed using JavaScript in seconds vs file size

Query speed in seconds (y) using pure JavaScript vs file size (x).

Figure 23: Query speed using jQuery in seconds vs file size

Query speed in seconds using jQuery (y) vs file size (x).

Putting it together

So far we have looked at pieces of the entire work-flow - network speed, parsing and query. Putting it all together what does it look like? Lets assume the developer has chosen the most compact form for each document (JSON and XML), presuming that would also be the most efficient form. What performance can we see across all devices, OS's and browsers tested ? The following figures show the full time for each test using only the most compact form of each document, tested using pure JavaScript and jQuery, both compressed and uncompressed HTTP transfers.

The results are somewhat surprising. It's not a great surprise that jQuery adds a significant performance penalty, but it is a surprise that across all ranges of platforms that the total time for JSON and XML using native JavaScript is effectively identical. Compare Figure 24 and Figure 26. This is still looking at the whole forest and not the trees, but it is surprising to the author that there appears to be no significant performance penalty using XML over JSON in pure JavaScript. However, jQuery does impose a significant performance penalty to both JSON and XML, much more so for XML.

Figure 24: Complete time uncompressed in pure JavaScript in seconds vs file size

Total time uncompressed in seconds. (transmission + parse + query) in pure Javascript (y) vs raw document size (x)

Figure 25: Complete time uncompressed in jQuery in seconds vs file size

Total time uncompressed in seconds (transmission + parse + query) in jQuery (y) vs raw document size (x)

Figure 26: Complete time gzip in JavaScript in seconds vs file size

Total time gzip compressed in seconds (transmission + parse + query) in pure JavaScript (y) vs raw document size (x)

Figure 27: Complete time gzip in jQuery in seconds vs file size

Total time gzip compressed in seconds (transmission + parse + query) in jQuery (y) vs raw document size (x)

Pulling it apart

It is interesting to see how everything adds up, but so far we've seen the forest not the trees. What does it look like for a particular user? Where is time spent for a specific document on a particular browser and OS?

Since the experiment collects data from such a wide variety of systems it's difficult to show a meaningful view of this. Averages and even percentiles mean very little when looking at data that spans orders of magnitude. Instead lets look at a couple typical test results which might help make sense of the big picture.

The following are the total of median times (median(data transfer) + median(parse) + median(query)) for the most compact form of the Twitter document in both JSON and XML using both pure JavaScript and jQuery across all browsers. These results span across a variety of devices, some browsers are obvious if they are mobile or desktop and some are not obvious.

In the following charts, the vertical axis is time in seconds (more is slower), and the horizontal axis represents one combination of "User Agent" + either JSON or XML (for every user agent there is two columns of data). Network transfer time is indicated as blue. Parsing time is indicated as red. Query time is indicated as green.

Figure 28: Browser Processing Speed, Uncompressed, JavaScript

Median total times in seconds (y) for Twitter document, raw file using JavaScript vs. user agent (y). User Agent is often a proxy for device type.

In Figure 28 we can see that the total time is dominated by network transmission time. Mobile browsers such as Android Webkit, IE Mobile and Opera Mobile take more time in the parsing and query layer probably due to their slower CPU.

Figure 29: Browser Processing Speed, Uncompressed, jQuery

Median total times in seconds (y) for Twitter document, raw file using jQuery vs. user agent (y). User Agent is often a proxy for device type.

In Figure 29 notice the change in the vertical access to accomidate for large time spent in query and parse on some devices. Network time remains the same but in many more cases are parse and query time a larger contributor.

Figure 30: Browser Processing Speed, Compressed, JavaScript

Median total times in seconds (y) for Twitter document, compressed using JavaScript vs User Agent (y). User Agent is often a proxy for device type.

In Figure 30 notice the change in vertical axis. The total time is dramatically smaller revealing the relative times for parse and query being a larger portion of the total time, especially on mobile devices.

Figure 31: Browser Processing Speed, Compressed, jQuery

Median total times in seconds (y) for Twitter document, compressed using jQuery vs User Agent (y). User Agent is often a proxy for device type.

In Figure 31 we can see the impact of using jQuery on mobile devices especially for XML.

Suggestions to Architects and Developers

Architects and developers who are seriously interested in maximizing performance should consider the following.