Balisage Paper: Loose-leaf publishing using Antenna House and CSS

Identifying Change Page Sets

Change boundaries are reflected in the area tree by inline objects with IDs that have a specific structure, created from areaTreeMarker elements generated as part of the HTML preprocess from the change marking processing instructions ("take markers") inserted by the authors.

The source as authored is:

<?pdf take-begin job="S01" firstpage="15"?>
<section id="x88B6B95E7E6A" data-type="chapter" >
 ...
<?pdf take-end job="S01" firstpage-ref="15"?>
...
</section>

Which results in this HTML input to the pagination process:

<areaTreeMarker id="take:take-begin:job=S01:firstpage=15:d16p512"/>
<section id="x88B6B95E7E6A" data-type="chapter" >
  ...
<areaTreeMarker id="take:take-end:job=S01:firstpage=15:d16p512"/>
...
</section>

And then these elements in the area tree:

<InlineArea
  id="take:take-begin:job=S01:firstpage=15:d16p512" 
  width="0pt" height="0pt"
  ...
/>
...
<InlineArea
  id="take:take-end:job=S01:firstpage-ref=15:15:d16p512" 
  width="0pt" height="0pt"
  ...
/>

These marker areas are then used by utility functions that find pages that are within change sets:

  <!-- Determine if the context element is the start of a take 
       for the specified update ID. A given page can have zero or more take starts
       for a given update ID.
       @param context A PageViewPortArea element (or other element that 
                      could be an ancestor of a take marker)
       @param jobID The update ID to check for
       @param doDebug Turns debugging on or off.
       @return true if a take begin marker is found for the specified update ID.
    -->
  <xsl:function name="at:is-take-start" as="xs:boolean">
    <xsl:param name="context" as="element()"/>
    <xsl:param name="jobID" as="xs:string?"/>
    <xsl:param name="doDebug" as="xs:boolean"/>
    
    <xsl:variable name="page" as="element(at:PageViewportArea)" 
         select="$context/ancestor-or-self::at:PageViewportArea"
    />
        
    <xsl:variable name="take-specifier" as="element()*"
      select="at:get-take-specifier($context, $jobID)"
    />
    <xsl:variable name="result" as="xs:boolean"
      select="exists($take-specifier)"
    />
    <xsl:sequence select="$result"/>
  </xsl:function>

  <!-- 
    Gets the element that specifies the start of a take for the
    the specified update.
    @context Element to look in for a take specifier
    @jobID The ID of the update to get the take specifier for
    @return The first element that starts a take for the specified update, if any.
    -->
  <xsl:function name="at:get-take-specifier" as="element()?">
    <xsl:param name="context" as="element()"/>
    <xsl:param name="jobID" as="xs:string?"/>           
    
    <xsl:variable name="take-specifier" as="element()*"
      select="($context//*[starts-with(@id, 'take:take-begin:')]
                          [contains(@id, concat('job=', $jobID))])[1]"
    />
    
    <xsl:sequence select="$take-specifier"/>
  </xsl:function>

These functions then make it easy to distinguish changed pages from unchanged pages:

<!-- Context is the area tree root element -->
<xsl:template name="update-page-numbers">
  <xsl:param name="doDebug" as="xs:boolean" tunnel="yes" select="false()"/>

  <xsl:for-each-group 
    select="at:PageViewportArea"
    group-starting-with="at:PageViewportArea[at:is-take-start(., $jobID)]">
    <xsl:choose>
      <xsl:when test="at:is-take-start(., $jobID)">
        <xsl:call-template name="update-page-numbers-for-take-group">
          <xsl:with-param name="doDebug" as="xs:boolean" select="$doDebug"/>
          <xsl:with-param name="pages" as="element(at:PageViewportArea)+" 
            select="current-group()"
          />
        </xsl:call-template>
      </xsl:when>
      <xsl:otherwise>
        <!-- Must be before first take -->
        <xsl:apply-templates select="current-group()" mode="update-page-numbers">
          <xsl:with-param name="doDebug" as="xs:boolean" select="$doDebug"/>
        </xsl:apply-templates>
      <xsl:otherwise>
    </xsl:choose>
  </xsl:for-each-group>    
</xsl:template>

At this point we know the start of a set of changed pages (the take start) but we don't know the end page.

A set of changed pages must always be an even number of pages.

If the explicitly-marked last page of the change set happens to be even then we're done.

However, if the last page is not even then a blank backing page has to be added to the change sequence.

The blank page can either come from the area tree if the page following the last changed page happens to be a blank even page (for example, a page generated in order to force the following page onto an odd page), otherwise it is necessary to synthesize a blank page. This is done by copying the current page and adding 1 to the page's absolute and ordinal page numbers. Only those parts of the page that are needed on a blank page are retained, such as the page edge region that contains the display page number or other edge region components that need to be kept. The XSLT provides simple configuration to adjust this as needed. This also requires a function that can distinguish blank pages from non-blank pages, i.e., pages where the effective string value of the page body is only whitespace or matches a configured "This page intentionally left blank" marker.^[3]

Another non-obvious complication is when a change set initially ends with a blank odd page. In this case, rather than adding a second blank even backing page, you simply omit the blank odd page. It should never be possible to have a change set that ends with three blank pages. Two ending blank pages can happen due to either author error where unnecessary page breaks have been forced or some other combination of factors results in two blank pages.

Constructing New Page Number Sequences

Given the set of pages in a change set we know number of pages in the set, the starting page number, and the page number of the page that follows the change set, if any.

With that it's a simple matter of calculating the difference between start + following and the actual count of pages. Any pages with an ordinal number greater than number of original pages must be a point page.

The XSLT then constructs a sequence of new page numbers and uses it to update the pages in the change set (some details omitted for brevity):

<xsl:template name="update-page-numbers-for-take-group">
  <xsl:param name="pages" as="element(at:PageViewportArea)+"/>

  <xsl:variable name="page-numbers" as="xs:string*">
    <xsl:variable name="first-page-number" as="xs:string?"
      select="at:get-folio($take-start)"
    />          
    <!-- All pages should have page numbers of one form or another 
         but it could happen that a page has no number -->
    <xsl:if test="exists($first-page-number)">
      <xsl:sequence select="at:calculate-page-numbers-for-take(
          $pages-in-take, $jobID)"
      />
    </xsl:if>
  </xsl:variable>

  <xsl:for-each select="$pages-in-take">
    <xsl:variable name="pos" as="xs:integer" select="position()"/>
    <xsl:variable name="new-page-folio" as="xs:string?"
      select="$page-numbers[$pos]"
    />
    <xsl:apply-templates select="." mode="update-page-numbers">
      <xsl:with-param name="new-page-folio" as="xs:string?" 
        select="$new-page-folio"
      />
      <xsl:with-param name="in-job" as="xs:boolean" select="true()"/>
    </xsl:apply-templates>
  </xsl:for-each>
</xs:template>

...
<xsl:function name="at:calculate-page-numbers-for-take" as="xs:string+">
  <xsl:param name="pages" as="element()+"/>
  <xsl:param name="jobID" as="xs:string"/>

  <!-- The number of pages whose page number does not need to change -->
  <xsl:variable name="ordinal-page-count" as="xs:integer"
    select="$next-page-num - $first-page-num"
  />
  <xsl:variable name="point-page-count" as="xs:integer"
    select="count($pages) - $ordinal-page-count"
  />
  <xsl:variable name="point-page-count" as="xs:integer"
    select="if ($point-page-count lt 0) 
            then 0 
            else $point-page-count"
  />
  <!-- The number of pages whose page number does not need to change -->
  <xsl:variable name="ordinal-page-count" as="xs:integer"
    select="$next-page-num - $first-page-num"
  />
  <xsl:variable name="point-page-count" as="xs:integer"
    select="count($pages) - $ordinal-page-count"
  />
  <xsl:variable name="point-page-count" as="xs:integer"
    select="if ($point-page-count lt 0) 
            then 0 
            else $point-page-count"
  />
  <xsl:variable name="before-points" as="element()+"
    select="$pages[position() le $ordinal-page-count]"
  />
  <xsl:variable name="point-pages" as="element()*"
    select="$pages[position() gt $ordinal-page-count]"
  />
  <!-- Generate display page numbers for each page before the point pages -->
  <xsl:for-each select="$before-points">
    <xsl:variable name="page-num-base" select="string($first-page-num + position() - 1)" as="xs:string"/>
    <xsl:number value="$page-num-base" format="{$page-number-format}"/>
  </xsl:for-each> 
  <!-- Generate display page numbers for point pages -->
  <xsl:for-each select="$point-pages">
    <xsl:variable name="point-number" as="xs:integer" select="position()"/>
    <xsl:sequence select="concat($point-page-base-formatted, '.', $point-number)"/>
  </xsl:for-each>
</xsl:function>

At this point, the display page numbers of each page in each change set have been updated to reflect the application of starting page numbers and point page numbers. References to these pages have not yet been updated.

This process also marks every page as being "in the job" or "out of the job", which is then used by the filtering step. Any page that is part of a change set or is a page that is always included (cover pages, insertion instructions, etc.) are marked as in the job, all other pages are marked as out of the job.

Note that the code does not bother to adjust the horizontal position of point page numbers on the pages for the simple reason that there's no easy way to know how the page number alignment should be adjusted: centered, left aligned, or right aligned? At least for the Municode styles, the visual affect of adding a point page or changing a number from 1 to 2 digits is minimal and would not normally be noticed.

However, references to the pages in some contexts do need to be adjusted, for example in the table of contents, as the numbers are consistently right-aligned and therefore a change in horizontal placement will be noticeable. These documents do not normally use page numbers for references in normal flowed text, which simplifies the problem.

If page numbers were used it would probably be necessary to reserve extra space around the numeric part of the page number reference or some marker technique used to indicate what the current alignment and justification are. Given that information there should be no problem adjusting the horizontal position of the text before or after the page number reference. In the worst case, the code would need to apply word or character spacing adjustments to avoid having a long number cause the line to end pass a margin (for example, where the text would end up overlapping a border or text not tagged as being in the same line on the page.

Because the area tree includes all geometric information down to the individual text string level, it's always possible to adjust the layout or otherwise detect overlaps but that level of sophistication should not normally be needed.

In legal publications the normal practice is to refer only to a section or paragraph number or a figure or table number (and possibly title), which simply avoids the problem. Page numbers are usually limited to generated navigation structures like tables of contents and indexes.

Page History Database

Conceptually the page history database consists of two separate data sets:

The physical page records enable generation of the update instructions and the list of effective pages. The element history records enable generating page number references to elements on unchanged pages.

The history database is maintained over time for a given publication, updated when a given update is published for delivery to the client.

The page history database could be implemented in many different ways. For this project, the initial implement uses an XML file that is maintained along with the publication's source in the version control system that maintains the publication source. The file is read during area tree post processing and updated when a final publication is produced.

A more robust implementation might use a dedicated database application and a REST service to manage the database but the scope and infrastructure for that was not available in this project.

Capturing Element Target Details In The Area Tree

In CSS a page number reference is a counter reference scoped to a specific element:

div.toc-entry > span.page > a.body:before,
div.minitoc.pg div.minitoc-entry > span.page > a:before
{
	content: target-counter(attr(href url), page);
}

The <a> element is also rendered as a navigable hyperlink using the AHF -ah-link: extension property:

a[href]		
{
 -ah-link: attr(href url) 
}

in order to mark references to page numbers the HTML preprocess adds spans with structured ID values that mark the text as a page number reference:

<div class="toc-entry">
  <span class="title">
    <a href="#x88B6B95B1C04">
      <span class="h1">
        <span class="uppercase">Officials </span> of the <span class="uppercase">Town of
          Lovettsville, Virginia At the Time of This Codification </span></span></a></span>
  <span class="page" id="page:d121e33">
    <span class="page-number-marker">&#x200B;</span>
    <a id="pageref:d121e36" class="frontmatter" href="#x88B6B95B1C04"></a>
    <span class="page-number-marker">&#x200B;</span>
  </span>
</div>

The span with a class of "page" marks its content as being a page number (full folio) using the structured @id value of "page:{generated-id}", while the <a> element's @id of "pageref:{generated-id}" marks it as a page number reference (just the page number part of the folio).

This results in reliably-findable markup in the area tree:

<InlineArea 
  id="page:d121e49"
  ...>
  <InlineArea ...>
    <TextArea ...
      text="&#x200B;"
    />
  </InlineArea>
  <InlineArea 
    internal-destination="x88B6B95B4E50" 
    id="pageref:d121e52"
    ...>
    <InlineArea ...>
      <TextArea ...
        text-width="5.37pt" 
        text="v" 
      />
    </InlineArea>
  </InlineArea>
  <InlineArea ...>
    <TextArea ...
      text="&#x200B;" 
    />
  </InlineArea>
</InlineArea>

The outer InlineArea is identified as a page (full folio). the InlineArea elements with content of &#x200B; (non-breaking space) mark the boundaries between the prefolio, page number, and post folio. There is no prefolio or postfolio in this example.

The page number is identified by @id "pageref:d12e52" in this example. Note that the TextArea specifies both the text ("v") and the rendered width ("5.37pt").

In addition, the target ID is automatically captured by AHF on the @internal-destination attribute of the InlineArea that results from the -ah-link: extension property.

The final challenge is finding the target element's starting position within the area tree in order to find the physical page referenced in order to then get the new page number for that page. While AHF will capture the ID of any element in the area tree, it won't necessarily do so at the right position.

Thus the HTML preprocess generates markers that signal the start and end of elements to which page number reference might be made (sections, figures, and tables in the current content set):

<section id="x88B6B95B4E50" ...>
  <header>
    ...
    <h1 style="line-height:2em;">
      <span class="uppercase">Boards and Commissions</span>
      <br /> of the <br />
      <span class="uppercase">Town of <br /> Lovettsville, Virginia</span></h1>
    <p data-type="startpage">7</p>
  </header>
  <areaTreeMarker
    id="marker:section:officialsoriginal:start:narrow:startpage=7:id=x88B6B95B4E50"
  />
  ...
  <areaTreeMarker 
    id="marker:section:officialsoriginal:end:narrow:id=x88B6B95B4E50"
  />
</section>

Note that the marker is emitted after the header, which produces the section title in the final result. This ensures that the marker in the area tree occurs on the same page as the title and not on the page before, for example, if the CSS for the header creates a page break or the keep rules result in a forced break before the title.

This then results in inline areas in the area tree that serve as reliable markers for the start and end of the section. Only the start marker is needed for page number updating but the end marker is needed in order to count the pages taken up by some sections, such as the tables of contents and indexes, where the number of pages is needed, for example to distinguish the count of generated pages from authored pages for billing purposes.

To get the actual page number for a given target element the code uses the same logic used during initial page update processing to find the rendered page number on the page. That is, one you have a page in the area tree, getting that page's display page number is a single function call to the appropriate utility function:

<xsl:variable name="target-page" as="element()?"
  select="at:get-referenced-page($context)"
/>
<xsl:choose>
  <xsl:when test="at:is-within-take($target-page)">
    <xsl:sequence select="at:get-folio($target-page)"/>
  </xsl:when>
  <xsl:otherwise>
    <xsl:sequence select="at:get-stored-page-number-for-pageref($pageListingDoc, $context, false())"/>
  </xsl:otherwise>
</xsl:choose>

Updating Page Number References

The display page numbers are constructed with a format that makes it reliable to find them:

<LineArea text-altitude="8.87939pt" ...>
  <TextArea text="&#x200B;" .../>
  <TextArea text="2" .../>
  <TextArea text="​&#x200B;" .../>
</LineArea>

The marker is a LineArea containing TextArea elements where the text contains a zero-width space (\u200B). Zero-width spaces are not otherwise used for anything in this content and thus make ideal marker characters as they have no visual effect.

The XSLT provides helper functions that work with display page numbers: finding them, getting the folio parts, etc.

The practical challenge with updating page numbers is that the width of the new page number will usually be different from the width of the original, which means that the horizontal position of the text area has to be adjusted to account for the difference in width.

However, the post process does not have any direct way to know what the rendered width of the new page number is.

One solution would be to use extension functions and something like the Java2D libraries to render the characters using the font details available in the area tree.

For this project we took a more brute force but more reliable approach, namely to include in the area tree an instance of every possible character that could occur in a page number (0-9, a-z, A-Z, ".", "-", ":", etc.) in every font and size used in the publication in a form that enables lookup of the rendered characters. This set of "character samples" is generated as part of the HTML preprocessing.

The HTML is:

<char-samples id="util:char-samples">
  <char-sample class="body">
    <char-set class="number-set sz83" id="util:char-set:body-sz83">
     <decimal>.</decimal><char>0</char><char>1</char><char>2</char><char>3</char><char>4</char><char>5</char><char>6</char>
<char>7</char><char>8</char><char>9</char><char>a</char><char>b</char><char>c</char><char>d</char><char>e</char><char>f</char>
<char>g</char><char>h</char><char>i</char><char>j</char><char>k</char><char>l</char><char>m</char><char>n</char><char>o</char>
<char>p</char><char>q</char><char>r</char><char>s</char><char>t</char><char>u</char><char>v</char><char>w</char><char>x</char>
<char>y</char><char>z</char><char>A</char><char>B</char><char>C</char><char>D</char><char>E</char><char>F</char><char>G</char>
<char>H</char><char>I</char><char>J</char><char>K</char><char>L</char><char>M</char><char>N</char><char>O</char><char>P</char>
<char>Q</char><char>R</char><char>S</char><char>T</char><char>U</char><char>V</char><char>W</char><char>X</char><char>Y</char>
<char>Z</char>
</char-set>
  ...
  </char-sample>
  ...
</char-samples>

Because eash <char-set> specifies one of the font-setting classes from the CSS, the font and size will be present in the area tree. Those values then serve as the lookup key for any character, along with the character itself. The only maintenance task is keeping the generating XSLT in sync with the CSS styles so that all relevant font and size differences are represented.

The area tree markup is then:

<BlockViewportArea id="util:char-samples" ...>
  <BlockArea 
    id="util:char-set:body-sz83" ...>
    <LineArea ...>
      <InlineArea ...>
        <TextArea ...
          width="2.224pt"
          font-family="NewCenturySchlbk LT Std"
          font-size="8pt"
          text-width="2.224pt" 
          text="." 
       />
      </InlineArea>
      ...
    <LineArea>
  </BlockArea>
</BlockViewportArea>

Because each source character is in a separate element it ensures that there is one text area for each character in the area tree. The text areas specify the exact width in points of the character.

This XSLT code then looks up the details of a character given the font details:

  <xsl:key name="char-samples" 
    match="at:BlockArea[starts-with(@id, 'util:char-set:')]//at:TextArea"
    use="at:make-char-sample-key(@font-family, @font-size, @text)"
  />

  <!-- Try to find a character sample for the specified font family, size, and text value. -->
  <xsl:function name="at:get-char-sample" as="element(at:TextArea)?">
    <xsl:param name="context" as="element()"/><!-- Area tree element -->
    <xsl:param name="font-family" as="xs:string"/>
    <xsl:param name="font-size" as="xs:string"/>
    <xsl:param name="text" as="xs:string"/>
    <xsl:variable name="key" as="xs:string" 
      select="at:make-char-sample-key($font-family, $font-size, $text)"
    />
    <xsl:variable name="result" as="element()?"
      select="key('char-samples', $key, root($context))[1]"
    />
    <xsl:sequence select="$result"/>
  </xsl:function>

  <xsl:function name="at:make-char-sample-key" as="xs:string">
    <xsl:param name="font-family" as="xs:string"/>
    <xsl:param name="font-size" as="xs:string"/>
    <xsl:param name="text" as="xs:string"/>
    
    <xsl:variable name="key" select="string-join(($font-family, $font-size, $text), ':')"/>
    <xsl:sequence select="$key"/>
  </xsl:function>

Then it's simply a matter of doing the math to the determine the amount of adjustment to apply to the text areas in the affected line area and update them, here for the case of page references within ToC entries that use leaders (the most common case):

<!-- Page references in lines with leaders (e.g., ToC pages). This will be the most common case. 
    
     There should only be one page reference, which means we can simply calculate the adjustment to
     the leader and all inline and text nodes following the leader should be good to go.
     
     The prefolio and postfolio should always be the same.     
-->
<xsl:template match="at:LineArea[at:is-within-page-reference-line(.)]">
  
  <xsl:variable name="pagerefs" as="element(at:InlineArea)+"
    select=".//at:InlineArea[starts-with(@id, 'pageref:')]"
  />
  <!-- There should be exactly one TextArea in the pageref InlineArea -->
  <xsl:variable name="pageref" as="element(at:InlineArea)"
    select="$pagerefs[1]"
  />
  <xsl:variable name="target-page" as="element()?"
    select="at:get-referenced-page($pageref)"
  />
  
  <xsl:variable name="page-ref-text-area" as="element(at:TextArea)?"
    select="($pageref//at:TextArea)[1]"
  />
  <xsl:variable name="orig-page-number" as="xs:string?"
    select="$page-ref-text-area/@text"
  />

  <!-- NOTE: This is the formatted page number, e.g., 'xix', not 19 -->
  <xsl:variable name="page-number" as="xs:string?" 
     select="at:get-referenced-page-number($pageref)"
  />
  
  <xsl:choose>
    <xsl:when test="empty($page-number)">
      <!-- Nothing to do, just use what we have -->
      <xsl:next-match/>
    </xsl:when>
    <xsl:otherwise>
      <xsl:variable name="current-width" as="xs:double" 
        select="at:get-length-value($page-ref-text-area/@text-width)"
      />
      <xsl:variable name="new-number-width" as="xs:double"
        select="at:get-string-width($page-ref-text-area, $page-number)"
      />    
          
      <xsl:variable name="width-difference" as="xs:double"
          select="$new-number-width - $current-width"
      />
      <xsl:choose>
        <xsl:when test="$width-difference ne 0.0">
          <!-- Now figure out how many dots to remove from the 
               leader to account for the added space: -->
          <xsl:variable name="leader-area" as="element(at:LeaderArea)" 
            select="$pageref/preceding::at:LeaderArea[1]"
          />
          
          <xsl:variable name="leader-string" as="xs:string" 
            select="$leader-area/at:TextArea/@text"
          />
          <xsl:variable name="leader-width" as="xs:double" 
            select="at:get-length-value($leader-area/at:TextArea/@text-width)"
          />
          <xsl:variable name="dots" as="xs:string*"
            select="tokenize($leader-string, ' ')"
          />
          <xsl:variable name="dot-width" as="xs:double"
            select="$leader-width div count($dots)"
          />
          <xsl:variable name="sign" as="xs:integer" 
            select="if ($width-difference lt 0) then 1 else -1"
          />
          <xsl:variable name="dot-count-diff" as="xs:integer"
            select="if (number($dot-width))
                    then (($width-difference idiv $dot-width) + 1) * $sign
                    else 0
                    "
          />
          <xsl:variable name="dot-count" as="xs:integer"
            select="$dot-count-diff + count($dots)"
          />
          <xsl:variable name="new-leader-string" as="xs:string?" 
            select="string-join(for $n in 1 to $dot-count return $dots[1], ' ')"
          />    
          <xsl:variable name="new-leader-width" as="xs:double"
            select="$dot-width * $dot-count "
          />
          <xsl:copy>
            <xsl:apply-templates select="@*"/>
            <xsl:apply-templates>
              <xsl:with-param name="new-leader-string" as="xs:string?" 
                tunnel="yes" select="$new-leader-string"
              />
              <xsl:with-param name="new-leader-width" as="xs:double?" 
                tunnel="yes" select="$new-leader-width"
              />
              <xsl:with-param name="left-adjust" as="xs:double" 
                tunnel="yes" select="$width-difference"
              />
              <xsl:with-param name="page-ref" as="element(at:InlineArea)?" 
                tunnel="yes" select="$pageref"
              />
            </xsl:apply-templates>
          </xsl:copy>        
        </xsl:when>
        <xsl:otherwise>
          <xsl:copy>
            <xsl:apply-templates select="@*"/>
            <xsl:apply-templates>
              <xsl:with-param name="doDebug" as="xs:boolean" 
                tunnel="yes" select="$doDebug"
              />
              <xsl:with-param name="new-leader-string" as="xs:string?" 
                tunnel="yes" select="()"
              />
              <xsl:with-param name="new-leader-width" as="xs:double?" 
                tunnel="yes" select="()"
              />
              <xsl:with-param name="left-adjust" as="xs:double" 
                tunnel="yes" select="0.0"
              />
              <xsl:with-param name="page-ref" as="element(at:InlineArea)?" 
                tunnel="yes" select="$pageref"
              />
            </xsl:apply-templates>
          </xsl:copy>                      
        </xsl:otherwise>
      </xsl:choose>
    </xsl:otherwise>
  </xsl:choose>
</xsl:template>