Balisage Paper: Syntax-From-Doc

Legacy XML Review

Foundational to this project was an effort to clearly define the needs of the proposed syntax service. The project began with an analysis of the existing legacy R&D-maintained syntax XML to define a satisfactory JSON model and to map documentation XML to the new model. Without a DTD, literal document-by-document analysis was required to discern structure and content. As discussed in the project justification section above, the shortcomings of both the structure and content of the XML were fairly clear as is the process for developing and maintaining this XML. From this baseline, the team next worked with R&D to define the object model for the new syntax prior to designing XML from which it would be derived.

JSON Design and Documentation

As part of a wider organization-wide initiative to standardize API development, SAS has adopted OpenAPI for API documentation and mandated that services should, at the least, support the application/json media type. With this requirement in mind, the R&D consumers of the syntax service have designed around and made a requirement that the service be primarily focused on syntax elements modeled in JSON objects. Thus, the team first established the JSON model with the intention of working from it to develop adequate XML markup. The design process began with an analysis of the language element it should model. Each SAS language element has lexical features unique to its element class relevant to its model. In the case of procedures, the model is as follows:

This structure is illustrated in Figure 2, which captures a brief SAS procedure and highlighting its arguments and statements.

Figure 2: Example Means Procedure

In addition to the literal statements and arguments associated with a given procedure, R&D identified additional required metadata that would be of use to syntax consumers at each level of a procedure to be included in the JSON object in the syntax service:

Additionally each level of the syntax requires support site information, such that consumers of the syntax object could easily link from a nested syntax object to the documentation on SAS support sites from which it was extracted, for supplemental information.

The final procedure JSON structure appears below.

{
  "id": "string",
  "name": "string",
  "updated": "string",
  "locale": "string",
  "sasRelease": "string",
  "productGroups": [
    {
      "name": "string"
    }
  ],
  "interactive": true,
  "statements": [
    {
      "name": "string",
      "description": "string",
      "help": "string",
      "aliases": [
        "string"
      ],
      "arguments": [
        {
          "name": "string",
          "optional": true,
          "placeholder": true,
          "followsDelimiter": "string",
          "aliases": [
            "string"
          ],
          "description": "string",
          "help": "string",
          "type": "string",
          "arguments": [
            null
          ],
          "supportSiteTargetFragment": "string"
        }
      ],
      "supportSiteTargetFile": "string"
    }
  ],
  "supportSiteInformation": {
    "docsetId": "string",
    "docsetVersion": "string",
    "docsetTargetFile": "string"
  },
  "version": 0
}

Documentation XML Design

SAS documentation has used a proprietary DTD since migrating from SGML. The traditional DTD, informally known as the authoring.dtd, had various means of documenting the SAS language, depending on the type of language element and the intent of the writer. Developed organically through feature requests and requirements changes, the DTD grew to contain over 500 elements. Several years ago, an effort to create a new, streamlined DTD began. The new DTD, named the document.dtd, was designed without extraneous tags and features until equivalency with the authoring.dtd was explicitly requested.

Analysis of the R&D syntax documentation requirements revealed both gaps in the authoring.dtd tagging structure and too lenient a tagging structure to produce consistent syntax extraction results. Rather than introduce breaking changes to the authoring.dtd, requiring conversion and manipulation of legacy content, the Document Architecture group decided to develop a new syntax documentation model in the document.dtd and to create XSLT to convert old authoring.dtd content to the new DTD to migrate all syntax documentation when necessary. Where possible redundant elements were consolidated and eliminated. The resulting final version of the document.dtd with full feature parity for authoring.dtd is currently 348 elements—a substantially smaller tag set. Some examples of element refactoring and consolidation are illustrated in Figure 3.

Figure 3: Refactored Elements Related to Syntax Extraction

Most important to syntax extraction is the <refBlock> element. It replaced two elements in the authoring.dtd and is designed to be flexible enough to capture syntax documentation for numerous SAS language elements, including statements, functions, formats, and informats. By designing this element to document most SAS language elements, the XSL for syntax extraction can share code for extracting these different elements and vary only slightly at the edges, reducing transformation complexity. <refBlock> are the child of either <refProc> or <refDictEntryCollection>. <refDictEntryCollection> is a tag retaining its authoring.dtd name intended to group numerous related refBlock describing language elements. <refProc> is a tag designed to document SAS procedures, and it contains both syntax and also information supplemental to the syntax documentation. It produces a special organizational structure in the output, for an example see the httpproccite in the Base SAS Procedures Guide.

From a <refBlock>, its <name> maps to the name of a language element, and its <shortDescription> maps to the description. The syntactic structure of the syntax is captured in <syntaxSimple> within the <syntax> element, while the arguments are documented in <argDescriptionPair> within the <syntaxDescription> element. To facilitate documenting nested arguments, <argumentDescription>s can contain nested <argDescriptionPair>, producing the nested argument objects as shown in Appendix A. Figure 4 demonstrates how some of the XML maps to content in the code completion window.

Figure 4: Procedure arguments and syntax help

The improvements over the legacy R&D maintained syntax XML are numerous. Whereas previously, the syntax for a language element was keyed in CDATA, the new XML contains numerous semantic tags. This produces better formatting in the code completion window and facilitates algorithmic identification of argument type in the extraction, such that writers do not have to tag arguments with their type. Procedure statements are now grouped in several semantically significant parent tags (<requiredArgGroup>, <optionalArgGroup>, <otherArgGroup>). By grouping in this manner, the service will be able to indicate which statements are required and which are optional—information previously only present in the CDATA of the <statementHelp> and not accessible to syntax documentation consumers. This should allow for future enhancements to the code completion to sort code completion suggestions and warn of missing required arguments. Additionally, now that the syntax information is extracted from the documentation, direct links from the code completion to relevant documentation are possible by including the UUID of source in the syntax objects. Prior to this new service, the IDE simply linked by way of a link to search results for the topic in our documentation.

Conversion and Reauthoring

Despite best efforts, the changes necessary to the document.dtd necessitated significant conversion and reauthoring work. There were breaking changes to the document.dtd, namely the elimination of redundant tags and some restructuring. Additionally, all existing syntax documentation was authored in the authoring.dtd, so the team developed both an authoring-to-document and a document-to-document XSL transformation.

To facilitate tech writer use, the team created a parameterized jenkins job through which authors could batch convert their documentation projects to the new DTD. Where possible, the conversion was hands off, but where incompatibilities and questions arose, the conversion inserted processing instructions with comments configured to surface when the author transformed the content to an HTML preview, if possible, or included DTD validation failures in the Jenkins job log to debug and triage.

Extracting Syntax From Doc

The majority of the SAS language documentation is authored in XML maintained in projects versioned in CVS. When writers commit changes to CVS, jenkins builds produce build artifacts (e.g., HTML, PDF, ePub) stored in artifactory. To deliver extracted syntax to the syntax REST service, the team added a new step in the HTML build, so that extracted syntax would end up in the HTML artifact after each build.

The XSL stylesheets consist of a controller stylesheet called from the Java-based build pipeline, executed using Saxon EE. The controller is an XSL 3.0 stylesheet that passes a flattened XML file using fn:transform() to external stylesheets that return a map of result XML documents. The controller then iterates through the returned map serializing each to JSON files using fn:xml-to-json(). Each SAS language element type has its own external stylesheet to handle the minor extraction differences between elements and to gather like elements based on structure and attributes. These stylesheets include a common stylesheet for shared templates that do not vary based on context. An example of the technique for controlling procedure extraction via external stylesheet is presented below.

<!-- The root template in ExtractSyntaxController.xsl contains a series of the following variable
  and call-template, each pointing to a different XSL stylesheet to extract different syntax doc 
  The source node is a flattened XML file of an entire "docset"-->

<xsl:variable name="stylesheetParams"
  select="
    map {
     QName('', 'output_dir'): $output_dir,
     QName('', 'docsetVersion'): $docsetVersion,
     QName('', 'docsetId'): $docsetId
    }"
/>

<xsl:variable name="procResults"
  select="
    transform(
       map {
          'source-node': .,
          'stylesheet-location': 'extractProcSyntaxJSON.xsl',
          'stylesheet-params': $stylesheetParams
          }
    )"
/>

<!-- This template iterates over the returned map of result documents
  and writes each to the file system, either as XML if debugging is enabled
  or as JSON using fn:xml-to-json() -->
<xsl:call-template name="outputResults">
    <xsl:with-param name="results" select="$procResults"/>
</xsl:call-template>

This stylesheet structure yields several benefits. By using a controller stylesheet and fn:transform(), Saxon is only invoked once, though the XML is processed several times, offering some performance benefit and moves process flow from its traditional place in the Java build to more easily maintained XSL. Furthermore, the syntax-from-doc project is an iterative one. The service is being introduced a syntax element at a time—first procedures, second functions, and so forth. Each stylesheet can be developed and tested on its own. Once finalized, adding an additional pass in the controller is trivial.

The Syntax Service

Consumers of extracted syntax information consume it via the syntax microservice REST API. The service itself is a springboot app utilizing httpd for routing and postgres as a database. It is deployed via docker container. Endpoints, HTTP requests supported, and JSON object models are documented in swagger OpenUI and accessible via an internal SAS Swagger Hub. The service is populated by a Jenkins job that runs nightly after the documentation builds. It first queries an internal existdb database containing all SAS XML documentation to compile a list of every project containing syntax. With a list of syntax projects, the service then retrieves the HTML artifacts from Artifactory for each project, extracts the JAR files, and collects the JSON objects for each syntax language element. Finally, the Jenkins job POSTs each object to the appropriate syntax service endpoint, replacing the existing object in the service with the most recently extracted content. The result of this pipeline process is a service easily maintained by the authors, wherein updates to syntax appear in the service in a fully-automated fashion.

The Syntax Viewer

For now, SAS customers will interact with the result of the syntax-from-doc project only in the SAS IDEs. On the other hand, internal users, especially SAS technical writers and editors, need visibility into the extracted syntax as they both confirm accuracy of extraction and revise and adapt their tagging to meet the more stringent requirements of this new service. For this purpose, the documentation engineering team developed a syntax viewing service. This viewer allows writers to see a user-friendly representation of the syntax information rather than expecting them to review raw JSON objects.

The syntax viewer is a relatively simple Vue.js front-end app built on technology already in use for several other internal documentation web application front-ends. Essentially, it provides a UI for the syntax service, so authors can explore and view the collections of syntax objects, with search, filtering, and sorting by element type and SAS release. Once viewing an individual syntax element, the UI is utilitarian, offering a formatted view of the key/value pairs of the JSON objects. Where appropriate, it translates escaped HTML and character entities to render text as it would appear in the IDE. It is a viewer only. If content changes are required, writers must go back to the source XML, make them there. When the writers finish updates, their committed changes will trigger a Jenkins build, and their updated syntax will appear in the syntax viewer.

Author Communication and Tagging Quality

At the heart of the syntax-from-doc project lies the new tagging introduced to facilitate extraction. The document architecture team invested considerable effort into the resulting tagging architecture. Despite this effort, its roll out was not seamless, and the new tagging still presents recurring questions and requires regular modification as requirements continue to be refined. While management held information sessions to explore the new tagging, conversion, and expectations, confusion still surrounds tagging best practices. Prior to this new service, writers needed only to tag to produce visually appealing HTML/PDF deliverables. Now, semantic information is far more important. Thus, writers must adopt more rigorous tagging conventions. With each language element added to the extraction, large swathes of tagging inconsistencies and workarounds are surfaced that produce passable deliverables but inaccurate syntax service objects. The cleanup effort is ongoing and monumental.

Confusion also still surrounds how the tagging effects the resulting extracted documentation, and work is underway to develop tagging best practices. Provided with the syntax viewer, writers can easily see an approximation of the extracted JSON from their documentation. However, the connection between what's displayed in the viewer and what will be displayed in the applications consuming the syntax is still befuddling writers. There is still room for improvement in conveying what particular key/value pairs in the syntax objects will control in the IDE and what those values are meant to indicate. Compounding the confusion is a side effect of a guiding principle to seek to reduce tagging burden on writers. Some information provided in the syntax is inferred from the contents of the syntax documentation, rather than keying off specific tags. For instance, data type of statement arguments is determined in the XSL transformation by a combination of identifying certain tagging structures and regex pattern matching. Lessons learned and practices adopted to address these issues:

Evolving Requirements

Several DTD changes have arisen after the initial DTD was released. On the XML authoring side, there are instances where tagging for one deliverable produces suboptimal artifacts for another. For example, some language elements are documented in multiple projects with differing content. The syntax service needs a single object for each element, so soon after developing a beta version of the service, a new @excludeFromSyntaxExtraction attribute was introduced to control which content is extracted. Conversely, there is concern that there are times when the tagging required to produce accurately extracted content may not be appropriate for documentation, such that an @excludeFromDoc will be warranted, if, for instance, it would make the documentation particularly verbose or redundant. As of yet, we have refrained from introducing this attribute, since it would then be another example of a failure in single-sourcing, requiring the content owners to maintain two sets of documentation of the same syntax. Finally, the team has also added several elements to provide more contextual information to the syntax consumers. @followsDelimiter indicates whether an argument group follows a specified delimiter, and @functionContext specifies in which software packages the descendant function documentation is applicable.

Formatting Syntax and Help Content

The challenge of formatting syntax help is one with which we are still grappling. The legacy XML made copious use of <![CDATA]]> wrapped text to allow for manual formatting of syntax and help content. In the new service, we have resorted to escaped HTML, new-line characters, and spaces to produce syntax examples and readable descriptions. With escaped HTML, the possibility is there for producing fairly well-structured syntax help, but it is far from the ideal solution. Already the service must produce italic text for syntax examples. Further, there are instances where the XSL is now extracting and producing escaped unordered lists in the help field. There are almost assuredly other edge cases yet uncovered by the proofing and revision process that may necessitate other formatting acrobatics. Here lies an underlying shortcoming with JSON as the de facto API standard for information exchange.

LaTeX Doc

Not all documentation at SAS is XML. A significant portion of the content documenting statistical procedures and tooling is authored, not by technical writers and editors, but by the developers themselves in LaTeX. This content follows its own production pipeline to be delivered along side XML-authored content in the customer-facing documentation. This LaTeX doc contains syntax information necessary to code completion, but, unfortunately, is not conducive to extraction and delivery in the syntax service. Its markup simply does not have the granularity or semantic information to produce content for the syntax service.

With no good options, the syntax team decided to use the legacy R&D-authored code completion syntax XML, convert it to the document.dtd, and store it in a project that will serve as the source for syntax extraction. By doing so, the documentation group takes ownership of the the content in the syntax service representing LaTeX-authored documentation, and its writers and editors can use SAS XML tooling to do maintain it. This solution is far from ideal and does not meet all the goals set out for the project. This XML is presently only used as the source for syntax extraction; thus, it represents duplicated content, as the LaTeX documentation continues to be the source for all other customer-facing documentation. Thus, this documentation requires dual maintenance, and breaks the direct connection between documentation revision and software or language behavior development. As of today, there is no regular revision schedule, nor any designated owner for the content. It may only be updated when defects are identified.

Further, since the XML that generates the syntax help for LaTeX documented language elements is not the source of the end-user documentation, establishing a connection between the syntax help and the user documentation was a project unto itself. The team eventually resorted to parsing other XML documentation that contains maintained links to the LaTeX doc and extracting those links to deliver in the service.