Balisage Paper: SGML in the Age of XML

Introduction

This year we celebrate the 30th anniversary of Standard Generalized Markup Language (SGML) becoming an international standard. Many reading this paper may never have heard of SGML or the role it played in the acceptance and success of the World Wide Web (WWW). In some cases it has also revolutionized the publishing of information by providing an easy way to output complex information to different media channels.

SGML via HTML allowed for the first time for Internet browsers the capability to display and publish information on the WWW. In December 1990, the Global Hypertext Project at CERN, European Laboratory for Particle Physics under the direction of Tim Berners Lee provided the capability of displaying and linking information across the internet. At this point in time the internet was mainly available to educational and government organizations. The Global Hypertext Project project developed a very simple SGML vocabulary for presenting and transporting information across the internet. Many working in the SGML space knew the flexibility and usefulness of SGML but HTML actually proved to the naysayers that there was both intellectual and monetary value in using markup to describe information. HTML became and still is the largest use of SGML (some nameless organizations dispute this fact but they haven't been able to prove it yet).

There were many pitfalls along the way. The U.S. Department of Defense was one of the first adopters of SGML for technical publications. This resulted in positive movement to adopting SGML by other organizations in manufacturing, data warehouses, publishing, etc. The negative side of the DOD jumping in so early was that they poured massive amounts of money into companies (in many cases traditional DoD contractors) to develop software applications to support DoD. This resulted in many of the early SGML software applications (authoring, databases, publishing, etc) were too costly for small and medium-sized organizations to adequately leverage the power of SGML.

With the success of HTML, visionaries saw that SGML really could be affordable to the masses. SGML could be used by small and medium organizations to manage and disseminate their information.

SGML wasn't without it's problems. XML was designed to alleviate some of the inherent problems and pain points that SGML had. XML was originally designed to be a subset of SGML. Some of these problems of SGML, either real or perceived were:

XML became a W3C standard in December 1998. Organizations quickly jumped on board and adopted XML for their data. Organizations that had originally adopted SGML were slower to switch to XML but as software applications improved and became more affordable than SGML software, as well as the declining SGML tool market, they also moved to XML. An educated guess would say that 98-99% of all organizations are using markup are using XML if they are using Markup for their data. Today, even popular software such as Microsoft Word uses XML for it's underlying data. Open a Microsoft .docx or .xlsx file in an zip application and take a peak inside - all the underlying data is XML.

However, there are a few organizations, mainly DoD who have not adopted XML and have stuck with SGML almost 20 years later. This paper is designed to help organizations navigate the necessity of delivering SGML in an XML world.

Authoring SGML Content

There are a few editors that still support SGML authoring. These editors originally started in the SGML world and still support SGML authoring. However, if you try researching their literature there is very little information about their SGML authoring capability. Each of these authoring tools support full SGML editing ,as well as support authoring using native SGML DTD's. These editors are:

If you do have the need to deliver SGML the easiest and most efficient way is to use one of the editors that support both SGML and XML authoring.

The same SGML editors can be used to create XML files. The above editors also allow you to save the file as XML or SGML. The big differences between an XML and SGML instance are:

<!DOCTYPE poem SYSTEM "poem.dtd"[
<!ENTITY author SYSTEM "poepic.jpg"  NDATA jpg>
]>
<poem id="poem1">
	<title>The Raven</title>
	<poet>Edgar Allan Poe</poet>
       <author-picture src="author"/>
	<stanza id="stanza1">
		<line>Once upon a midnight deary, while I pondered, weak and weary,</line>
		<line>Over many a quaint and curious volume of forgotten lore-</line>
		<line>While I nodded, nearly napping, suddenly there came a tapping</line>
		<line>As of some one gently rapping, rapping at my chamber door.</line>
		<line>"‘Tis some visitor," I muttered, "tapping at my chamber door-</line>
		<line>Only this and nothing more."</line>
	</stanza>
...	

</poem>

Authoring in Native XML Editor

If you already have an XML editor in-house and prefer using your favorite editor this can be accomplished. You just need to be aware of the slight differences in the SGML/XML editor.

Native SGML Publishing and Specifications

Early SGML publishing was accomplished using proprietary publishing systems. These systems were very expensive. Several of these publishing systems are still available and are still quite costly. In the late 1980's DoD started developing a specification for publishing SGML. The specification was called Formatting Output Specification Instance (FOSI).

The specification MIL-PRF-28001 (MARKUP REQUIREMENTS AND GENERIC STYLE SPECIFICATION FOR EXCHANGE OF TEXT AND ITS PRESENTATION) was originally published in 1992.  The last printing was MIL-PRF-280001C which was published in 1997.  MIL-PRF-28001 specified the use of SGML for all new technical manuals within DoD.  Each branch of DoD (Army, Navy, Air Force) developed DTD’s for use within their individual organizations based on their specific requirements. These DTDs adhered to the requirements in MIL-PRF-28001. 

In addition to specifying the SGML constructs for developing DTD’s, MIL-PRF-28001 provided a DTD and specification for applying styling to the SGML.  The styling specification was called Formatting Output Specification Instance (FOSI).  Appendix B of the specification contained the DTD that supported the use of FOSI for presentation.  Several SGML vendors were part of the working group and worked toward developing a FOSI-based publishing system.  Two vendors DataLogic and Arbortext successfully developed FOSI-based formatting within their product.  However, the implementation was slightly different in both systems based on different interpretations of the specification and ambiguity of the DTD.  The result was that a FOSI developed for one system could not be used in the other system.

FOSI's are still used today by DataLogic and Arbortext. Arbortext has slowly tried to replace the FOSI with their own style specification called Styler. Styler uses both FOSI constructs and its own style constructs.

SGML and Loose-Leaf Publishing

When SGML was a new standard, large publishing environments required loose-leaf publishing. In the 'olden days' large organizations published administrative and technical manuals in paper. When modifications to the manuals were made only the pages that were modified were printed and sent to the users. A manifest was sent with the 'change pages' which told the manual administrators or librarians which pages to remove and add to the paper document. The manifests were often printed on blue paper and the manifest and change pages were called 'blue pages'. Many organizations still used these manifests in their XML, as well as SGML publishing and still call them 'blue pages'. An example of a manifest document is available at the Patent and Trademark Office

Figure 1: Change Pages Manifest Example

The users would remove old pages and insert new pages into binders. If pages ran longer than the original page then the page numbers reflected the new page with a different numbering sequence. For example, if page 1200-2 is modified and the revised page resulted in running over to the next page the users would receive page 1200-2 and page 1200-2a.

Figure 2: Page Numbering

SGML publishing systems that supported loose-leaf publishing compose the SGML first and places processing instructions at the point of a page break in the SGML. When revision elements are placed in the SGML, the publishing system makes a second run through the document to determine where page breaks occur and then calculates the correct page breaks and page numbers.

Modern technology has negated the need for loose-leaf publishing because manuals can be disseminated in total without the necessity to print and disseminate single paper pages. Most of today's workforce are used to on-line, PDF or e-book technology and prefer electronic dissemination of information to paper.

However, there are still pockets of organizations who are stuck in the 1970's and still require loose-leaf publishing capability. Therefore suppliers to these organizations have a requirement to provide information in antiquated paper pages which includes change pages.

If you are required to support loose-leaf publishing, it can still be done with XML but will require out-of the box thinking and additional processes in order to emulate loose-leaf publishing. XSL-FO does not support loose-leaf publishing.

Creating Published Documents from XML

There are several possibilities for creating PDF output from the native SGML. As previously discussed you can use one of the tools that support the SGML style specifications. There are also proprietary publishing systems that support SGML. For most organizations proprietary publishing software is cost inhibitive. Organizations who can't afford the software and/or the expertise to develop the stylesheets will use a 3rd party composition company, which can also be expensive.

However XSLT and XSL-FO are obvious choices for creating PDF and printable files. As stated previously converting the SGML files to XML or vise-versa is relatively trivial. XSLT is a relatively easy skillset to obtain internally or externally. XSL-FO expertise is a little harder to obtain but should be easy to either train individuals internal or obtain outside help.

There are also standard XML vocabularies that have standard stylesheets available. Docbook and DITA are two that are commonly used. If the presentation of the SGML is relatively straight-forward it might be worthwhile to convert the document to Docbook or DITA and modify the stylesheets that come as part of these specifications.

DITA would be the best choice for DoD technical manuals, however extensive modification of the stylesheets would be required and wouldn't be the easiest approach. Some DoD contractors have been able to negotiate with their DoD customers to either deliver a complete PDF book as a 'new book' without the loose-leaf publishing requirement. In this case XSL-FO is used to create the PDF of the technical manual.

Others have negotiated to supply the SGML to the DoD customer as well as an HTML rendition of the document [IETM]. In this case the DoD facility has the capability to publish the SGML in-house using the JCALS (Joint Computer Aided Logistics Support) system that is still have available. JCALS was a joint program with the Army, Navy and Air Force that developed a publishing system that included custom and proprietary software in the mid-1990's. Arbortext Editor is the editor and Datalogics DL Composer is the composition software.

Conclusion

In conclusion, 30 years after SGML became an international standard it is still being created and used. If you and/or your organization find yourself in a position where you need to deliver SGML documents, it is still possible. I will take careful thought to develop the document constructs and the workflow. Hopefully, in the not too distant future, organizations will eventually move from SGML to XML.