Balisage Paper: CodeUp: Marking up Programming Languages and the winding road to an XML Syntax.

What is 'Text'

For the purposes of this discussion I will define 'Text' as the 'Written representation of natural languages'. This is a narrow definition, but it is what we tend to mean when talking about 'Marking up Text'. Furthermore I am jumping forward a few centuries and narrow the meaning further to mean 'The machine readable digital encoding of written representation of natural human languages' aka Bytes, Characters, Codepoints, Words, Sentences, Paragraphs in a computer representation.

So now that we know what 'Text' is ... what does that imply ? Humans don't natively encode their thoughts and language directly (aka telepathically) into computers. They speak or write or type onto some media which is then encoded or transcribed into a computer. That very process changes the original and loses information. In speech, intonation and body expressions are lost when transcribed. In writing the media and physical representations (style, color, size, and emphasis) is lost when encoded to computer media. Typing via a keyboard generally limits the representation to uniform characters. Even before computers this was so natural to humans that the expression of language can affect it's meaning that different language is used when speaking and writing. Written (and printed) language contains information with the intent of adding back things missing from the spoken word in the form of style, font, size, spacing, as well usually a clearer and more precise use of language.

Unlike interactive communication, the nature of Text (and recorded speech) extracts itself from time and space. Text is read, it is usually at a different moment in time and a different place in space from when it was authored. Since language requires a shared understanding, much is inherently lost in text. The reader has indirect access at best to the environment or context of the original event. Some of this context is recreated in the text itself through dialog, prose, notes and contextual references, but requires the author to imagine what context may be lost and provide a substitute. This is one reason why ancient texts are more difficult to interpret than modern texts; the author was unable to predict and accommodate for the greater difference in context of the modern reader.

What is the purpose of Text ?

The purpose of Text, as a representation of natural language, is to express and ideally communicate thought. Expressing thought implies an Author. Communicating thought implies an Audience. We were taught to "Consider the audience" when writing, but that is a difficult and often impossible goal. At best we may consider the Intended Audience, but who is to say who may want to read our text in the future and where or when that might be? Since we can't embed our entire context in every writing there is always an assumption as to the shared understanding of the intended audience.

Why mark up text?

Why do we mark up text? Especially digital text and when the markup itself is text, why not just add to the text? Marking up text is essentially just adding more text. But it's a different sort, it is 'out of band'. It represents a different dimension or layer from the text itself. Why do we do this ?

I asked this question to several groups in the hopes of getting opinions I had not previously considered, or at least to clarify my preconceptions. As expected the answers varied in detail greatly by the person. Obviously (in hindsight) people mark up text for different specific purposes and express these in different degrees of abstraction.

Some example responses indicating a range of reasoning, edited and out of context for brevity:

Distilling these responses with my own reasoning I propose the following concepts for why we mark up text. The reasons vary by author, context, and purpose. Some reasons are complementary and some are contradictory. Yet still we find markup useful.

It is relevant to note that many of these reasons only require markup because software to date is not smart enough to actually parse and understand natural languages well enough. With human consumers, the semantics of text representing "Title", "Paragraph", "sentence" "emphasis" are usually quite obvious to the human reader. So are generally the more linguistic attributes such as verb, noun, and phonetics. Even the context is often known implicitly by the reader, say when reading a book by Charles Dickens one may have a reasonably accurate internal model of the Victorian period by which to frame the experience.

Thus much of the value of markup is to encode these things which the primary consumer (humans) already knows to make it easier for computer processing.

What is a Programming Language?

At first glance, Programming Languages seem an altogether different beast than natural Languages. The language is artificial and designed for a single purpose - to instruct a computer on how to accomplish a task. All the information necessary for this task is already encoded in the language - so what need is there for markup?

The hidden assumption in these questions is that Programming Languages (or rather it's textual representation - "The Source Code") have a single target audience - the compiler. But this is blatantly untrue. The truth is that Programming Languages not only have multiple audiences for multiple purposes but also usually multiple authors. So what really is a Programming Language?

An instance of Programming Language is expressed as one or more text documents which comprise "The Source Code". These documents are encoded typically in 'plain text' with a limited range of characters available. The syntax of the programming language is intended to be humanly readable and writable as well as readable by a computer.

What is the purpose of a Programming Language?

A programming language serves multiple purposes. The obvious purpose is to instruct a computer to perform a task. But there are other purposes. A programming language is intended to be written and understood by humans. Similar to a text document, there is an implicit context in which the language resides which may not be necessary for the compiler, but is necessary for humans in authoring, editing, reading and understanding the program.

Different languages recognize and support these purposes to different extents by allowing types of markup as part of the language itself. Comments are the main form, but also common is markup which may be recognized by the compiler or ancillary programs such as pragmas, annotations, and micro-markup within comments such as JavaDoc JavaDoc, HTML HTML etc. The syntax of comments and micro-markup is specific to each language and generally not interoperable. To make things more difficult, languages are often embedded within other languages within the same document. For example a Java Java program which includes HTML that has JavaScript JavaScript , or C C Program with embedded SQL SQL . The syntax rules for what constitutes a comment and where it can be placed can get extremely confusing. Furthermore, most markup defined in programming languages is a point, not a container; e.g. a comment occurs at some point in the text stream but does not explicitly identify it's target (if any) that is to what does the comment refer. It is up to humans and convention to decipher the target, scope, and intent.

Why markup programming languages?

Programming languages need to be marked up for generally the same reason as Text. Source code is a human readable text document that has many of the same characteristics and needs as text for human languages. Reasons for marking up Programming Languages include:

Programming languages have some pre-existing mechanisms for markup which may suffice for some of these needs. Are these sufficient? Is there benefit in learning from the concepts, tools, and techniques of Text markup?

A very significant difference between computer languages and Text is that computer processes can already (and must) explicitly extract the semantic meaning of most language constructs. The main exception being comments which suffer from the same problems as Text, they are not currently readily parsable by computers.

The problem however, is that even though "The Computer" can parse the semantics of a programming language, it greedily hides this information in the bowels of it's internal data structure giving no heed to the concept that maybe there might be additional consumers who could value from this information. Thus much of this information for which Text Markup is required is only needed because Programming Language consumers (parsers, compilers etc.) are not designed for any other purpose.

The problem with Programming Language AS a Markup Language

A common sentiment regarding marking up programming languages is that it is unnecessary because programming languages are themselves a markup language, or at least provide a strict syntax which is computer parsable. There are several problems with this reasoning. A big problem is that the syntax and semantics of programming languages is primarily aimed at one consumer - the compiler. This syntax can be extremely tedious to use for other purposes. In general it requires a complete specific language aware parser to extract the program structure from the syntax. Even extracting comments requires a language aware parser.

For example, consider this Java snippet:

Figure 1

System.out.println("This is /* A comment */ or // is this " + /* A comment */ " or // is this ") ; // 
"A comment"

Or this XQuery statement

Figure 2

           

return <element>
    (: Is this a comment :) { () (: or is this a comment :) } <!-- What about this -->
</element>

            

Reliably extracting out the comments from the code is a very tedious task. Furthermore, it is language specific. Every programming language has it's own rules and quirks. Of the same can be said of all languages including XML, however programming languages tend to have non regular structures that are particularly difficult to parse.

But suppose we could easily parse and extract comments, and identify where they were in the code text - what do they mean? By their very nature comments have no explicit semantics. They are just plain text. The traditional way of handling this is to add a kind of micro-format or embedded markup within comments used by particular tools. For example JavaDoc uses a set of special tags as well as HTML to mark up portions of the text within comments. This becomes a very complex 'Russian Doll' layering of different markup languages embedded within each other.^[1] Even then, the target audience of JavaDoc is limited. It can be parsed by very few tools and the output is limited to few uses. To target other uses we need different markup - and special tools to parse that.

Figure 3: JavaDoc Example

Nested markup in JavaDoc

From Saxon 9.4 source.

Imagine we decided on a good markup for embedding in comments, there is a fundamental limitation imposed by the syntax - they are missing a target. Comments appear at a single point in the text stream. There is nothing explicit that indicates to what they refer. There are conventions that are generally humanly understandable, such as placing a comment preceding a function means it applies to that function, but the conventions are easily broken.

Consider this simple Java snippet

Figure 4

	            
/*
 * This file implements the PixieDust portion of Magic Products.
 */
 class PixieDust {
 
 // The pixie function
 private int magic = 0;     // Magic 
 public int pixie( int dust )
 { 
       /*
       * Be careful that magic is not set somewhere else
       */
      return ((magic < 10) ? 0 : magic)   + dust ;  // Conditionally add dust
 } 
} 

To a human it is fairly clear which comment applies to what. One generally follows the convention that block comments precede their target and inline comments follow their target. But from a parsers perspective it's entirely unclear what comment is associated with which language construct, if any. Note, for example, the comment "The pixie function" immediately precedes the variable declaration for "magic" yet is obvious to human readers that it refers to the function declaration instead. The comment "Be careful that magic is not set somewhere else" doesn't target the following expression exactly, it's a more out of bands annotation concerning the algorithm. We could of course start adding markup in the comments to clarify to what they refer.

              
/*
 *  <file name="pixiedust.java">This file implements the PixieDust portion of Magic Products.</file>
 */
 class PixieDust {
 
 
  // <function name="pixie">The pixie function</function>

  private int magic = 0;     // <description variable="magic">Magic </description>
  int pixie( int dust )
  { 
       /* <annotation>
       * Be careful that magic is not set somewhere else
         </annotation>
       */
      return ((magic < 10) ? 0 : x)   + dust ;  // Add magic to dust if >= 10 ;  // 
      <explanation reference="((magic < 10) ? 0 : magic)">Conditionally add dust</explanation>
      
  }
}
	        

This is starting to look pretty crazy. Unfortunately this is the essence of what implementations of embedded markup in programming languages end up doing. Furthermore this is still an extremely fragile binding. Without a full language parser available, the comment markup processor has no way of identifying and validating the targets. For example, if I wanted to query "Give me the code for function 'pixie'" the above markup would not be able to answer that.

We could go further and try to express the more of the structure in comment markup using containment instead of positioning and named references.

Figure 5

 
	        
/*
 *  <file name="pixiedust.java">
 *  <description>This file implements the PixieDust portion of Magic Products.<description>
 */
 class PixieDust {
 // <variable name="magic">
  int magic = 0;     // <description>Magic</description>
 // </variable>  
 // <function name="pixie">
 // <description>The pixie function</description>

 int pixie( int dust )
 { 
       /* <annotation>
       * Be careful that magic is not set somewhere else
         </annotation>
       */
        // <explanation>
        
       return ((magic < 10) ? 0 : magic)   + dust ; //  <description>Conditionally add dust</description>
      // </explanation>
 }
 //< /function>
 //</file>

Now we approach a markup, embedded in comments, that has more semantics. it's also redundant and extremely difficult to read, edit, maintain and validate. it's likely that for these reasons markup in programming languages is general limited to embedding documentation. When simply using markup for documentation (as opposed to the above structure) the markup is "close to" the code but it really doesn't markup the code. This is very useful as documentation that is close to the code is generally more likely to be updated and kept in sync but it's really just a way of managing documentation and adds little if any value to the code itself

Pragmas and Annotations

Another form of programming language markup is Pragmas and Annotations. These are designed to be part of the language itself and thus have semantics as opposed to comments. Java is a good example with a quite expressive annotation capability. The problem with annotations and pragmas however lies specifically with their coupling to the language. Not only are they language specific but have predefined or proprietary semantics and are part of the language itself, rather than meta-data about the language. Thus by adding pragmas or annotations you are changing the program itself; this violates many of the goals of markup in that it changes the underlying semantics.

It is possible that programming languages could be extended to allow for more structured documentation or markup such as annotations that does not affect the semantics of the program in any way. However this would still require a full language parser and would inherently be language dependent.

A journey through Literate Programming

A discussion about marking up programming languages is not complete without consideration of Literate Programming LiterateProg.

Literate Programming, as introduced by Donald Knuth, is a very interesting concept. The idea is that commentary (prose) is 'interwoven' with code and that a post processing phase separates the two into either documentation or source code. This has some overlap with the concept of marking up programming language but I feel it is a quite different path.

The perspective I propose is that source code be treated more like Historic or Primary documents. That is that the source code is the primary information and that markup is annotation or meta-data applied to document.

This philosophical distinction has practical implications. With Literate Programming the source code is "tangled". That is snippets of source code are interspersed throughout the prose and not necessarily even in the same order as the final resulting program. This has the advantages of providing an emphasis on providing the prose as the primary source so let's the reader understand the exposition in a more customary way, leaving the actual implementation (the code) as almost a side thought.

This is useful for describing the thought processes of design but I argue that it is problematic for the actual engineering of software. The fact is that software source code (or rather, compilers) is very particular about the exact representation. It takes significant effort to produce source code that is parsed and works exactly as intended. Much of this difficulty is in fact the sort of things literate programming attempts to hide such as ordering and placement of statements, code blocks, and compilation units. I suggest that one reason Literate Programming has not achieved wide adoption is that it makes the everyday programmers task more difficult. Curiously, similar statements have been made about C++ for example. C++ is an excellent language for hiding the trivial and mundane details of a program... like where the bugs are.unknown I belie tangling already difficult to understand code within prose makes the job of both authoring and debugging the software harder, not easier.

With these thoughts in mind I suggest diverging from the Literate Programming perspective and instead look to the source code as the primary information, not something to be hidden within prose, but rather annotated.

Transparent Markup

A critical requirement for source code markup is that the "original" text needs to be preserved. Similar to Text Documents, "original" may be a temporal quality (i.e. there existed prior in time an "original" document which is later marked up). But a similar case is to consider "original" to mean "what would the text be if there was no markup", regardless of which was authored first or authored at the same time. In both cases the requirements are the same: the "original" must be unchanged by markup and be able to be extracted from a marked up document. The simplest case of this is the use of "transparent markup". This is a premise in many text oriented schemas, that is the concept that if you remove all markup what is left is the "original".

Applying that to CodeUp we can start with the first step, a root element.

	            
<program xmlns="http://xml.calldei.com/codeup">
/*
* This file implements the Pixie portion of PixieDust Products.
*/

// The pixie function
int x = 0;     // Magic 
int pixie( int dust )
{ 
/*
* Be careful that x is not set somewhere else
*/
return ((x &lt; 10) ? 0 : x)   + dust;  // Conditionally Add dust

}
</program>
 

This simple step is easy to apply, and in fact could be programmatically applied to a set of source code. There is a complication that the text of the code itself may overlap with XML markup; it needs to be escaped with entity escapes or CDATA sections, but that is fairly easy to accomplish. CDATA markers can easily be added by hand to the beginning and end of the file. A simple text parser or an XML enabled GUI tool can accomplish entity encoding easily and risk free. The direct cost of adding this markup is practically zero. However it exposes the indirect costs: the entire development and compilation ecosystem is now broken. In addition if simple hand entering or editing of the source code is done care must be taken to escape what was source code but in the context of a marked up document would be misinterpreted as markup. More complex escaping may need to be done if the code contained XML-like markup to begin with such as JavaDoc with embedded HTML, especially if the contained markup itself has CDATA sections or entity encoding.

	            
<?XML version="1.0" encoding="UTF-8"?>
<XSL:stylesheet xmlns:XSL="http://www.w3.org/1999/XSL/Transform" version="2.0">
<XSL:output method="text"/>
<XSL:template match="*">
    <XSL:value-of select="."/>
</XSL:template>
</XSL:stylesheet>

The rocky road from A Text Syntax to an XML Syntax

Once we've accomplished the initial step of converting a programing language to an XML Document, and all that entails including editors, IDE's, tool-chains and work-flows we can start to explore the benefits and costs of enriching the markup.

Continuing under the premise of "Transparent Markup" what can be improved while maintaining the original text as source code? Much of this can be done as language neutral markup because it is entirely optional. Strip out the markup and you get the source code. Markup need only be informative or helpful but need not be semantically perfect.

Imagine the following 'improvements' to the original sample.

	            
<program xmlns="http://xml.calldei.com/codeup" language="Java" file="pixiedust.java">
    <comment scope="file">
    /*
    * This file implements the PixieDust portion of Magic Products.
    */
    </comment>
    <class>  
    class PixieDust {
    
    <comment inline="true">// The pixie function</comment>
    private int magic = 0;     // Magic 
    <method> 
    public int pixie( int dust )
    { 
      <comment>
      /*
      * Be careful that magic is not set somewhere else
      */
      </comment>
      return ((magic &lt; 10 ) ? 0 : magic)   + dust ;  <comment inline="true">// Conditionally 
      add dust</comment>
    
    } 
    </method>
    </class>
}
</program>

This begins to add the sort of semantics common in text markup while still being transparent. The simple XSLT still produces the original source code by stripping all markup, and using an XML IDE starts to provide value in terms of authoring

For example the following is the Author view in Oxygen Oxygen ^[2]with a very simple CSS applied to the schema

Figure 9: Oxygen Author mode view of the above source.

The Dead End of Transparent Markup

The previous attempt has some valuable markup but still lacks significant semantic value. For example the question "what is the method called 'pixie'" or "To which language element is a comment referring" cannot be answered with only this XML markup. Suppose we go further and embed more of the language semantics within the markup, while still staying with the goal of Transparent markup. Something like this is an obvious next step.

Figure 10

	            
<program xmlns="http://xml.calldei.com/codeup" language="Java" file="pixiedust.java">
    <comment scope="file">
    /*
    * This file implements the PixieDust portion of Magic Products.
    */
    </comment>
    <class name="PixieDust">    
    class PixieDust {
    
   <comment inline="true">// The pixie function</comment>
   
   <variable scope="private" name="magic" type="int" initializer="0">
    private int magic = 0;     // Magic 
   </variable>
   
    <method name="pixie" scope="public" return="int">
    <parameter name="dust" type="int"/>
    public int pixie( int dust )
    { 

    <comment>
    /*
    * Be careful that magic is not set somewhere else
    */
    </comment>
        <statement type="return" expression="((magic &lt; 10 ) ? 0 : magic)   + dust">
        return ((magic &lt; 10 ) ? 0 : magic)   + dust ;  <comment inline="true">// Conditionally 
        add dust</comment>
    </statement>
    } 

    </method></class>
    }
</program>

With a little CSS magic Oxygen Author mode displays the additional semantics including outlining and collapsing. Using the XSLT in Figure 7 the result is the original text.

Figure 11: Oxygen Author mode view of the above source.

At this point we have captured a significant amount of the language semantics entirely in XML Markup and can indeed perform some semantic queries such as "What is the body of the pixie function" or "What are the method names, parameters and return values in the pixiedust.java file". We could use this information to achieve many of the goals of Markup such as production of different representations, limited semantic queries, publication, aggregation even code refactoring - all using XML tools only.

Unfortunately to achieve this we have suffered some of the main problems of embedded markup: that of duplication of information, fragility and difficulty authoring, editing and viewing. Also the markup does not embed the full semantics of the language. For example call graphs or object dependencies cannot be generated because the semantics of expressions is not modeled. It seems we have reached a dead-end as it's unlikely software authors are willing to put the effort into this level of markup considering that it's really duplicating the code in two different ways serving the author little gain and adding great effort. Furthermore the chances of the markup being valid over the life-cycle of the program are slim. It would take a very dedicated author to make sure that the entire markup is updated as code is updated and there is no automatic way of validating it without comparing the results of a full language parser against the markup. So in reality the value is negative and the cost is high.

Semi-transparent Markup

Considering the dead end attempting pure transparent markup, let's back up and reconsider the original goals. When we wanted to maintain the integrity of the "original text" of the programming language, perhaps the goal can be achieved by relaxing the concept of "original". Instead, considering we have already added a process of transformation to extract the "original", perhaps it's sufficient that the transformation itself can produce some of the text as a side effect of markup instead of requiring that all output be literal text. As long as the results are identical to the intended original we can consider it semantically equivalent.

For purposes of this discussion I will call this markup style "Semi Transparent". That is, a combination of markup and plain text, when transformed with a known transformation is able to produce what would be the "original" text if we did not use markup.

Consider this simple fragment

class PixieDust { body } 
 

One could represent this using markup but without duplication by something like

<class name="PixieDust">
body
</class>

These two are semantically equivalent. Using a simple transformation the Intended Text could easily be produced but without duplication of data.

Expanding on this let's see what the full sample might look like using a semi-transparent markup.

Figure 12

<program xmlns="http://xml.calldei.com/codeup" language="Java" file="pixiedust.java">
   <comment scope="file">
   * This file implements the PixieDust portion of Magic Products.
   </comment>
   <class name="PixieDust">    
   <comment inline="true">The pixie function</comment>
        
   <variable scope="private" name="magic" type="int" initializer="0"/>
   <method name="pixie" scope="public" return="int">
    <parameter name="dust" type="int"/>
    <comment>
    * Be careful that magic is not set somewhere else
    </comment>
        <statement>
            return ((magic &lt; 10 ) ? 0 : magic)   + dust ;  <comment inline="true">Conditionally 
            add dust</comment>
        </statement>
   </method></class>
</program>
 
  

With some more CSS magic Oxygen Author mode displays the additional semantics.

Figure 13: Oxygen Author mode view of the above source.

A slightly more complex XSLT is required to produce the Intended Text

Figure 14

<?XML version="1.0" encoding="UTF-8"?>
<XSL:stylesheet xmlns:codeup="http://xml.calldei.com/codeup" 
                xmlns:XSL="http://www.w3.org/1999/XSL/Transform" 
                version="2.0">
    <XSL:output method="text"/>

    <XSL:template match="codeup:comment">
        <XSL:if test="@inline"><XSL:text>// </XSL:text></XSL:if>
        <XSL:if test="not(@inline)"><XSL:text>/*</XSL:text></XSL:if>
        <XSL:value-of select="text()"/> 
        <XSL:if test="not(@inline)"><XSL:text>*/</XSL:text></XSL:if>
    </XSL:template>
    
    <XSL:template match="codeup:variable">
        <XSL:value-of select="@scope"/>
        <XSL:text>  </XSL:text>
        <XSL:value-of select="@name"/>
        <XSL:text>  </XSL:text>
        <XSL:value-of select="@type"/>
        <XSL:text>  = </XSL:text>
        <XSL:value-of select="@initializer"/>
         <XSL:text> ;</XSL:text> 
     </XSL:template>
    
    <XSL:template match="codeup:class">
        <XSL:value-of select="@scope"/>
        <XSL:text> class </XSL:text>
        <XSL:value-of select="@name"/>
        <XSL:text>{</XSL:text> 
        <XSL:apply-templates select="node() "/>
        <XSL:text>}</XSL:text> 
    </XSL:template>
        
    <XSL:template match="codeup:method">
        <XSL:value-of select="@scope"/>
        <XSL:text> </XSL:text>
        <XSL:value-of select="@return"/> 
        <XSL:text> </XSL:text>
        <XSL:value-of select="@name"/>
        <XSL:text>(</XSL:text>
        <XSL:apply-templates select="parameters"/>
        <XSL:text>){</XSL:text>
        <XSL:apply-templates select="node() except element(parameters)"/>
        <XSL:text>}&#10;</XSL:text>
    </XSL:template>

</XSL:stylesheet>
 
  

Further down the path of an XML Syntax

There is much further we can take this exploration. For examples the comments are still unreferenced; they appear inline with code but no clear semantic reference to which they refer. That could be solved with cross reference tags. Statements and expressions are yet to be fully decomposed. The full syntax of statements and expressions could be marked up. If we keep on this path we will end up where XSLT and ultimately XQueryX have. A fully semantic markup that in this author's opinion is impractical for human authoring and consumption.

There is a reason that programming languages are largely expressed using more unstructured human readable syntax. That is that humans are ultimately the main creators and consumers of the language. Even though a computer has to interpret the language, so it must be quite explicit and unambiguous, humans have to write, read, maintain, and debug it.

That doesn't mean there isn't room for improvement. As we have seen with markup such as JavaDoc there is still a strong desire to add layers of semantics to programming languages. A universal markup for programming languages may well be a practical and useful tool. And possibly some compromise between no markup and fully semantically marked up might be a 'sweet spot' for some programming languages.

The Golden Road (to Unlimited Devotion)

Like the song of the same name GoldenRoad, applying markup to Programming languages to it's full conclusion may require unlimited devotion. Even if every single statement, expression and symbol was marked up the resulting information would not include all the information that a compiler understands by parsing the plain text due to the compiler understanding the semantics of the entire corpus. This means even if we follow the Golden Road to it's end we wont end up with the information a language specific parser does. So in a way this seems like a dead end road. What can markup gain us?

With Text markup, the data added by markup supplies us with generally not available with the original text. With Programming Languages, the information contained in the program statements themselves is ultimately available, even if deeply hidden in the bowls of the compiler. Perhaps it is best if that in information be pulled from the compiler's grasp instead of duplicated by markup. But what of information the compiler discards, is unaware, or is unimportant to the compiler but useful to humans ? That may be a realm ripe for markup.

So what information in software and it's incarnation in programming language text not available or not of interest to the compiler? Clearly comments and documentation come to mind. But there is more

The very tedium of engineering required to construct a good program, and which the compiler assumes but generally not part of the compilers undemanding, or discarded as part of the compilation may be ripe for markup: information such as the structure of a complex system. What pieces need to go together and what is irrelevant (and may be changed). What is the rationale behind seemingly arbitrary choices such as naming conventions, compilation units and program source structure. For example in version of troff Troff which I was fortunate enough to port in the mid 80's from Unix to VMS, the C Source was written (originally ported from PDP11 assembly) assuming that global variables were aligned in memory in the order declared. This was implicit but required by the program and broke in exiting ways on VMS where the C compiler ordered variables alphabetically not by declaration order. Perhaps Markup may have indicated this requirement?

Sometimes certain functions, variables and comments should belong together. Markup could indicate this where a compiler does not. Functions describing a public and private interface could be indicated in languages where the compiler does not provide such a facility. There are many areas where the compiler is ignorant but the programmer has important knowledge. Markup could be effectively used for these situations.

Markup for high value and low cost

Examining what types of markup are required to encode what semantic information leads one wondering "is it worth it?". There are some goals with moderate value that can be achieved at low cost, but much which can be achieved only with very high cost if at all, and the value is debatable.

Presuming you consider the retooling necessary for wrapping programming languages within markup (inversion) a fixed sunk cost, what goals can be achieved with minimal additional work?

Semantic Analysis

The really high value possibilities lie in the area of semantic analysis. For example call graphs, dependency checkers, flow control graphs, algorithmic analysis etc. These are areas that could be of extremely high value. Unfortunately they have high cost in terms of both markup and execution. To do something as simple as a call graph, all expressions and identifiers would need to be marked up. In addition full language semantics would need to be understood in order to resolve overloads, virtual functions, namespaces, indirect calls etc. It is not sufficient to simply mark up every bit of the program; a semantic analysis requires full understanding of the language itself.

Essentially the entire semantics of the program would need to be marked up and available to analysis. To achieve this would require markup as verbose complex as XQueryX Figure 16 as well as a knowledge engine that understands the meaning of the language constructs.

Experience has shown that people do not want to author in this level of markup. The history of mathematics and programming languages has generally been towards concise and humanly readable syntax. In fact many of the 'pure XML' syntax languages like XSLT and XProcXProc have active groups attempting to reduce the markup in the language providing for an alternate syntax which does not require the verbosity of XML markup. Furthermore describing the information inferred by a compilation of programming languages is extremely difficult in pure XML vocabularies. So even if the syntax was available to XML tools as fully marked up software, the semantics the representation is difficult.

Standardized Transparent Markup

A standardized schema or family of schemas for transparent markup should be considered. If a standard or convention is developed this may encourage tools makers to adopt the concept into IDE's and other components of the software lifecycle. It should be a fairly easy task to accept a form of transparent markup - even if only to completely ignore it. Simply allowing software to be marked up without interfering with the toolchain opens it to more diverse purposes. If built into IDE's then problems with escaping special markup characters could be minimized. The IDE itself may be able to add some markup based on out of band information such as project structure. Rich text editing support could be added for sections which allow it such as currently supported in JavaDoc. The co-existence of transparent markup with the entire toolchain of software development should be a fairly low cost and high value proposal.

Standardized In-Comment markup

If transparent markup is too high a cost, then consider standardiazing in-comment markup similar to JavaDoc. I feel that in todays cross language environments that a language neutral method of marking up languages is valuable. While it is difficult to get perfect parsing/extraction of comments from programming languages, it is not that difficult to get "usually works" extraction of comments. For example any line that starts with only punction characters then includes say "@@" could indicate a markup tag.

   // @@Tag  A C or C++ or Java inline comment tag
   /*
    * @@Tag A C or C++ or Java block comment tag
    */
   (: @@Tag An XQuery comment Tag :)
   # A Shell or Perl comment tag
   <!-- @@Tag An XML comment tag -->
 
    

The choice of @@ is largely arbitrary. Any similar token that is reasonably parsable within language comments could be used, including plain XML or perhaps a infrequently used PI like <?CODEUP?> . While this is not as technically desirable as pure transparent markup - because of the chance of mis-parsing actual code as markup, the cost is significantly lower as it would not impact existing software lifecycle (compilers, editors etc.) so may be a pragmatic compromise.

Standardized compiler data structure outputs

Compilers should not be black holes for information. A standard or convention for representing the information consumed and produced by compilers should be considered. If there was a common data structure defined for representing the semantic information derived as a side effect of compilation then compilers could output this as part of the compilation process and expose this rich source of data for other uses. Designing a structure or schema which can be easily produced by a variety of compilers as well as usefully consumed by other processes may be a challenging task, but I suggest it is an avenue that could lead to great rewards.

Put it all together

In conclusion I propose that a mixture of additional annotation or markup on programming languages, in combination with more visibility to the byproducts of compilation together could expose great previously untapped value. More work is necessary to investigate and implement real world uses for this hidden knowledge but the potential is large. Lets get to work !