Balisage Paper: Invisible Fish: API Experimentation with InvisibleXML

Consistency sets expectations

Even where function syntax is determined by the programming language, the syntax of the function and parameter names is not. The ordering of parameters is a choice. Even in cases where parameters are identified by name rather than ordering, the order used in examples and API documentation is still a choice. The set of parameters and how to expose them is a choice. Each of these choices creates expectations that make the API easier or more difficult to grasp.

Let's look at a simple example.

Figure 3 shows a set of function names. In this case the names are all consistently a target (a noun) followed by an action (a verb). Almost. The morpheme "properties" is not an action at all: it names the result of the action. Still, the symmetric force of the API encourages us to think of it as an action, perhaps an elided form of "get-properties".

document-get()
document-delete()
document-properties()
collection-get()
collection-delete()
collection-properties()
link-get()
link-delete()
link-properties()

What other inferences does this API encourage us to make? We probably assume that "document", "collection", and "link" are all alike in some way, and that "get", "delete", and "properties" are all alike in some way. The API is also telling us that the most important thing is the target. What the functions actually produce is less important. We probably further assume that if we encounter a new relevant target "thingy" and a new relevant action "reckon", that we would find functions named thingy-reckon() and collection-reckon().

In this example we're looking at pure function names, but this is the same approach taken when we have classes named for targets in object oriented programming and methods named for simple actions. This is the same approach taken when creating fine-grained namespaces that name targets and simple action names for functions in that namespace.

document:get()
document:delete()
document:properties()
collection:get()
collection:delete()
collection:properties()
link:get()
link:delete()
link:properties()

What happens if we turn things around: action first?

get-document()
get-collection()
get-link()
delete-document()
delete-collection()
delete-link()
properties-document()
properties-collection()
properties-link()

First, the non-action "properties" starts looking ever more problematic. It really isn't an action, and this formulation makes that gratingly obvious. "get-properties" really would work much better here.

We're still going to assume that "document", "collection", and "link" are all alike in some way, and that "get", "delete", and "properties" (urgh) are all alike in some way, although the mismatch between what we expect to be an action and its English name might make us start to assume that there is something really quite different about the "properties" series of functions. We're going to expect get-thingy() and reckon-collection(). Furthermore, this leads us to think of the actions as the main thing. We may even expect to see generic get(), delete(), and properties() methods that dispatch to these specific ones. This is not at all an expectation from the target-first approach, and indeed with that approach it can be harder to imagine such methods or how to name them.

This is the approach taken when we use namespaces for specific kinds of actions, where the actions operate over different targets.

get:document()
get:collection()
get:link()
delete:document()
delete:collection()
delete:link()
properties:document()
properties:collection()
properties:link()

From a conlang perspective we are collecting vocabulary lists here, and inventing grammatical categories and syntax rules for combining items in these categories. Users of these APIs can use these rules and implicit categories to navigate the API and make predictions about it. Developers of the API can use the rules to make decisions about how to extend it.

Where there are no rules, APIs become less predictable and therefore less usable. People being pattern-finding creatures, they will try to make sense of the mess in front of them, finding faces in the random rocks of Mars.

get-document()
document-delete()
properties-of-document()
collection-get()
delete-collection()
collection-properties()
link()
link-delete()
link-properties()

Folks might assume from the collection of names in Figure 7 that "document", "collection", and "link" are all different. They might assume that "get" is an operation that doesn't apply to "link", that there is something fundamentally different about "properties-of" applied to "document" and "properties" applied to "collection". Perhaps getting properties is a more complex operation for "collection" and "link". What should we predict about "thingy" and "reckon"? thingy-reckon() or reckon-thingy()? Hard to know. As folklore has taught us for millennia: names matter.

People will invent reasons for differences and similarities, regardless of intention. Best make them intentional.

Word order

Let's take another example. Consider what it looks like when you are applying a series of operations one after another. Programming languages tend to conceive of operations as being done by some anonymous and ubiquitous agent (the computer) hence the imperative style. The functional language viewpoint, however, takes the "thing being operated on" (in the classic view) as the agent, the one performing the action. So let's run with that viewpoint and deem that agent as the subject (S), the action as the verb (V), and the operands as the object (O). When we combine a series, one whole operation (e.g. S+V+O) embeds in the next as its subject. What does embedding look like with different word orderings?

Consider this representation in XQuery arrow syntax of a series of operations: first a negative translation, then a scaling by half, and then a positive translation.

$g=>
  transform:translate(-800,-800)=>
  transform:scale(0.5,0.5)=>
  transform:translate(800,800)

This kind of embedding is like English syntax, and for English speakers reads naturally from left to right, telling us what happens in temporal order. It isn't the only way to go about things, however. The conlang perspective tells us to consider the range of possibilities, to look outside our familiar tongue.

Here is the same set of operations, but represented as the value of an SVG transform attribute. The operations apply from the right to the left.^[1]

<g transform="
  translate(800, 800) scale(0.5, 0.5) translate(-800, -800)
"/>

SVG appears to use this ordering to be consistent with the ordering you'd see if the transforms were partitioned into child groups instead. You can add more operations by prepending them to this string or by wrapping the transformed element in a new group with a new transform attribute with the new operator. The transforms apply from the inside out: in text order, from right to left. You can think of this as the "but first" approach: "translate by 800, but first scale by half, but first translate by negative 800". You're saying the most important thing is what happened last, but straining the memory of your readers on figuring out the order of events.

XQuery functional syntax uses center embedding with VSO order.

transform:translate(
  transform:scale(
    transform:translate($g, -800, -800),
    0.5, 0.5
  ),
  800, 800
)

The English gloss of this is something like "translate the scaling of the translation of the item by -800 by half by 800". In natural language, center embedding often leads to comprehension difficulties, stressing human short-term memory as you build up a stack of pending items before getting to the center where you can start to unroll things.

There are other logical possibilities.

SOV ordering is actually more common in languages of the world than English's SVO. With left embedding this is the order of certain kinds of calculators.

$g ⟽ (-800, -800)translate ⟽ (0.5, 0.5)scale ⟽ (800, 800)translate

This ordering tends to emphasize the operands over the operators and creates a sense of closure for each operation. This is also an "about-that, and-then" kind of formulation, something like "take minus 800 and translate by that, and then take half and scale by that, and then take 800 and translate by that".

OVS and OSV orders are quite rare in natural languages, but they do occur.

(800, 800) (0.5, 0.5) (-800, -800) $g translate scale translate

This ordering reminds me a little COBOL data and procedure sections: first all the data you need, then the series of operations. For that matter, a lot of GPU programs look like this too. An English gloss of is the "respectively" construction, something like "take 800, and half, and -800: now translate, scale, and translate by those respectively".

Finally, for completeness' sake, here's what OSV with right embedding looks like, a combination of "but first" and operand-forward:

(800, 800)translate (0.5, 0.5)scale (-800, -800) translate $g

"Take 800 and translate by it, but first take half and scale by it, but first take -800 and scale by it."

Grammatical gender

English is relatively impoverished when it comes to markings for case, mood, tense, person, gender, and so on. Our computer languages need not be similarly impoverished. One way to think about Hungarian notation is that it constructing grammatical gender classes for variables and marking them with gender prefixes. For example the variables named szName, pName, and nName are marked with the affixes sz, p, and n to indicate that they are, respectively, a zero-terminated string, a pointer, and a count. Number (1 versus many) is also marked: aszName for an array of strings as opposed to szName for a singleton.

Other gender and number systems are possible. Sometimes APIs will use their own markers for some idiosyncratic purpose. For example, the Windows' API uses m_ to mark a data member of a class, g_ for a global, cr for a colour reference value, and so forth. These combine, so that m_psz is a pointer (p) to a zero-terminated string (sz) that is a member of the class (m_).^[2]

This kind of thing gets criticized for getting in the way of readability (when the markers are small abbreviations), writability (when they are longer words), and for introducing the possibility of a mismatch between what a name claims and what it actually is and thus impeding refactoring. It is most useful when the naming convention tells you something the compiler or interpreter couldn't already tell you, such as whether a string is safe or unsafe, that is, an internal string or a string coming from input that needs to be sanitized.

Mood

Computer languages tend to operate in an imperative mood, giving commands to the computer, as it were. But let's consider, in the conlang way, the full range of possibilities. Is imperative really the best for a particular situation? Is there really only one mood in play? And if not, how should we indicate the variation?

First, let's look at some of the possibilities (by no means an exhaustive list):

Natural languages don't generally distinguish all these situations in a systematic way, and English typically just uses modal verbs of some sort for most of these. We can analyze what some computer languages do in these terms as well.

Consider SPARQL CONSTRUCT versus ASK. It is in the usual imperative style, but we could also analyze it as "CONSTRUCT" being a modal verb to mark the indicative and "ASK" as marking the interrogative. Given that analysis, the distinction could have been marked in some other way.

If we consider a language such as RDF for representing knowledge, wouldn't it be interesting to consider the range of moods to express state of certainty (inferential, presumptive, hypothetical, et al) and finding a way to express that in the knowledge representation language?

At the very least, it is worth considering whether a particular part of the API is operating in imperative, interrogative, or indicative mood. This does get reflected even in names. For example, get-properties() is imperative ("(computer) get me the properties of ..."), and properties() is indicative ("the properties of ... are ..."). I find my libraries in functional languages are drawn towards the indicative because it describes an outcome: this and that will happen (somehow). The imperative tends to suggest an order of operations: do this then do that.

The language is for humans

The conlang approach suggests that we consider that the language is for use by human beings. Software developers often forget this, assuming that the purpose of a computer language is to be interpreted by some bit of software. While that is true, it is also a misleading and limited view. Software APIs need to communicate with human beings as well: humans who use them, humans who debug them, humans who refactor and extend them.

The argument for Hungarian notation markers is exactly this: knowing at a glance that something is a counter versus an index helps humans better understand what the software does. Software libraries are full of all kinds of little conventions like this, even if they are not systematically defined. Personally, I will use ix to mark an index that may be unordered rather than strictly sequential. Thus $jix is a positive integer in some range, but $j will be a positive integer running from 1 to the upper limit.

A consequence of keeping humans in mind is that it becomes important to maintain fairly consistent rules even for aspects of the API that aren't important to the computer. We've seen a few examples of this already: a standard ordering for targets and actions in the naming of functions, or the use of markers to indicate classifications of variables that aren't important to the compiler.

Another example to consider is the robust, cross-linguistic observation that higher frequency (Zipf36), more predictable (Levshina22), or simpler (Lewis16) terms tend to be shorter. Thus: "a", "it", "cat" versus "jaguar", "caterpillar", "transcendentalism". Default case, gender, tense, etc. markers tend to disappear entirely. Thus we have "il entend" (present) but "il entendra" (future) and "il entendit" (past).

The application to APIs is direct: simple common functions should have short names. Markers for default or common situations should be shorter than those for non-default and less common situations, or absent entirely. This is a plausible rationale for the use of "properties" as an action in the prior examples: keeping common operations shorter in their expression.

Another thing humans expect to see is short and consistent case frames. Here's one analysis of case roles for English clauses (Winograd83, Figure B-13):

The idea is that there are a few central roles for each kind of process. Given a set of API functions it is useful to think through the types of processes they represent, the central participants for a collection of similar operations, and be consistent in how they are ordered and indicated. The types of processes and their participants are not necessarily identical to those of English. It is also useful to distinguish central participants (such as Medium in Material Process) from secondary participants (such as Manner in Circumstances). There is always a temptation in software to start larding up functions with an endless array of parameters and options. The conlang lesson here is that less is more and that a little analysis and organization can prevent a complicated and unusable mess down the road.

A quick note on ambiguity

A number of the grammar fragments in this section are ambiguous, allowing multiple possible parses. While it is prudent to avoid ambiguity as much as possible, I don't tend to worry about it much. I use a system (CoffeeSacks) that lets me define a function to resolve ambiguity appropriately.

The function I use takes a greedy approach and always picks the option with the longest matching prefix. This resolution mechanism is easy to implement, easy to understand, and gives me good control over the results. It is conformant with how many regular expression processors resolve ambiguity, so it also has the advantage of familiarity. I recommend it.

Representing things: characteristics of things

When something is drawn, it will have various characteristics: size, colour, and so on. How should these be expressed? There are a number of options: as a single function with a set of parameters, as a specific function with named parameters, in a more natural English style where the parameters become attributes modifying a noun, or expanding our horizons in the conlang way, through using attributive affixes. Each of these variations uses a different IXML grammar to produce the same XML for the driver to process. Let's look at a few of them, all to express the idea of a group of five small fish rendered in red with a speckled stylistic effect.

thing(fish, 5, 0.25, red, speckled)

thing = -"thing", -"(", s?, (kind, -",", s), (count, -",", s), (size, -",", s), (colour, -",", s), (styling), s?, -")" .
@kind = "fish" | "octopus" | "jellyfish" | "ship" .
@count = integer .
@colour = name .
@size = number .
@styling = ("speckled" | "crosshatched") .

<thing kind='fish' count='5' size='0.25' colour='red' styling='speckled'/>

fish(count=5, size=0.25, colour=red, style=speckled)

thing = kind, (s?, options)? .
@kind = "fish" | "octopus" | "jellyfish" | "ship" .
-options = -"(", s?, option++(-",", s?), s?, -")" .
-option = count | colour | size | styling .
@count = -"count", -"=", integer .
@colour = -"colour", -"=", name .
@size = -"size", -"=", number .
@styling = -"style", -"=", ("speckled" | "crosshatched") .

<thing kind="fish" count="5" colour="red" size="0.25" styling="speckled"/>

5 wee red speckled fish

thing = (options, s)?, kind .
@kind = "fish" | "octopus" | "jellyfish" | "ship" .
-options = (count, s)?, (size, s)?, (colour, s)?, (styling)? .
@count = integer .
@size = (-"enormous", +"4.0") |
        (-"large", +"2.0") |
        (-"standard", +"1.0") |
        (-"small", +"0.5") |
        (-"wee", +"0.25")
.
@colour = name .
@styling = ("speckled" | "crosshatched") .

<thing count="5" colour="red" size="0.25" styling="speckled" kind="fish"/>

5 red bespeck-fishies

thing = (options, s)?, (prefixes)?, kind, (suffixes)? .
@kind = "fish" | "octopus" | "jellyfish" | "ship" .
-options = (count, s)?, (colour)? .
@count = integer .
@colour = name .
-prefixes = styling, -"-" .
@styling = (-"bespeck",+"speckled") | (-"behatch",+"crosshatched") .
-suffixes = size .
@size = ((-"zilla"|-"zillas"), +"4.0") | ((-"adon"|-"adons"), +"2.0") |
        (+"1.0") |
        ((-"ling"|-"lings"), +"0.5") | ((-"y"|-"ies"), +"0.25") .

<thing count="5" colour="red" size="0.25" styling="speckled" kind="fish"/>

Things, spaces, arrangements

The basic drawing unit is not just some things (such as 5 wee red fish) but a thing at a particular location within some space. More generally it is a group of things within a set of spaces arranged in some particular way. For example, we might say "5 fish fill small box in spiral". The fish are the things, the box is the space, and the spiral is the arrangement. Each of these is constructed and managed in its own way. Arrangements are not rendered directly: only the locations associated with them matter. Groups may or may not be rendered, depending on their characteristics. A blue box would need to be rendered, but a box with no colour or other drawing characteristics would not. A thing is always rendered. The arrangement is relative to the space. Here's the thing: the same kind of item may operate as a thing, a space, or an arrangement. How can we indicate and distinguish these roles?

Another way to think about this is: the major participants in a Drawing process are a Group (of things), a Division (of spaces), and an Arrangement (of locations). Given that case frame, how should we mark the case roles?

5 wee circles fill large circle in standard circle

drawing = thing, s, -"fill", s, space, s, -"in", s, arrangement .

thing = (options, s)?, kind .
@kind = "circle",(-"s")? .
space = (options, s)?, kind .
arrangement = (options, s)?, kind .
-options = (count, s)?, (size)? .
@count = integer .
@size = (-"enormous", +"4.0") | (-"large", +"2.0") | (-"standard", +"1.0") |
        (-"small", +"0.5") | (-"wee", +"0.25") .

<drawing>
   <thing count='5' size='0.25' kind='circle'/>
   <space size='2.0' kind='circle'/>
   <arrangement size='1.0' kind='circle'/>
</drawing>

5 wee circles large in-circle standard at-circle

drawing = item++s .

-item = thing | space | arrangement .
thing = (options, s)?, kind .
@kind = "circle",(-"s")? .
space = (options, s)?, -"in-", kind .
arrangement = (options, s)?, -"at-", kind .
-options = (count, s)?, (size)? .
@count = integer .
@size = (-"enormous", +"4.0") | (-"large", +"2.0") | (-"standard", +"1.0") |
        (-"small", +"0.5") | (-"wee", +"0.25") .

<drawing>
   <thing count='5' size='0.25' kind='circle'/>
   <space size='2.0' kind='circle'/>
   <arrangement size='1.0' kind='circle'/>
</drawing>

large in-circle 5 wee circles standard at-circle

<drawing>
   <space size='2.0' kind='circle'/>
   <thing count='5' size='0.25' kind='circle'/>
   <arrangement size='1.0' kind='circle'/>
</drawing>

5 circley circleadontown circledot

drawing = item++s .

-item = thing | space | arrangement .
thing = (quantifier, s)?, kind, suffixes .
@kind = "circle",(-"s")? .
space = (quantifier, s)?, kind, suffixes, -"town" .
arrangement = (quantifier, s)?, kind, suffixes, -"dot" .
-quantifier = count .
@count = integer .
-suffixes = size .
@size = ((-"zilla"|-"zillas"), +"4.0") | ((-"adon"|-"adons"), +"2.0") |
        (+"1.0") |
        ((-"ling"|-"lings"), +"0.5") | ((-"y"|-"ies"), +"0.25") .

<drawing>
   <thing count='5' kind='circle' size='0.25'/>
   <space kind='circle' size='2.0'/>
   <arrangement kind='circle' size='1.0'/>
</drawing>

Open classes, closed classes

So far the grammar fragments have enumerated the allowable things, spaces, and so on. The real more complex grammar leaves some of these open. A thing can be anything. This is an open class. A simple space is also an open class, but started as a closed class, with an enumeration of valid members:

thing = thing-pre-options, kind, (s?, options)? .
@kind = simplename .

space = space-pre-options, space-kind, (s?, options?) .
@space-kind = "box" | "circle" | "ngon" | "canvas" .

The driver looked up all the kind attributes to load the appropriate components. As the language developed, and the component bindings for more spaces were created, I shifted space to an open class. The grammar uses the rename marker (working draft) to turn the space-kind attribute to kind. The lookup code stayed the same and any rules in the grammar using space-kind stayed the same:

thing = thing-pre-options, kind, (s?, options)? .
@kind = simplename .

space = space-pre-options, space-kind, (s?, options?) .
@space-kind>kind = simplename .

I could have just changed space-kind to kind throughout the grammar, but I liked having the grammar rules be more specific in this way.

Locations created a stronger challenge. Location is a closed class, complicated by the presence of additional information for some kinds of locations and not others. Specific locations such as "center" are unique. There can be only one. But random locations could be multiple and accept a count. I wanted to simplify the representation for the processing so that there would always be a location-kind attribute along with a count attribute where relevant. Renaming solves this:

location =
  location-kind |
  (random-location-kind, ((-"-" | s), count)?)
.
@location-kind =
  ("center" |
  "top-left" | "bottom-right" | "top-right" | "bottom-left" |
  "top-center" | "bottom-center" | "left-center" | "right-center"
  ), -"s"?
.
@random-location-kind>location-kind =
  ("random" | "radial" | "poisson"), -"s"?
.

Both the location kind and the count will appear as attributes on the location element, and the naming of the kind will be consistent regardless of whether it is a countable kind or a unique kind. The grammar properly constrains the use of the count.

A similar problem arises with drawing options. Rather than having a special attribute for colour, and another for size, and then separate elements for other various options, I wanted it all unified and consistently represented using the element option with a name attribute and then a type-based attribute as appropriate.

The grammar uses renaming extensively to normalize the names:

thing = thing-pre-options, kind, (s?, options)? .
@kind = simplename .
-thing-pre-options = size-option?, colour-option? .
size-option>option = size-option-name, size, s .
@size-option-name>name = +"size" .
@size>number =
  (-"enormous", +"4.0") |
  (-"large", +"2.0") |
  (-"standard", +"1.0") |
  (-"small", +"0.5") |
  (-"wee", +"0.25")
.

colour-option>option = colour-option-name, @colour, s .
@colour-option-name>name = +"colour" .

-options = -"(", option++(s?,-",",s?), -")" .
option = @name, s?, -"=", s?, value .
-value = @number | @string | @boolean | @colour .

Although the size option is defined in terms of strings, it ends up being represented as an option with a numerical value. A string like "wee red fish(styling="speckled")" produces a consistent set of options:

<thing kind="fish">
  <options>
    <option name="size" number="0.25"/>
    <option name="colour" colour="red"/>
    <option name="styling" string="speckled"/>
  </options>
</thing>

This simplifies passing drawing options to the maker functions.

Grammatical gender

Some things cover area. We could use them as clipping bounds or as spaces. We could decorate the area they cover in various ways (crosshatching or speckling, for example). Some things consist of strokes. They can be used as paths. We could draw them with different pens (tapered strokes or dashed strokes, for example). They have no interior to decorate. Some things may be able to be viewed in either light. Is a red circle a circular red line or a circular red area?

It is the kind of distinction that grammatical gender can help with.

thing = thing-pre-options, kind, gender?, (s?, options) .
@gender = (-"-", "stroke") | (-"-", "zone") .

The driver can then determine that:

5 wee red dashed circles-stroke

5 wee red dashed circles-zone