Balisage Paper: Metadata for Creators

Metadata, annotated

Here is some actual embedded metadata from one of my experiments, slightly redacted. Let's examine it in detail.

First up is the colophon, a longer bit of descriptive text along with a copyright statement, all embedded as the title of an SVG path that draws a signature. This is all signature metadata. The signature is directly visible, while the rest pops up when the mouse hovers over it.

<svg:path id="signature" style="fill:white;stroke-width:0.26"
      d="m 7.58 181.6 c 0.51 -0.49 3.84 ..."
      transform="scale(0.5,0.5)">
   <svg:title>Copyright© ... 2022...
A #genuary work
Randomized flow field based on Perlin noise...
   </svg:title>
</svg:path>
      

Here is a bit of variable process metadata: a couple of time stamps, the initial creation time and the time of stylesheet rendering.

<svg:metadata xmlns:svg="http://www.w3.org/2000/svg">
<metadata xmlns="http://mathling.com/art">
  <created>2022-01-21T09:05:07.868096-08:00</created>
  <processed>2022-01-21T09:23:01.901638-08:00</processed>
      

There follows a bit identifying text containing the project and variant tags, and a brief description from a subcomponent. These are organizational.

  <description>genuary j21combine</description>
  <description>genuary: Flow fields</description>
      

Parameters (name/value pairs) are named using poor man's namespaces: tags associated with various components to avoid name collisions. Here we have j7 and knot, which reference the main subproject and a component for making modulated torus knots. Using simple strings helps ease-of-use at the point where parameters are referenced in projects or used in searches. This is all fixed process metadata: the point is to capture aspects of the algorithm that generated this work. For example, the knots used in this algorithm are all symmetric between the x and y coordinates (knot.p-same=100) and 15 percent of then are involuted (knot.p-involuted=15).

  <component name="j7">
    <parameter name="j7.field">adhoc</parameter>
    <parameter name="j7.gradient">oxygen oxygen-reverse plasma</parameter>
    <parameter name="j7.grid.size">200</parameter>
    <parameter name="j7.obstruction.gradient">oxygen</parameter>
    <parameter name="j7.obstruction.type">ellipse</parameter>
    <parameter name="j7.obstruction.visibility">0</parameter>
  </component>
  <component name="knot">
    <parameter name="knot.garden.fade">true</parameter>
    <parameter name="knot.max-factors">3</parameter>
    <parameter name="knot.openness">1</parameter>
    <parameter name="knot.p-involuted">15</parameter>
    <parameter name="knot.p-same">100</parameter>
    <parameter name="knot.p-skewed">10</parameter>
    <parameter name="knot.precision">1</parameter>
  </component>
      

There are various standard canvas sizes and scalings, identified by name. The resolution "medium" is for 1600x1600 images. This is a bit of intrinsic metadata, because the actual canvas size is manifest in the image.

  <parameter name="resolution">medium</parameter>
      

An "algorithm" is a randomization algorithm. Because we have an explicit descriptor for these rather than pure code, it is possible to report on the randomizers being used in a structured fashion. Again, we're using the poor man's namespace technique to keep randomizers from different components separated. This is all fixed process metadata.

  <component name="j7">
    <algorithm name="j7.density" distribution="normal"
               min="0.1" max="0.9"
               mean="0.6" std="0.1"/>
    <algorithm name="j7.drop.percent" distribution="constant"
               min="5"
               cast="integer"/>
    <algorithm name="j7.obstruction.n" distribution="constant"
               min="0"
               cast="integer"/>
    <algorithm name="j7.line.length" distribution="uniform" 
               min="16" max="32"/>
    <algorithm name="j7.line.width" distribution="zipf"
               alpha="1.01"
               max="16"
               post-multiplier="1"
               cast="integer">
               <sums>0.294431920541423 0.440198890360574 0.5371753519488989 ...</sums>
    </algorithm>
  </component>
  <component name="knot">
    <algorithm name="knot.p" distribution="uniform"
               min="3" max="11"
               cast="integer"/>
    <algorithm name="knot.q" distribution="uniform"
               min="2" max="5"
               cast="integer"/>
    <algorithm name="knot.r" distribution="uniform"
               min="-1" max="1"
               cast="decimal"/>
    <algorithm name="knot.x-y-same" distribution="flip" p="100"/>
    <algorithm name="knot.skewed" distribution="flip" p="10"/>
    <algorithm name="knot.involution" distribution="flip" p="15"/>
    <algorithm name="knot.stretch" distribution="normal"
               mean="1" std="0.5"
               min="0.5" max="2"
               cast="decimal"/>
    <algorithm name="knot.n-factors" distribution="uniform"
               min="2" max="3"
               cast="integer"/>
  </component>
  <component name="flowfield">
    <algorithm name="flowfield.blend.fraction" distribution="uniform"
               min="0" max="1"/>
    <algorithm name="flowfield.obstruction.angle" distribution="uniform"
               min="0" max="90"
               cast="integer"/>
    <algorithm name="flowfield.obstruction.size" distribution="zipf"
               alpha="0.5"
               max="7"
               cast="integer">
               <sums>0.2287577393348463 0.5225872304516045 0.6369661001190277 ...</sums>
    </algorithm>
    <algorithm name="flowfield.noise.scale" distribution="multimodal"
               min="0.0001" max="0.1">
      <algorithm name="1" distribution="normal" mean="0.001" std="0.0005"/>
      <algorithm name="2" distribution="normal" mean="0.01" std="0.005"/>
    </algorithm>
    <algorithm name="flowfield.obstruction.ratio" distribution="normal"
               mean="3" std="0.5"
               min="0.1" max="10"/>
  </component>
      

Stylesheets have their own set of parameters. There are stylesheet-level components as well, which are identified through introspection. The XML representation passed to the stylesheet made use of three identified stylesheet components: polygons, paths, and graphs. The relevant parameters will only be reported here if the XML contains constructs relevant to those components.

  <stylesheet-parameters>
    <component name="polygon">
      <parameter name="polygon.fill"/>
      <parameter name="polygon.opacity"/>
      <parameter name="polygon.filter"/>
      <parameter name="polygon.gradient"/>
    </component>
    <component name="path">
      <parameter name="path.filter"/>
      <parameter name="path.fill">none</parameter>
      <parameter name="path.opacity"/>
    </component>
    <component name="graph">
      <parameter name="graph.fill"/>
      <parameter name="graph.opacity">1</parameter>
      <parameter name="graph.filter"/>
      <parameter name="graph.edge-mode">graph</parameter>
      <parameter name="graph.gradient"/>
      <parameter name="sand.grains">10</parameter>
      <parameter name="sand.size">2</parameter>
    </component>
      

These parameters were selected via the randomization algorithms at run time. Here the specific values selected are reported, so this is variable process metadata. For example, j7.density was generated from the j7.density randomizer reported previously. Multiple values for j7.flow.swirliness were selected, and we get a report on each. They aren't really "stylesheet" parameters: it is an artifact of the generation that causes them to be reported here.

    <component name="j7">
      <dynamic-parameter name="j7.density">0.67</dynamic-parameter>
      <dynamic-parameter name="j7.points.n">1075</dynamic-parameter>
      <dynamic-parameter name="j7.steps.n">70</dynamic-parameter>
      <dynamic-parameter name="j7.line.length">32</dynamic-parameter>
      <dynamic-parameter name="j7.line.width">1</dynamic-parameter>
      <dynamic-parameter name="j7.drop.percent">5</dynamic-parameter>
      <dynamic-parameter name="j7.flow.center.1">1321,657</dynamic-parameter>
      <dynamic-parameter name="j7.flow.swirliness.1">360</dynamic-parameter>
      <dynamic-parameter name="j7.flow.center.2">435,524</dynamic-parameter>
      <dynamic-parameter name="j7.flow.swirliness.2">720</dynamic-parameter>
      <dynamic-parameter name="j7.flow.center.3">1574,618</dynamic-parameter>
      <dynamic-parameter name="j7.flow.swirliness.3">180</dynamic-parameter>
    </component>
      

Not all the stylesheet properties have been componentized. This is an ongoing bit of refactoring work. The dividing line between process metadata and intrinsic organizational metadata gets a little murky at this point. These parameters generally control details of the final appearance, and as such they end up being manifest in the result. For example, background-fill controls the background colour, which is certainly manifest in the result, so the report of it explicitly makes it intrinsic metadata.

    <parameter name="debug">false</parameter>
    <parameter name="palettes">oxygen oxygen-reverse plasma</parameter>
    <parameter name="effects"/>
    <parameter name="signed">true</parameter>
    <parameter name="signature.colour">white</parameter>
    <parameter name="stroke-class"/>
    <parameter name="stroke-cap">round</parameter>
    <parameter name="stroke-join">round</parameter>
    <parameter name="stroke-opacity">1</parameter>
    <parameter name="stroke-fill"/>
    <parameter name="stroke-filter"/>
    <parameter name="default-stroke-width">2</parameter>
    <parameter name="max-stroke-width">500</parameter>
    <parameter name="background-fill">black</parameter>
    <parameter name="background-opacity">1</parameter>
    <parameter name="background-filter"/>
    <parameter name="body-fill"/>
    <parameter name="body-opacity"/>
    <parameter name="body-filter"/>
    <parameter name="body-reach"/>
    <parameter name="dot-shape">circle</parameter>
    <parameter name="dot-fill">black</parameter>
    <parameter name="dot-opacity">1</parameter>
    <parameter name="dot-size">3</parameter>
    <parameter name="dot-filter"/>
    <parameter name="box-fill"/>
    <parameter name="box-opacity">1</parameter>
    <parameter name="box-filter"/>
    <parameter name="shape-fill"/>
    <parameter name="shape-opacity">1</parameter>
    <parameter name="shape-filter"/>
    <parameter name="text-size">20</parameter>
    <parameter name="colours.1.gradient">plasma</parameter>
    <parameter name="colours.2.gradient"/>
    <parameter name="colours.3.gradient"/>
    <parameter name="colours.4.gradient"/>
    <parameter name="colours.5.gradient"/>
    <parameter name="colours.6.gradient"/>
    <parameter name="colours.7.gradient"/>
    <parameter name="colours.8.gradient"/>
    <parameter name="colours.9.gradient"/>
    <parameter name="colours.10.gradient"/>
  </stylesheet-parameters>
</metadata>
</svg:metadata>

Process metadata that captures details of the generation and styling allows me to go back to an experiment later (perhaps months later) and answer the basic question of how the image was made and what settings I can use to create a similar image. Organizational metadata allows me to search for works of particular kinds, and helps with writing up contextual information when I publish particular works. And signature metadata allows me to assert authorship and copyright for published works, give them titles, and so forth.

The notion of capturing process metadata is not new. Lubell, for example, talks extensively about the importance of capturing metadata around the process of creation, although the focus of that work is more about the long-term preservation and archiving of such information.

Embedded metadata works better for me than external metadata because it can't get disassociated from the work. It enables workflows that amount to "look at image, examine information about how image was made, make new image using the same components based on those settings". The mechanics of capturing and embedding metadata can get complicated, particularly for certain more elaborate creation workflows. Let's look at what some of the issues are that I encountered on this journey, and lessons learned in addressing them.

Lesson: Embed Metadata as Part of Creation Process

The first lesson is to embed metadata into the works as part of the process of creation. By making metadata part of the process up front, it is more likely to happen reliably. When metadata is embedded rather than created separately, it is less likely to become separated from the work, and is ready to hand for use.

Without concrete action, a digital artwork is unsigned. I find this unsatisfactory. I want it to be signed. I want it to have a copyright statement embedded within it. It is really very important that signature metadata not be lost, no matter where the work travels. For digital artwork, screen-scraping robots putting other people's work up for sale as NFTs is a significant problem. They can certainly strip embedded metadata too, but at least it gives them another hurdle to cross.

First I constructed an SVG path representing my signature with the help of a drawing tool (InkScape). Then I added some templates and stylesheet parameters to control where the signature appears as well as its scaling and colouring. I attach a copyright statement plus a free-form textual description of the work as the SVG title of the signature, so that hovering over the signature brings it up. The main template for the root element calls the appropriate templates to define the define and place the signature so that it appears on top of the rendering. A standard colophon function provides this free-form text.

Embedding the rest of the metadata is more straightforward. The output of the XQuery phase is a domain specific XML vocabulary. Embedding metadata into that output is trivial: just add metadata to the output, wrapped in a new element in the vocabulary.

declare variable $N as xs:integer external;
...
declare function this:make-some-art() 
{
  let $scale := 3.2
  let $kind := "tree"
  let $result := this:something-cool($N, $scale, $kind)
  return (
    <art:canvas>
      <art:metadata>
        <art:parameter name="n">{$N}</art:parameter>
        <art:parameter name="scale">{$scale}</art:parameter>
        <art:parameter name="kind">{$kind}</art:parameter>
      </art:metadata>
      {$result}
    </art:canvas>
  )
};
        

The output of the XSLT phase is SVG, also an XML vocabulary, so embedding metadata in that output is also trivial: adding a template for the metadata element that copies it into the svg:metadata element along with whatever else needs embedding at the stylesheet level.

<xsl:template match="art:metadata">
  <svg:metadata>
    <art:metadata style="display:none">
      <xsl:copy-of select="node()"/>
      <art:stylesheet-parameters>
        <art:parameter name="fish.brush">
          <xsl:value-of select="fish.brush"/>
        </art:parameter>
        <art:parameter name="default-stroke-width">
          <xsl:value-of select="default-stroke-width"/>?
        </art:parameter>
        <art:parameter name="background-fill">
          <xsl:value-of select="background-fill"/>?
        </art:parameter>
      </art:stylesheet-parameters>
    </art:metadata>
  </svg:metadata>
</xsl:template>
        

Lesson: Names for Everything

Names come into play in two ways: the names for the metadata fields and the use of names as the contents of metadata fields. Both are important.

It is easy to focus on the content of a metadata field, but the field name is important too. Field names are how we relate metadata to the source data. When it comes to process metadata, we need to relate those names to things happening in the process that produced the works. When it comes to process metadata, so much is idiosyncratic. Having a naming system so that it is easy to figure out where to look and spot similarities and differences between different processes and their generated works becomes crucial.

Using names for the contents of the metadata fields is helpful as well. Naming conventions can be used to identify standard ways of modifying or combining behaviours, which helps with metadata linkage, ease of implementation, and ease of experimentation. The more things in the process have readily identifiable names, the easier the association between metadata and the source process gets.

For example, SVG allows for complex filter effects. Rather than just create explicit SVG code when I use them, I have names for some basic effects and a naming scheme for combining or modifying them. Thus the name softbloom·khaki-filigree names an SVG filter that combines the filter named filigree, modified to use khaki coloured lighting, with the SVG filter named softbloom, which is the bloom filter modified for a prespecified narrower range.

Notice how a system that makes capturing process metadata easier also has the effect of making the implementation of the process itself easier. To both enable and document complex behaviour, it need only be named.

Figure 6: Abstract Trees

Image description

Abstract trees. Circles filled with intricate filigrees act as leaves. The leaves are rendered with the softbloom·khaki-filigree filter.

A similar scheme is in play for colour gradients. I have a library of base (named) gradients, and a naming system for creating variations of them. Thus oasis names a particular colour palette, oasis-slant orients that gradient on a diagonal, and oasis-outflow creates a circular gradient from it, and so on. Reporting the colour scheme of an image in the metadata reduces to reporting such names. On the field name side, a consistent naming convention help keep things clear: xxx.gradient identifies the gradient used for the component xxx.

Image description

Rectangles filled with colour gradients: straight, slanted, and in a bullseye pattern.

Providing names for everything helps with accessibility also: during the publication phase alternative text can be written based on the named components and visual elements. For example, Figure 6 uses a branching component, the aforementioned softbloom·khaki-filigree effect for the leaf components which have the shape kind circle. There is a turbidity-outflow gradient for the background. This information can be readily translated into a useful textual description.

Lesson: Use Descriptors for Coherent Pieces of Behaviour

An important aspect of these generative art pieces is randomness, and different random distributions are used for different effects. For example, a normal distribution might be nice for how close one visual element is to another; a Zipf distribution for how large the components should be. Tweaking these distributions is a large part of the experimental process. What should the mean be? Should there be a maximum value? To understand how to replicate a piece, or what the difference is between a good outcome and a bad one, it is important to have a report on the random distributions used in their production.

The solution to this problem, which has the double benefit of making tweaking the distributions easier, is to use descriptors of the distributions and one generic function that knows how to interpret them and another than knows how to dump out a structured description suitable for inclusion in the metadata.

map {
  "distribution": "normal",
  "mean": 0.25,
  "std": 0.1,
  "min": 0
}
      

Descriptors can play a more general role in process metadata generation. For simple components, a descriptor can define the key aspects of that component and how it is to be produced, which one function can interpret and another can emit as metadata. As with the naming schemes, this simplifies both metadata linkage and implementation and experimental tweaking. One can, of course, go one step further, and gives names to particularly fine descriptors.

Figure 8: Example: A Knot: Descriptor, Description, and Rendering

  map {
    "kind": "modulated-knot",
    "p": 3,
    "x-factors": (
      map {"q": 7, "r": 0.52, "skew": true()},
      map {"q": 4, "r": 0.11, "skew": false()}
    ),
    "y-factors": (
      map {"q": 5, "r": -0.9, "skew": false()},

      map {"q": 4, "r": 0.7, "skew": false()}
    ),
    "stretch": 1.25,
    "openness": 1
  }
      

x=1.25*cos(3θ) * (0.7*sin(7θ) + 0.11*cos(4θ))
y=cos(3θ) * (-0.9*cos(5θ) + 0.7*cos(4θ))
z=0.2*sin(7θ)
      

Image description

Perspective rendering of modulated torus knot as a series of blocks.

Descriptors may carry a rich information set. The metadata description of descriptors is therefore frequently not just a simple name/value pair. Recall, for example, the structured information about randomizers we saw in the annotated metadata example:

    <algorithm name="j7.density" distribution="normal"
               min="0.1" max="0.9"
               mean="0.6" std="0.1"/>
      

Lesson: Automate

The problem with having to manually add code to emit metadata is that it is too easy to be lazy or forget to update the metadata. This is especially likely to happen early in the creation process when there is the most variability and having good process metadata is most important. The work of manually updating code to emit the metadata is even more pressing on the XSLT side, because the XSLT tends to be relatively static, and updating it every time to dump the right stylesheet parameters gets annoying.

Anything that can be done to reduce the distance between implementation and instrumentation increases the reliability of the metadata produced. To the extent that things can be automated, they should be automated. An intermediate step is to use programming conventions and templates to reduce the amount of effort required to instrument a particular work.

I use a combination of automation and programming convention. Every project uses a common execution driver function and uses a map of parameters to hold important tuning values and a map of randomizer descriptors to hold the important randomization algorithms. A standard parameter initialization function handles the overriding of values with external parameters. The driver gets a callback to a content generation function and to a metadata generation function as well as the parameter and randomizer bundles. It is the driver's job to assemble the content and the metadata. Some metadata is still manual (whatever is in the metadata callback), but mostly the driver just dumps out the parameter and randomizer bundles plus standard timestamp values.

declare function this:generate(
  $n-tries as xs:integer,
  $tag as xs:string,
  $metadata as function(*),
  $content as function(*),
  $canvas as map(*),
  $randomizers as map(*),
  $parameters as map(*)
)
{
  for $i in 1 to $n-tries return (
    document {
      <art:canvas width='{$canvas("width")}' 
                  height='{$canvas("height")}'
      >{
        <art:metadata>
          <art:description>
            {$tag}: {$parameters("description")}
          </art:description>
          {this:created-at()}
          {this:dump-parameters($parameters)}
          {this:dump-randomizers($randomizers)}
          {$metadata($i, $canvas, $parameters)}
        </art:metadata>,
        $content($i, $canvas, $parameters)
      }</art:canvas>
    }
  )
};
        

Things are a little more primitive on the XSLT side. Each stylesheet parameter is declared separately and the metadata templates just enumerate them all. There is a named template that can be overridden in a project-specific stylesheet to dump out project-specific stylesheet parameters.

<xsl:template match="art:metadata">
  <svg:metadata>
    <art:metadata style="display:none">
      <art:processed>
        <xsl:value-of select="current-dateTime()"/>
      </art:processed>
      <art:description>
        <xsl:value-of select="$meta-description"/>
      </art:description>
      <xsl:copy-of select="node()"/>
      <art:stylesheet-parameters>
        <xsl:call-template name="dump-project-parameters"/>
        <xsl:call-template name="dump-parameters"/>
      </art:stylesheet-parameters>
    </art:metadata>
  </svg:metadata>
</xsl:template>
        

Lesson: Merge Information from Components

More complex artworks rely on the integration of multiple components, each with their own set of parameters. I expanded programming conventions a little bit to cover these components: each of them has its own metadata function, parameter bundles, randomizers, and colophons. The framework combines component parameter and randomizer bundles with the local bundles to provide a complete sets of values. Similarly, the framework merges information from metadata callbacks for the appropriate set of components to get the full set of relevant metadata, and merges information from colophon callback to produce a full colophon.

To avoid name collisions I use poor man's namespaces^[2]: every name is prefixed with a unique tag associated with a particular component. For example, in Figure 11, the prefix astro names a component that produces astronomical objects, and the prefix mountain names a component that produces mountain ranges. The component map ($COMPONENTS) defines which components are in play, so that their information is merged automatically. Components may have subcomponents, which are merged in the same way within their parent component.

Originally I had the main project explicitly perform the appropriate merges, but as more components came into play, automation has become more important, another reminder of that lesson.

(: Framework functions to merge component parameters :)
declare function this:component-algorithm-parameters(
  $components as map(*),
  $resolution as xs:string,
  $canvas as map(*)
) as map(*)
{
  fold-left(
    $components=>map:keys(), map {},
    function($map, $key) {
      let $component := $components($key)
      let $fn := 
        function-lookup(QName($component("namespace"), "algorithm-parameters"), 2)
      return (
        if (empty($fn)) then $map
        else (
          map:merge((
            $map,
            $fn($resolution, $canvas)
          ), map { "duplicates" : "use-last" })
        )
      )
    }
  )
};

declare function this:component-metadata(
  $components as map(*),
  $canvas as map(*),
  $randomizers as map(*),
  $parameters as map(*)
) 
{
  for $key in $components=>map:keys()
  let $component := $components($key)
  let $fn :=
    function-lookup(QName($component("namespace"), "metadata"), 3)
  return (
    if (empty($fn)) then ()
    else $fn($canvas, $randomizers, $parameters)
  )
};
        

(: 
 : Main project
 :)

(:  
 : Component declaration: passed into framework 
 :)
declare variable $COMPONENTS as map(*) :=
map {
  "astro": map {
    "namespace": "http://mathling.com/art/astronomy"
  },
  "mountain": map {
    "namespace": "http://mathling.com/art/mountain"
  }
};

(: 
 : Local properties: component properties
 : already included automatically
 :)
declare function this:algorithm-parameters(
  $resolution as xs:string
) as map(*)
{
  map {
    (: Overrides :)
    "astro.star.density": 20
    "mountain.crowding": 5,
    "mountain.spikiness": 10,
    "mountain.smoothing": 0.5,
    (: Local parameters :)
    "j25.slant": true()
  }
};

(:
 : Local metadata function: 
 : Since all component metadata has already been included, 
 : as well as all local parameters and randomizers, only
 : exceptions need to be handled here.
 :)
declare function this:metadata(
  $canvas as map(*),
  $parameters as map(*)
)
{
  ()
};
        

Lesson: Be Selective

The mechanism for embedding stylesheet parameters I described before is relatively simple: just dump out all the stylesheet parameters. The problem with this is that as time has gone by I have developed more and more components with more and more parameters. It is hard to pick out important values from one big long undifferentiated list of parameters, most of which are irrelevant to a particular work. This list just gets longer and longer. I'd also like to partition the implementation, so that the parameter dumping doesn't have to have knowledge of all the components. My solution to this problem is introspection: I use a metadata stylesheet mode to calculate which stylesheet components are actually in play, and only dump out the parameters specific to that component if it is in evidence. Shared parameters always get dumped out.

Figure 12 shows a snippet from a stylesheet component. There are a number of different parameters that control how the elements art:ghost and art:circle-ghost will be rendered. In the metadata mode, the first of these seen in the document will trigger the dumping of this component's stylesheet parameters into the metadata output.

<xsl:param name="ghost.x">1</xsl:param>
<xsl:param name="ghost.y">1</xsl:param>
<xsl:param name="ghost.n">100</xsl:param>
<xsl:param name="ghost.opacity">0.01</xsl:param>
<xsl:param name="ghost.fade">1</xsl:param>
<xsl:param name="ghost.shrink">1</xsl:param>
<xsl:param name="ghost.scale">1.01</xsl:param>
<xsl:param name="ghost.rotate">30</xsl:param>

<xsl:template match="(//(art:ghost|art:circle-ghost))[1]" mode="metadata">
  <art:component name="ghost">
    <art:parameter name="ghost.x">
      <xsl:value-of select="$ghost.x"/>
    </art:parameter>
    <art:parameter name="ghost.y">
      <xsl:value-of select="$ghost.y"/>
    </art:parameter>
    <art:parameter name="ghost.n">
      <xsl:value-of select="$ghost.n"/>
    </art:parameter>
    <art:parameter name="ghost.opacity">
      <xsl:value-of select="$ghost.opacity"/>
    </art:parameter>
    <art:parameter name="ghost.fade">
      <xsl:value-of select="$ghost.fade"/>
    </art:parameter>
    <art:parameter name="ghost.scale">
      <xsl:value-of select="$ghost.scale"/>
    </art:parameter>
    <art:parameter name="ghost.rotate">
      <xsl:value-of select="$ghost.rotate"/>
    </art:parameter>
    <xsl:apply-templates select="*" mode="metadata"/>
  </art:component>
</xsl:template>
        

Lesson: Add Hooks for Dynamic Values

Fixed parameter values are great, but they are not enough. An important aspect of these generative art pieces is randomness. We saw how we can use descriptors to capture the random distribution used. A more vexing problem remains: what about the actual value that was picked? That is not going to be a fixed value in the parameter bundle. How can we handle variable process metadata?

The basic solution to this problem is to provide a new element in the XML vocabulary that represents dynamic values of this kind. Stylesheet templates gather up all the art:dynamic-parameter elements and copy them into the metadata element. The metadata mode gathers up all the metadata. The metadata element is constructed by running through the document in that mode.

<xsl:template match="art:canvas" mode="metadata">
  <svg:metadata>
    <art:metadata style="display:none">
      <art:processed>
        <xsl:value-of select="current-dateTime()"/>
      </art:processed>
      <art:description>
        <xsl:value-of select="$meta-description"/>
      </art:description>
      <xsl:apply-templates select="art:metadata" 
                           mode="metadata"/>
      <art:stylesheet-parameters>
        <xsl:apply-templates select="* except art:metadata"
                             mode="metadata"/>
        <xsl:call-template name="dump-project-parameters"/>
        <xsl:call-template name="dump-parameters"/>
      </art:stylesheet-parameters>
    </art:metadata>
  </svg:metadata>
</xsl:template>

<xsl:template match="art:dynamic-parameter"/>
<xsl:template match="art:dynamic-parameter" mode="metadata">
  <xsl:copy-of select="."/>
</xsl:template>  

<xsl:template match="art:component"/>
<xsl:template match="art:component" mode="metadata">
  <xsl:copy-of select="."/>
</xsl:template>  
        

When it comes to components and subroutines, we need a little more here so that we can return the values we care about as well as the captured variable process metadata. My solution here is to use wrappers: maps that have a content entry that holds the main data and a dynamics entry that holds the variable metadata. Making this consistent across APIs so that it is more automated is still a work in progress. Designing for this in the first place would have been easier in the long run.

(: 
 : Create a wrapper for an astronomical object
 :)
declare function this:object(
  $content as item()*,
  $dynamics as element()*
) as map(*)
{
  map {
    "kind": "astronomical",
    "content": $content,
    "dynamics": $dynamics
  }
};

...
  (: A particular astronomical object, wrapped :)
  let $dynamics := (
    <art:dynamic-parameter name="moon.base.gradient">{
      $background
    }</art:dynamic-parameter>,
    <art:dynamic-parameter name="moon.texture.gradient">{
      $moon-gradients[$gradient-ix]
    }</art:dynamic-parameter>,
    <art:dynamic-parameter name="moon.texture.scale">{
      $texture-scale
    }</art:dynamic-parameter>
  )
  return (
    this:object(
      ($moon, $rings),
      $dynamics
    )
  )
        

Lesson: Use Annotations and Introspection to Document Complex Functions

Random distributions are constrained enough that simple descriptors can be passed to a single driver function and interpreted. There are other more complex functions where that approach doesn't work so well. For example, consider noise functions. These consist of base functions combined with each other and various modifiers to create programmatic textures. They are implemented as functions that return anonymous functions to perform the combinations. Defining them in this way allows them be bound to parameters, and referenced by name, allowing for more variation and tweaking.

map {        
  "planet.topography":
    noise:min(
      fbm:noise2(
        fbm:context()=>
          fbm:octaves(14)=>
          fbm:frequency(1.0)=>
          fbm:lacunarity(2.208)
      )=>noise:scale-bias(0.375,0.62)
      ,
      fbm:noise2(
        fbm:context()=>
          fbm:octaves(7)=>
            fbm:frequency(0.5)
       )=>noise:curve($shoreline)
     )=>noise:clamp(-1.0, 1.0)
...
}
        

In Figure 15, noise:min, noise:scale-bias, noise:curve, noise:clamp, and fbm:noise2 all return functions. The overall function is a chain of calls to produce the final value for a given input point. There are dozens of modifier functions like this and they can be nested and combined to an arbitrary degree. It is certainly possible to imagine them producing descriptors instead that then are interpreted to call the appropriate function, but this amounts to double the work every time a new modifier function is created. There is another downside: noise functions tend to be called a great deal in inner loops, potentially once for each point in a canvas. For a high resolution image that is millions of times. One of the advantages of using function values is that complicated algorithms can pre-compute some values and tables based on the specific parameters as part of the closure rather than as part of the body of the function, speeding them up.

There is another approach to reporting these functions in metadata: using function annotations. This approach depends on the availability of certain extension functions to evaluate ad hoc XQuery strings and to access the function annotations so they can be reported on. I have only worked out how to do this in Saxon and MarkLogic. For Saxon, Saxon-PE or Saxon-EE is required; for MarkLogic, the http://marklogic.com/xdmp/privileges/xdmp-eval execute privilege is required.

The first step is to annotate the functions being returned by the function library. Each (anonymous) function gets annotated with its name, which can then be accessed via the appropriate extension function.

Here is the Saxon version of a function to access a function name, either for a named function or an annotated anonymous function.

declare function this:function-name($f as function(*)) as xs:string
{
  if (empty(fn:function-name($f))) then (
    let $fn := 
      saxon:function-annotations($f)[
        map:get(.,"name")=xs:QName("art:name")
      ]!map:get(.,"params")
    return (
      if (empty($fn)) then "[anon]" else string($fn)
    )
  ) else (
    string(function-name($f))
  )
};
        

The MarkLogic version is similar.

declare function this:function-name($f as function(*)) as xs:string
{
  if (empty(function-name($f))) then (
    if (empty(xdmp:annotation($f, xs:QName("art:name")))) 
    then "[anon]"
    else xdmp:annotation($f, xs:QName("art:name"))
  ) else (
    string(function-name($f))
  )
};
        

However, we want to get a view into the nesting of calls of these anonymous functions, and to do this we need to do more. The idea is to call an annotator that constructs the more elaborate annotation we want. We end up with a library of functions that return functions, all wrapped inside a call to an annotator function to construct the dynamic annotations. Figure 18 shows one of the noise modifier functions: it computes the minimum of the output of two other noise functions. We annotate it with a base function name, and then call the annotator to add more information.

declare function this:min(
  $f1 as function(map(*)) as xs:double,
  $f2 as function(map(*)) as xs:double
) as function(map(*)) as xs:double
{
  this:annotate(
    %art:name("mod:min") 
    function ($point as map(*) as xs:double {
      fn:min(($f1($point), $f2($point)))
    },
    ($f1, $f2)
  )
};
        

Since function annotations cannot be dynamic, here is where we rely on an extension function to evaluate the dynamic string in the proper context. The trick is to wrap the function we care about in another anonymous function with the appropriate annotation. This is done inside an evaluation string. The tricky part is that the function variable is not part of the closure for the evaluated function (in Saxon, at least) so we have to add it as an argument and use partial evaluation. Figure 19 shows the Saxon 11 version, where we pass the function in the map of external variables to the saxon:xquery extension function.

declare function this:annotate(
  $f as function(map(*)) as xs:double,
  $parms as function(*)*
) as function(map(*)) as xs:double
{
  if (empty($parms)) then $f
  else (
    let $name := 
      this:function-name($f)||
      "("||string-join($parms!(this:function-name(.)),",")||")"
    return (
      saxon:xquery("
declare namespace art='http://mathling.com/art';
declare variable $f as function(map(*)) as xs:double external;

%art:name('"||$name||"') function (
  $point as map(*), 
  $f as function(map(*)) as xs:double
) as xs:double 
{
  $f($point)
}"
      )((), map {xs:QName("f"): $f})(?, $f)
    )
  )
};
        

The Saxon 10^[3] version is slightly different, using the pair of extension functions saxon:compile-query and saxon:query. Since saxon:query only takes nodes as external parameters and a function isn't a node, we have to pass it via the context item instead. Figure 20 shows what that looks like.

declare function this:annotate(
  $f as function(map(*)) as xs:double,
  $parms as function(*)*
) as function(map(*)) as xs:double
{
  if (empty($parms)) then $f
  else (
    let $name := 
      this:function-name($f)||
      "("||string-join($parms!(this:function-name(.)),",")||")"
    return (
      saxon:query(
        saxon:compile-query("
declare namespace art='http://mathling.com/art';
declare variable $f as function(map(*)) as xs:double := .('f');

%art:name('"||$name||"') function (
  $point as map(*),
  $f as function(map(*)) as xs:double
) as xs:double 
{
  $f($point)
}"
        ), 
        map {"f": $f}, ()
      )(?, $f)
    )
  )
};
        

In MarkLogic things are a little more straightforward, because parameters are passed as a map^[4] to the xdmp:eval function and the function variable is part of the closure, so no partial evaluation is required.

declare function this:annotate(
  $f as function(map:map) as xs:double,
  $parms as function(*)*
) as function(map:map) as xs:double
{
  if (empty($parms)) then $f
  else (
    let $name := 
      this:function-name($f)||
      "("||string-join($parms!this:function-name(.),",")||")"
    return (
      xdmp:eval("
declare namespace art='http://mathling.com/art';
declare variable $f external; 

%art:name('"||$name||"') function ($point as map:map) as xs:double {
  $f($point)
}", 
        map:entry("{}f", $f)
      )
    )
  )
};
        

Putting it all together, we can now emit the annotation from the noise function in Figure 15 as part of our metadata through a call to function-name, and get something like this:

noise:clamp(noise:min(noise:scale-bias(fbm:noise2),noise:curve(fbm:noise2)))
      

Here I have just shown the dynamic annotation including the chain of functions, but clearly the scheme could be extended to include information about constant parameters as well.^[5]

A low-tech alternative when these extension functions are not available is, of course, to manually emit the appropriate metadata. The risk is that since the metadata is separated from the actual function value, it may be wrong. It is too easy to allow metadata and data to drift apart when a manual act is required to keep them in sync: automated metadata is more reliable as long as it can be made accurate.

Lesson: Pay Attention to the Whole Publication Chain

The publication part of the standard workflow creates a new wrinkle: the problem of maintaining the rich metadata across conversion to a new format. Public sites where images may be shared generally do not permit the posting of SVG images (alas!). Converting the image itself is not usually a problem. There are plenty of conversion tools. I generally use InkScape to convert to JPEG or PNG files. The key problem is how to preserve the rich metadata.

Fortunately both JPEG and PNG have slots for metadata of various kinds. Unfortunately, the image tool support for metadata is weak. InkScape does a great job of conversion, but just drops all metadata on the floor. ImageMagick is less reliable for conversion of SVG, but it does at least have hooks for copying metadata. My solution is a bit of a Rube Goldberg device:

bin/saxon bin/getmeta.xqy +FILE=$fn > ${fn/.svg/.meta}
inkscape --batch-process --export-type=png --export-filename=- $fn | \
convert -comment @${fn/.svg/.meta} - ${fn/.svg/.png}
      

When publishing the converted work I have to manually (alas!) copy some of the signature metadata out of the extracted metadata file to accompany the work, since it is not readily visible in the same way it is in SVG. A similar manual process is required to produce alternative text descriptions for the images.

Figure 23: Various Workflows

Image description

The diagram shows seven boxes with flows between them. The main flow is from the box labelled XQuery (producing XML files) to XSLT (producing SVG files) then to Convert (producing PNG files) and finally to Publish. There are alternative outputs and flows. XQuery may produce raw image files (PPM) or SVG files which then flow directly into Convert, or it may produce PNG files which flow directly into Publish. A side flow takes PNG files and flows into VQGAN/CLIP which either produces an MP4 file, flowing directly into Publish, or PNG files, which flow through Enlarge and Sign before reaching Publish. Parameters are shown as being provided at each step in all the flows.

While most of my art runs through SVG, there are additional workflows that don't follow that path. Some workflows produce raw images (PPM format) directly or PNG (PNG) files directly. Other workflows involve taking a generated image and running it through some machine learning models with additional parameters. Let's look at some of the metadata capture issues of these two workflows.

When PPM files are generated directly, metadata can be embedded in comment fields directly. The API is a little different from the API for the standard workflow, but conceptually there is not much difference. Conversion also works slightly differently, because InkScape cannot convert PPM images. I use ImageMagick here instead. However, PPM images are bitmaps: arrays of pixel values. I still want to sign them, visually. The solution here is to take a conversion of the SVG signature path, scaled and coloured appropriately, and compose it with the converted image as the final step, again using ImageMagick.

When PNG files are generated directly, I use the Saxon Java reflexive mechanism to call the Java PNG APIs. These have hooks for embedding metadata, but they aren't actually implemented: metadata just vanishes, so more machinations are required.

What about the machine learning workflow? The actual steps are quite complex, and additional metadata is actually being produced at each step:

Let's focus on step four. We have metadata from the original SVG, metadata in the images produced from step two (which includes information about the machine learning process, but not from the original image), and metadata from the expansion phase (which includes information about parameters for that process). For that matter, we also have metadata about the images at each step of the process which the various tools also add (e.g. height and width). Capturing and combining all of this is another Rube Goldberg device, using exiftool to extract information from the various intermediate images, a little massaging with XQuery and shell scripts, and ImageMagick to slam it all into the final output.

While all this image conversion and hackery may seem like a niche concern, similar issues occur with other kinds of digital artifacts. For example, what metadata gets carried across when you convert a word processing document to a published PDF? What controls do you have over that process? What metadata can you add to convey information about that specific step in the process? Any process step that crosses tool or format boundaries is a place of danger for metadata preservation.

Lesson: Everything Starts as a Special Case

Processes are as varied as the human imagination. Just when you think you have everything captured in the framework, along comes another special case. It can take some time to figure out how to organize some new set of cases into an expanded framework for capturing them. That's OK. Sometimes the best you can do is cobble something together that works for today's problem. Things evolve over time. It is important to keep a manual escape hatch and be prepared to refactor later.

We've seen cases of this already. Originally every bit of metadata was captured manually. Then some of the fixed process metadata was captured automatically. Then the component metadata was captured automatically. What started as ad hoc functions with a set of parameters became a system of descriptors, captured automatically. What started with a set of named gradients and filters became a naming system allowing for more flexible combinations.

Lesson: Standardization Gets in the Way, Mostly

I should probably offer a word or two about a common thrust of a lot of discussion of metadata: standardization. All the properties mentioned here are ad hoc and idiosyncratic with their own naming scheme. Is there no place for reference to standards here?

The short answer: not really. As Laplace probably didn't say: I had no need of that hypothesis. The bulk of the information being captured here is for an audience of one: to the extent there is standardization, it is a standardization of a peculiar kind. Yes, "standardization" of naming conventions is helpful, but it doesn't have to reference any external set of properties and it isn't fixed. Refactoring and reimagining of the conventions happens all the time. Further, process metadata is wildly variable: trying to shoehorn it into someone's idea about how to represent such information (to the extent such a thing exists at all) would be an exercise in frustration. Using external standards increases the distance between implementation and instrumentation and, as I said before, that reduces the reliability of the metadata produced. It would be a net loss. It would, in fact, reduce the reusability of the information for the creator.

It is ever thus. Sticking to standards slows the creative process. Not sticking to standards reduces the global usability of the information. Sticking to standards increases the reusability of the information provided. Not sticking to standards decreases the amount of information provided.

Is there a place for using external standards for certain information? Perhaps at least the information that is targeted to external audiences, such as the signature metadata? Yes. It would be useful to capture at least signature metadata and possibly some organizational metadata in some more (globally) standard way. On the other hand, the specifics of the publication process drive which standards would be most useful. This is metadata from the third-party perspective, added after the process of creation to interface with the world, to stand between the creator and the consumer. As with everything in the publication phase of the process, there are external constraints and translations necessary. It isn't clear that picking which standard to use up front actually helps.