Balisage Paper: Knock Down This Wall

Introduction

Thank you all for coming. I am speaking to you today from land that is historically part of Kah’p’oo Owinge, also known as Santa Clara Pueblo in New Mexico.

On Monday, Tommie Usdin invited us into a secret garden [Usdin 2023]. It’s secret, and I imagine it as a walled garden because how else can a garden be secret? Also, there’s a long tradition of Christian iconography around the image of the hortus inclusus (or enclosed garden).^[1]

I hope that you have enjoyed our visit this week to this secret garden, but what I want to suggest now in closing is that perhaps the time has come to tear down those enclosing walls. I don’t want to tear up the garden, but I do want to knock down the wall or at least put a couple of gates in it.

And to explain why, I think I’m going to need to tell you a story. Some years ago, when I was a senior in college and avoiding whatever work I was supposed to be doing at the time, I was wandering around in the stacks of the main library at my university, picking up books that looked interesting; and I spent half an hour looking at a book on Indian logic. The preface explained that the author had for many years been hoping to write an account of classic Indian logic and how it compares with western logic derived from Greek practice. Eventually he had succeeded in putting a first draft together and had shown it to some friends, and they were enthusiastic about it. One of the nice things about finishing the first draft was that it gave him a better overview of the subject matter and allowed him to see that a different organization would be much better, so he set about the work of restructuring the presentation. He was writing it again from scratch when he was interrupted by some of his friends who had apparently despaired of his perfectionism and his unwillingness to call anything finished. They presented him with galley proofs produced by a typesetter to whom they had given the first draft, and they said, We’re going to publish this. Would you please proofread these galleys and write a preface?

This has been on my mind I think partly because it illustrates a tension that is visible in many places in the world between a desire to get something right — to get the right answer to a question no matter how long it takes — and deadlines — the desire to get some results, even if they are imperfect.

The Right Thing

The descriptive markup community has plenty of people who continue to seek the right answer to the question however long it takes. Patrick Durusau’s paper on Monday is, I think, a good example [Durusau 2023]. I have been hearing people worry about overlap since at least the late 1980s; I have done a fair amount of worrying about overlap myself. And some people are still trying, as Patrick’s paper illustrates, to find a solution that works as generally, and is as convincing for the general case, as XML is for the case where you only really care about one hierarchy. They are trying to find some solution that has a convincing story about a serialization form and a data structure and the conception of a document vocabulary as a document language for which there could be a grammar and for which you can imagine some kind of validation. Patrick’s answer may or may not be the general solution we’ve been looking for. I’m sure the discussion will continue. Of course, many people have solved this problem in a purely pragmatic way; if they had deadlines, they chose an approach, and they did it.

But even if what you really need is a result, things can change — things can get better. Amanda Galtman demonstrated a specific technique using xsl:accumulator that can help a lot with the kind of discontinuous elements that sometimes trouble people [Galtman 2023].

Of course, looking for the right answer no matter how long it takes works most easily when you don’t have a deadline. I talked the other day about finally finding a solution to a practical problem that arose in generating an ebook for Frege [Sperberg-McQueen 2023], but the first time I thought about that ebook was 2014. And I have had the luxury of being able to spend off-and-on nine years thinking about it, before iXML came along and I realized that it was the key to my solution.

Not everybody has that luxury because sometimes you do have a deadline, and when you do have a deadline, quite often you just need results, no matter how you get them. I have always thought that processing instructions in both SGML and XML are a signal that the designers of those languages included people who knew that sometimes you might need to resort to impure methods to get the results, but also people who cared enough about the difference between pure methods and impure methods to want to mark the places where you had resorted to impure methods, to processor-specific instructions and so on, in order to get your results. If you mark those places, you give yourself the chance to go back later and fix things to work with purer methods.

Getting to Results

Sometimes, of course, results matter, deadline or no deadline. Some years ago, I was at dinner with a bunch of people in the XML Schema working group; and the discussion turned to artificial intelligence and to the shift in artificial intelligence that Elisa Beshero-Bondar talked about, away from symbolic computation and towards purely statistical computation [Beshero-Bondar 2023]. And somebody said, You know, it’s really kind of a shame because with symbolic AI once a problem is solved, the result represents insight; the problem crystallizes things, and it presents an advance in human understanding. And Paul Biron, one of the editors of the data types specification, who worked at that time for a very large healthcare organization, said, It doesn’t matter in the least to me. If a black box that we don’t understand will give us better patient outcomes, I would much rather have better patient outcomes than better insight that doesn’t lead to better patient outcomes. What I care about is the result, not the process. And it is a fact that symbolic AI produced some really nice results, but statistical AI has produced more results and better results — more impressive results.

When Things Don’t Go as Expected

Sometimes I think that the shift in AI research from symbolic to statistical computation is just one instance of a much larger and more complex pattern. What happens when there is a group of people who expect things to go in one way and find that they don’t go that way? What happens then? This is particularly visible and interesting in cases where you have a community of people who would like to change the world — and who expect to change the world — and then encounter a certain recalcitrance in the world as the world refuses to change in the way that they expected. Early AI researchers thought that general machine intelligence was just around the corner; from the late 1950s on, there were predictions that general intelligence was maybe five years away. But the five-year interval never got shorter and eventually became a joke, until finally AI as a field said We’re not getting anywhere this way; we need other approaches.

For a long time, socialists and Marxists the world over expected world revolution; world revolution was just around the corner. And it is interesting to look at the history of the various flavors of socialism to see how different people react to the fact that world revolution hasn’t happened and now looks less likely than ever. Sometimes people change gears — they change paths — the way the AI field did. The old ideas have failed; we’re going to try something new. So there are plenty of Communists and Bolsheviks who became neoconservatives.

When I first learned SGML and learned about descriptive markup, I found it very hard not to think that world revolution was just around the corner. It was so obvious that this was a better way to do things, so obvious that it was useful both in the short-term and in the long-term that surely everybody will soon see that this is a better way to do things. Surely, very soon the whole world will be using SGML, because it’s so obviously the right thing.

But … Life Happens

It’s important to remember that no single technology or family of technologies is ever going to be the only thing in the world. However good the technology is, complications will arise, complications of very different kinds, sometimes non-technological.

Albania

One of the big challenges when you expected the world to change and then it didn’t, is that when you are thinking about how the world is going to change and how things are going to be, you make certain plans and adopt certain behaviors, and then when the world doesn’t change the way you expected, you find yourself confronted with the continued existence of people and institutions whose continued existence was not part of your original plan or expectation. In the case of AI, those were, for example, problem areas that resisted solution. In the case of the Communist International, they were the continued existence of non-socialist countries. For XML people who expected the entire world to start using XML, it is perhaps the continued existence of non-XML formats and non-XML users.

There are several things that people can do in this case. Sometimes there are people who refuse to give in, who just continue working for the original goal. Leon Trotsky never gave up on world revolution; that was one of the serious substantive policy differences between Trotsky and Stalin. Trotsky wanted world revolution, and Stalin said, you know, we have control of this country; we really need to try to make it work within this country. And socialism within one country required a certain amount of attention that Trotsky didn’t want to give it because Trotsky wanted to focus on world revolution. But eventually, for most purposes, Trotsky became irrelevant to the way the world went.

But even under Stalin, the Soviet Union took an attitude that, I guess, could be described as hostile co-existence: the non-socialist world continues to exist, but it won’t forever because we are in a contest; ultimately one system will outlive the other. And you get remarks like Khrushchev’s remark when he visited the UN in 1960, We will bury you which sounded really, really threatening to many Americans because it sounded as though Khrushchev had plans to bring about the state of affairs in which it would be necessary for him to bury us, even though in reality, as I learned years later, he was just citing an old Russian proverb that means We will outlive you.

At this point, I find it necessary, however, to remind people that world revolution was not necessarily part of everyone’s original plan for XML. Converting everyone to XML is not one of the goals in the XML specification. The database management people showed up at W3C and at XML meetings not because anybody told them to, but of their own accord. I remember, again, a Working Group dinner at which I said, Don’t take this wrong. I don’t want to offend you, but why you here? What does XML have that you need? You know, XML is for documents, and databases are really not document-like. And I’m happy that you’re here, but I don’t understand why. And they said, Well, we have interchange problems. When we need to move data from one database to the other, we have trouble. And I said, What do you mean? You’ve got comma-separated values; surely that’s all you need. I mean, I’m not really a database guy, but I have worked enough with relational databases to know that the tables fit really nicely in common-separated values. And they said, Yes, but comma-separated values are responsible for about half of our of our support costs because, well, for one thing, the format looks so simple that many programmers don’t bother to look for a library. They figure they can write it on their own, and they do, and they don’t always get it right, so they have quoting problems and white space problems. And the other thing is that, even if they got it right, comma-separated value formats have no place to identify the character set being used. So the fact that XML was created in part by character set geeks who had been struggling with character set issues for years and therefore built an encoding declaration into XML was partly responsible for the interest of database people in XML. Because XML had a way to declare the character encoding.

Now, being part of a movement that may change the world is a lot of fun. It’s very satisfying; it gives context and significance to our lives. And giving up on the idea that it’s going to change the world can be psychologically very difficult. But I submit that being part of that kind of movement can give our work meaning whether we end up taking over the world or not. I’ll also note that Charles Goldfarb, for what it’s worth, was always careful, at least when I heard him speaking, not to imply that SGML was out to conquer the world or that SGML must conquer the world or count as a failure.^[2] On the contrary, he sometimes positioned SGML and descriptive markup as a sort of niche technology for people who could not afford to take sides in battles between proprietary formats backed by large organizations that they did not control. I remember a talk in which one slide read When giants do battle, choose a different location for your picnic.

There’s a threat to all of us when organizations that are larger than we are are fighting each other and we risk becoming collateral damage. But there’s also a threat when others are competing with us and see our non-existence as advantageous to them. Even if we don’t seek world domination ourselves, they may be seeking world domination. After all, a lot of people have learned the lesson of the network effect and think This world will be so much better if our technology is adopted universally. And any technology that’s not theirs is a threat to them.

So what do you do if you find yourself living in a world full of threats? One approach is to build a wall: ignore them, defend yourself, and pull back your attention to focus on what is close at hand. In a political environment, maybe the best example of this is Albania which was extremely isolated not just from the non-socialist west, but also the rest of the socialist world. East Germany also spent a lot of resources trying to isolate itself.

On a smaller scale, we can build a wall around our garden to make it safe for us to tend our own garden, as Voltaire put it. But sometimes the threats that led us to shield ourselves from the outside world turn not to be not quite so threatening as we thought. You may remember that ominous picture of a very threatening-looking snake in Tommie’s slides [Usdin 2023]. That was a garter snake, of no conceivable threat to any human being attending this conference or not attending this conference.

Sometimes we decide there’s really no threat. We can live in the world; we can interact with the rest of the world in the same way that Scandinavian Social Democrats do not find it necessary to shield themselves off from non-socialists. They engage with others in parliamentary democracies, they enter into coalitions, they form governments. They’re just another political party. They have certain policy preferences, but they don’t regard the rest of the political spectrum as an enemy with whom it is dangerous to enter into any commerce. In a similar way, there are XML users and tools that engage with other formats. And we can learn from their experience.

Unicorns

You know, there is another reason I was thinking about that Indian logician. The real reason I was thinking about that Indian logician is a problem related to existential and universal quantification. By universal quantification, I mean logical sentences like All human beings are mortal; and by existential quantification, I mean sentences like Some human beings are mortal. And it sticks in my mind, in part, because the Indian logician taught me something, in that thirty minutes I spent reading a book that I will never find again (because I can’t remember his name and I don’t remember enough about the book ever to find it again).

He said that the treatment of quantification is one of the crucial differences between Indian logic and Greek logic. The sentence All human beings are mortal is a stronger sentence than Some human beings are mortal, and the stronger sentence entails the weaker sentence. So, in Indian logic, if the universal quantification is true, then an existential quantification will also be true.

Greek logicians made an opposite choice. They said, consider the sentence All unicorns have one horn. Well, that’s true because there are no counter-examples. There are no unicorns that don’t have one horn. And so they established the principle that if the class involved is empty, then the universal quantification is always true.

But an existential quantification — a sentence like Some unicorns have one horn — that, the Greeks really wanted to be true only if there existed at least one that had one horn. And that means the truth of the universal statement does not imply the truth of the existential statement. And in general, in Western logic, an existential statement is taken to entail, as the term existential suggests, the existence of members of that class.

I think of it as natural that we should call a statement using the word some an existential statement, but that’s only because I’ve spent a lot of time thinking about it within the framework of Western logic. The word some and the word existence have no etymological tie, and their semantic tie is, as the Indian author made clear, the doing of Greek logic. And he described the difference in approach this way:^[3] he said, two logicians — an Indian and a Greek — went for a walk. And there were sharp rocks, and they hurt their feet on the rocks. And the Greek logician said, You know what we should do? We should get a big piece of leather, and we should cover the entire globe, so when we go for a walk, there is always a layer of leather between our feet and the rock, and we can’t hurt our feet. And the Indian said, Having leather between the rock and our feet would be a good idea, but I think it’s simpler if we just wear sandals.

Conclusion

Invisible XML is focused on moving things into XML. And I think there is continuing pain and thus a continuing opportunity in the area of export from XML. iXML simplifies import. You could always get stuff into XML; you could write a parser in XSLT or XQuery, using the facilities that are built in. But iXML makes it a lot easier because all you have to do is specify a grammar and an ixml processor will parse your input into XML for you. I wonder if there is an opportunity for a notation that could similarly define export so that we have a declarative notation that we could compile to XQuery or XSLT that would produce a suitable export into a text stream that matches the grammar or obeys the rules that we’ve laid out for the notation.

It seems to me that there are two difficulties in data interchange — I thought this for a long time; this was part of our thinking when we were working on the Text Encoding Initiative — one problem is understanding the structure of the data you acquired from somebody else. Where are the field boundaries? Do some things contain other things? Can things repeat? What is data here, and what is delimiter? And stuff like that. And the second arises typically after you have answered the first question and understood the structure of the data; that’s the problem that arises when you discover that the information you wanted is not there, at least not in the form that you wanted.

Maybe they analyzed the world in a different way so it’s going to be more complicated to get the information you want. Maybe the information you want just isn’t there because, oddly enough, their view of the world is different from your view of the world, and the things that they are interested in turn out not to be the things that you’re interested in. The TEI, I always used to say, can help with the first, but it really cannot help with the second. The only way to help with the second would be to prescribe that everybody has to care about the same things. And that’s not something most people want to do, or even if they do want to do it, it’s not something that would ever succeed; we’re not ever all going to be interested in the same things.

And analogously there are two difficulties in moving data into and out of XML. The first is just a difference in format which we can solve with better import/export tools. We can get data into XML. But the second is the difference — I’ll call it a difference in worldview — that people who created that external data format don’t think the way we do. The people who use it often don’t think the way we do. And so what you get will be XML, but it will not necessarily be XML that uses the principles of descriptive markup. And it’s important to be aware of this, so we can prepare ourselves. There are cases where we will have to relax our concerns about that second thing. We are in a position to make it easier to move data into and out of XML; we are not in a position to make the entire world do descriptive markup. And just getting the data into XML helps a great deal.

If you doubt that, ask yourself: would you rather work with a Word 95 binary file or with the XML that you can extract from a .docx file? I submit to you that no one who has ever worked with either of those, and absolutely no one who has worked with both of those formats, will have any doubt that they are much happier working with the XML than with the binary format. If descriptive markup is as helpful as we say it is, things will always be easier with descriptive markup than without it, and the work we do will be better. And if XML is as helpful as we say it is, then things will be better with XML than without it, and our work will be better.

Gardens are very nice places to retreat from the world, but having rested here, we need to go back out into the world in part because we have things to offer the world. If we tear down the wall or at least punch a couple of gates in it, there are new places we can go. But be careful. The world out there is full of stones, and we’re not going to manage to cover it all with leather. So we’re going to need leather on our feet; put on your shoes. Let’s go some places, and then come back next year and tell us where you’ve been.

Thank you for attending Balisage.