Balisage Paper: Eating Your Own Dog Food

Version One

The very first version of the PayPal button took me all of fifteen minutes to copy, paste, and add to the summary HTML.^[4] The code example I grabbed from the PayPal site added script tags:

<script src="https://www.paypal.com/sdk/js?client-id=sb"></script>
<script>paypal.Buttons().render('body');</script>

And the form itself:

<form action="https://www.paypal.com/cgi-bin/webscr" method="post">
  <input type="hidden" name="cmd" value="_xclick"/>
</form>

PayPal also suggested adding variables to send more information (currency, VAT, etc) to PayPal:

<INPUT TYPE="hidden" NAME="currency_code" value="GBP"/>

Unfortunately, after spending some time trying to add the amount to a hidden variable, I decided to read the documentation and realised that was one of the things you couldn't add.

Version Two

As an alternative, Paypal offers an online tool that can be used to generate custom PayPal button code. All you have to do is define your parameters, including amount, currency, etc, hit Generate, and this sort of thing will result:

<button id="early-bird-sandbox">
    <form
        action="https://www.sandbox.paypal.com/cgi-bin/webscr"
        method="post"
        target="_top">
        <input
            type="hidden"
            name="cmd"
            value="_s-xclick"/>
        <input
            type="hidden"
            name="hosted_button_id"
            value="PM4DRL6BX3226"/>
        <input
            type="image"
            src="https://www.sandbox.paypal.com/en_US/GB/i/btn/btn_buynowCC_LG.gif"
            border="0"
            name="submit"
            alt="PayPal – The safer, easier way to pay online!"/>
        <img
            alt=""
            border="0"
            src="https://www.sandbox.paypal.com/en_GB/i/scr/pixel.gif"
            width="1"
            height="1"/>
    </form>
</button>

While easy enough to use, this resulted in a fair number of buttons; for example, a £150 amount payable would require generating two sets of code, one for the sandbox and one for the live environment. Every new amount, then, will result in two additional pieces of code, one for testing and one for live.

The generated buttons also didn't really solve the problem with custom discounts—we'd have to generate a button for each and every discount when the need arose. Nevertheless, I wrote some XQuery that used XPath expressions to pick the right button element from a button lookup XML document containing every possible button type and environment combination. Here, for example, are the buttons required for the sandbox:

<buttons>
    
    <!-- Early-bird sandbox -->
    <button id="early-bird-sandbox">
        ...
    </button>
    
    <!-- Academic sandbox -->
    <button id="academic-sandbox">
        ...
    </button>
    
    <!-- PC sandbox -->
    <button id="pc-sandbox">
        ...
    </button>
    
    <!-- Full sandbox -->
    <button id="full-sandbox">
        ...
    </button>
    
    ...
    
</buttons>

This was cumbersome, to say the least. Thankfully, I found a third option.

Version Three

In the documentation, there is a reasonably thorough, step-by-step description of how to integrate PayPal into your website, complete with code examples in a number of languages^[5] (see PayPal Checkout - Basic Integration). What tickled my imagination was that the example code included an amount:

<script>
  paypal.Buttons({
    createOrder: function(data, actions) {
      // Set up the transaction
      return actions.order.create({
        purchase_units: [{
          amount: {
            value: '0.01'
          }
        }]
      });
    }
  }).render('#paypal-button-container');
</script>

Now, I don't do JavaScript so a lot of what's in there is magic to me,^[6] but clearly, it should be possible to express 0.01 as a variable. Exciting! Shouldn't it be possible to inject some XQuery into that thing, like so:^[7]

purchase_units: [{{
  amount: {{
    value: '{$amount}'
  }}
}}]

Following the example, I added return pages (the actions.redirect() calls) and various other stuff that made testing and deploying the code easier (see section “Sandbox Vs Live” and section “The Admin Page”). This is what my JavaScript/XQuery end result looks like:

<script>
    
  paypal.Buttons({{
    createOrder: function(data, actions) {{
      return actions.order.create({{
        purchase_units: [{{
          amount: {{
            value: '{$amount}'
          }}
        }}]
      }});
    }},
    onApprove: function(data, actions) {{
      return actions.order.capture().then(function(details) {{
        // alert('Transaction completed by ' + details.payer.name.given_name);
        
        actions.redirect('{$muk:config//domain/text()}/exist/rest/db/apps/MUK-reg/xq/thanks.xquery?tmp-file={$registered}');
        // Call your server to save the transaction
        return fetch('{$muk:config//domain/text()}/exist/restxq/paypal-transaction-complete', {{
          method: 'post',
          body: JSON.stringify({{
            orderID: data.orderID,
            custom: {string($uid)}
          }})
        }});
      }});
    }},
    

  onCancel: function(data, actions) {{
    actions.redirect('{$muk:config//domain/text()}/exist/rest/db/apps/MUK-reg/xq/cancel.xquery');
    }}
  }}).render('#paypal-button-container');
  
</script>

Note the custom: {string($uid)} key-value pair that sends the user ID of the registration as a custom variable to PayPal; this is returned to the summary page (thanks.xquery, above) when confirming, along with the PayPal transaction ID.

And this is actually most of what we need, except for a tiny little detail where we find out if the payment was successful.

You Talk and I Listen

A listener, apparently, is something that listens to incoming calls and, when catching one, does things to the call and responds in some predetermined way. A call, in this case, is an http POST request that we, after acknowledging it with an empty POST, return to the sender just the way we found it.^[9]

And this is where my ignorance started to show.

PayPal, it turns out, provides an IPN Simulator, essentially something that sends an IPN string to your listener and checks if the response is what it should be. It's tremendously useful. The problem was that I had no idea what a listener looked like.

I tried a few variants, using this web form:

<form
    target="_new"
    method="post"
    action="http://localhost:8080/exist/rest/db/apps/test/test.xquery"
    enctype="multipart/form-data">

    <input type="hidden" name="payer_id" value="LPLWNMTBWMFAY"/>
    <input type="hidden" name="payment_status" value="Completed"/>
    
    <!-- code for other variables to be tested ... -->
    <input type="hidden" name="custom" value="my_info"/>
    
    <input type="submit"/>
</form>

The URL pointed to this test XQuery that would simply echo what it caught:

xquery version "3.1";
declare namespace exist = "http://exist.sourceforge.net/NS/exist";
declare namespace xmldb="http://exist-db.org/xquery/xmldb";
declare namespace request="http://exist-db.org/xquery/request";
declare option exist:serialize "method=xml media-type=text/xml indent=yes";

let $post-data := request:get-data()

return
<post-data>
   {$post-data}
</post-data>

I was quickly able to catch whatever the web form sent to the XQuery. However, when pointing the IPN Simulator to the XQuery, I registered nothing and the simulator reported a fail. No matter what I did, it always failed.

Desperation

In my desperation, I did what people do,^[10] I asked for help. Adam Retter, one of the core eXist programmers, the inventor of RESTXQ, and an all-around good guy who will forget more about XQuery than I will ever know, replied to my email within hours, attaching an IPN listener. He didn't have an XQuery parser available but nevertheless provided something that, with minor fixes, parsed and worked.

Adam's listener, of course, was RESTXQ-based, and while I knew about the concepts behind it, I had never written any myself and it took me a few days of testing to understand how it worked.

I hooked up the listener to my eXist-DB test setup and managed to get the IPN Simulator to complete successfully. In other words, Adam's listener quite clearly responded with the required two messages—an empty POST 200 followed by a copy of the IPN message.

However, I still couldn't extract any part of the IPN message locally.

A Simpler Solution

After some (= a lot of) research,^[11] it dawned on me that the difference between my form and the IPN Simulator is what they send. PayPal sends an application/x-www-form-urlencoded POST request, which apparently means that the message body is a bunch of parameters, like this:^[12]

verify_sign=AtkOfCXbDm2hu0ZELryHFjY-Vb7PAUvS6nMXgysbElEn9v-1XcmSoGtf&
payer_email=gpmac_1231902590_per%40paypal.com&
txn_id=61E67681CH3238416&payment_type=instant&last_name=User&
address_state=CA&receiver_email=gpmac_1231902686_biz%40paypal.com&payment_fee=0.88&
receiver_id=S8XGHLYDW9T3S&txn_type=express_checkout&item_name=&
mc_currency=USD&item_number=&residence_country=US&test_ipn=1&
handling_amount=0.00&transaction_subject=&payment_gross=19.95&shipping=0.00

Now, if I had read up on the subject properly, I might have been able to solve the problem earlier. The fact is that the message is a bunch of key-value parameters, not a body per se. I'm not that clever, however, and so understanding what actually went on took a far longer time than I thought.

To test my understanding, I dabbled in ways to read the parameters and eventually wrote a minimal listener to catch the PayPal confirmation only, ignoring the intricacies of the IPN process and thinking that the reason I wasn't able to catch the message was due to the inherent asynchronicity in http communication:

declare
    %rest:POST("{$body}")
    %rest:path("/paypal-transaction-complete")
function pay:listener($body) {
    
    let $json := fn:parse-json(util:base64-decode($body))

    return
    xmldb:store($muk:transactions-collection-uri,concat('uid-',$json?custom,'.xml'),
    <transaction tstamp='{current-dateTime()}' 
        orderID='{$json?orderID}'
        userID='{$json?custom}'/>),
    pay:all-good()

    
};

declare function pay:all-good(){
    (: Reply with an empty 200 response to indicate to paypal the IPN was received correctly. :)
        (
            <rest:response>
                 <http:response status="200" reason="OK"/>
            </rest:response>,
            ""
        )
};

This is a ridiculously simple listener; all it does is to save the message locally in XML (rather than JSON) format and send back a POST OK 200. It is the minimum functionality, what I had to have^[13], and it tied beautifully to the summary page with the button.

All Together Now

So, now I had a registration summary HTML page^[14] that presented the registration information from the XForm and integrated a PayPal button with a custom amount and the current user ID as input, and also included return pages—functionality to be triggered after completing (or cancelling) the transaction.

All I had to do now was to tie it all together:

Sandbox Vs Live

As it is a really bad idea to develop and test all this in a live environment, PayPal provides a sandbox environment, essentially a copy of the live environment but with means to create fake sellers and buyers, complete with fake credit cards and other means of paying.

Figure 2: The Sandbox Is Making Some Real Money

The way to keep the two separate is simply to point the button to two different URLs using a variable, $url:

<head>
    ...
    <script
        src="{$url}">
    </script>
    
    <script>
        <!-- PayPal button JavaScript -->
    </script>
</head>

Locally, I also needed to be able to store the registration data in two separate locations in eXist:

Figure 3: Locations for Registration Data

When switching between live and sandbox, I had to change the button URL but also redirect any read and write operations to the appropriate locations. I was willing to bet that I'd forget at least one sooner rather than later, so I added a config file that provided paths and settings for both environments, parameterising every path in every function so they would always be able to look up the currently applicable paths:

<config>
    <!-- This is a MUK Registration app config file -->
    
    <!-- MUK info -->
    <business>
        <name>Markup UK Conferences Limited</name>
        <email>info@markupuk.org</email>
        <pid>...</pid>
        <bank>HSBC</bank>
        <account>...</account>
        <client-id>
            <sandbox>...</sandbox>
            <live>...</live>
        </client-id>
    </business>
    
    <!-- eXist-DB Domain -->
    <domain>http://localhost:8080</domain>
    
    <!-- Data root for MUK-data and subcollections -->
    <data-root>/db</data-root>
    
    <!-- Collection names in eXist -->
    <collections>
        <root>MUK-data</root>
        <sandbox>sandbox</sandbox>
        <live>live</live>
        <sub>
            <tmp>tmp</tmp>
            <transactions>transactions</transactions>
            <data>data</data>
            <pdf>pdf</pdf>
        </sub>
    </collections>
    
    <!-- Current env -->
    <current>sandbox</current>
    
    <!-- The currently *selected* path (live or sandbox, basically) -->
    <data-uri>/db/MUK-data/sandbox</data-uri>
    
    ...
    
</config>

Now, all I had to do was to change a single field in the config (<current>sandbox</current>; the sharp-eyed reader will wonder about the data-uri field in the config fragment, but this is changed inline by an XQuery Update instruction, triggered by an update in the admin XForm described in section “The Admin Page”).

In the HTML/XQuery in eXist-DB, $url is set like this:

let $clientid := if ($muk:config//current='sandbox')
                  then (string($muk:config//client-id/sandbox))
                  else (string($muk:config//client-id/live))

let $url := concat('https://www.paypal.com/sdk/js?client-id=',
                    $clientid,
                   '&currency=GBP&intent=capture&commit=true')

Here $clientid is the ID that PayPal uses to identify where the URL points—sandbox or live. $muk:config is the above config file, and so, $muk:config//current tells us what environment the app is currently running in.

The Admin Page

Yet another complication was that if I wanted to move the application between servers—for example, from http://localhost:8080 on my local PC to the Markup UK server URL—it also had to go into the config file, and it also had to be changed when going live. As shown in the config file fragment, above, the domain is set in <domain>http://localhost:8080</domain>.

And of course, I also needed the config to include the registration types and amounts, as well as any special discounts:

<!-- Pricing -->
<pricing>
    
    <types>
        <type use="yes" inc="204" exc="170">
            <name>Student (£204 incl VAT)</name>
            <value>academic</value>
        </type>
        <type use="yes" inc="258" exc="215">
            <name>Early Bird (£258 incl VAT)</name>
            <value>early-bird</value>
        </type>
        <type use="yes" inc="336" exc="280">
            <name>Full (£336 incl VAT)</name>
            <value>full</value>
        </type>
        <type use="yes" inc="204" exc="170">
            <name>Programme Committee Member (£204 incl VAT)</name>
            <value>pc</value>
        </type>
        <type use="yes" inc="0" exc="0">
            <name>Speaker (£0 incl VAT)</name>
            <value>speaker</value>
        </type>
    </types>
    
    <discounts>
        <discount use="yes" inc="243" exc="203">
            <label>PRAGUE2019</label>
        </discount>
        <discount use="yes" inc="1.5" exc="1.25">
            <label>TEST</label>
        </discount>
        
    </discounts>
    
</pricing>

Finally, rather than updating the config file directly, I added an XForm for the purpose. The following sets the domain, environment and the collection names:

Figure 4: Admin Page, Basic Configuration

And here's a GUI for setting the pricing:

Figure 5: Admin Page, Pricing Settings

Note the buttons that allow us to add and delete custom discounts.

Programming

I still claim I'm not a programmer, but I can't deny I'm leaning more towards an, um, developer kind of mentality. For this paper, I was tempted to add a section on modularising, testing, and refactoring, all of which I began to appreciate in earnest while writing the app, but really, that section would have been out of scope for this paper.

You could easily argue that XSLT, XProc, and XQuery are all about programming, just as writing schemas is, but the difference to me is about the basic paradigm—the core technologies used here are all based on a single paradigm, a declarative approach common to pretty much every XML markup technology I use.

Odds and Ends

A couple of odds and ends: