Balisage Paper: Extracting Funder and Grant Metadata from Journal Articles

Funding Data in STM Articles

Over the past few years we've done extensive work to develop techniques to identify and extract metadata from inside various textual collections, including grant information in STM (Scientific, Technical and Medical) journal articles.

While grant information might be found almost anywhere in an article, it is often contained in either a section of several paragraphs identified as an acknowledgement section or in footnotes near the beginning of articles. The work described in this article assumes that such a section exists and has been identified. Following is an example of a typical acknowledgement section:

It's typically in prose, may identify one or more funding, and may or may not identify specific grants or other identifying characteristics. It may also acknowledge other sources of assistance who were not necessarily funders of the project. While simpler constructions refer to only one or two funding sources and follow regular grammar, most such sections are more complex. Also, since many authors are not totally fluent in English, the grammar is sometimes convoluted.

Humans are quite good at reading these sections and identifying the relevant information. But the process is labor-intensive, expensive, and because it relies on judgement calls, doesn't always provide consistent results. The increasing speeds required in the publishing industry, and the constant pressure on costs, cries out for automation to speed up the process, reduce manual costs, and provide consistent results. Automating the process also holds the promise for continual improvement as the feedback is used to improve the process.

Automating language analysis of course isn't easy to do reliably because of ambiguities in the use of languages, lack of grammatical knowledge, and the absence of standardization in how people use language ("if everyone followed the rules, this would be easy, alas..."). Also, while this article focuses on English language sources, the techniques we describe can be adapted to other languages, though the analysis of grammatical constructs would need to be adapted to the specifics of the language.

Worth noting is that it could contain multiple institutions, several grants within an institution, and may also include non-funding support from various sources. Let's unpack this even more with the question, Why is extracting funder and grant information important? Here are some answers:

Let's define some terms in our process:

Methodology

The following provides an overview of our methodology; it's not intended as a machine learning tutorial, but rather to provide the context for what we were able to accomplish, and where we think there is room for further development.

Included is discussion of the following methods we applied in order to improve the initial results:

The document set we used for this analysis consisted of a corpus of 1000 xml journal articles in which we identified, and segregated, the sections which would be most likely to contain funding data, which was identified as the Acknowledgement section or something similar. While journals from various publishers have some variance in how they handle this, most do have the information in some identified section.

Our approach, using context-dependent tokenization of the text, was to parse the token stream to find sentences relating to grant funding. For each document the section containing the words: grant, trust, fund, and support were extracted, stripped of any XML tags and used to create a free-text data stream. ^{[1] [2]}

Let's break down how we parse the token string. In the example shown, figure 1, having identified the appropriate sentence because it contains the word "Grant", we would then be looking for a start word (a token), like "supported", which possibly identifies that a supporting organization follows. In the example we're looking at, the organization providing the support follows somewhere down the line, and the funding instrument and recipient follows in due course. Typical starting words include "grant", "support project", "award", "contract", "agreement", and "scholarship". Once you've identified the start word, it will define the context of the rest of the sentence, and lets you parse the remainder of the sentence based on the following rules:

Figure 1

Figure 2

Our primary goal was to identify all grant funders and the grant numbers within the text, with a secondary goal to locate grant recipients, which we searched for with logic similar to the above. Internal divisions such as punctuation and parenthesis aided in the transition between "grant text" and "non-grant text". In addition to looking for the first capitalized word, we did some more analysis using pattern matching and textual context. We also used the start words as stop words for the prior segment. This was to let us know that we have probably identified all the information for that funding source and that we were probably now looking at another funding source. At the end of this analysis we would expect to have found a set of candidate phrases where each phrase had a candidate grant funder, zero or more grant numbers, and zero or more recipients.

Classification

Classification is a key methodology we used next. The above process alone correctly identified over 50% of the granting institutions, along with grant ID's, with a lower yield for recipients. These results, while impressive, were not sufficient for a production setting. Our goal was to get the automated process into the 80-90% area for the process to be considered useful. Getting into the 90-95% was the ultimate goal.

In order to improve precision we added a machine learning classifier trained on proper names taken from our corpus of documents. The internal functioning of a classifier is outside the scope of this article, but a good overview is available in this article: https://en.wikipedia.org/wiki/Support_vector_machine. The basic idea was to identify which combination of words was likely to indicate it being a funding sourcing. This was done by working with training sets, which we could then manually analyze and determine whether the selected items fit into the category (valid funding source) or not. If the training set was sufficiently representative, than the computed ML algorithms should correctly predict the correct answers for the other sets that would be analyzed, which would then be an automated process.

The training of the classifier was based on a set of natural language processing tools from R (a set of open access software tools from the The R Project for Statistical Computing, www.r-project.org. The first step was to convert the phrase into a "bag of words" by dividing the textual stream into a set of words. "Bag of Words" is a term used in ML when you want a list of all words appearing in a set, but don't care what the order might be. The words are stemmed to normalize and simplify the analysis (e.g. walk, walker, walking, walked all have the same stem) and common stop-words (a, the, and, etc.) are removed. A term frequency vector was calculated, and words with low frequency, in our case those occurring in less than 10 documents, were removed from the analysis. Surprisingly, after this analysis, on 1000 documents, only about 50 words remained.

The subsequent words were used to build a document term matrix where each column was either 0 or 1 to indicate the presence or absence of a word and each row contained a place holder for one of the 50 words used in the analysis. From the above data, a Support Vector Machine (SVM) was trained. Background on SVM is available at https://en.wikipedia.org/wiki/Support_vector_machine.

The purpose of the classifier, once trained, is to return an evaluation for each instance, of whether the instance is a funding institution or not. The model based on initial testing showed an F1-measure of .86; the F1-measure is considered a measure of "goodness of fit", and without getting too deeply into the statistics, tells that the chances that we guessed correctly is about 86%.

Next Steps

To further improve the accuracy of our results, the next step would be to compare them to a known list of granting organizations. If there was a complete, authoritative database, we probably wouldn't need some of the sophisticated techniques we've discussed. But that doesn't exist today. The Funder Registry maintained by Crossref (formerly known as FundRef) is considered the most authoritative, and while it is being regularly improved, it is not yet complete and there are some ambiguities.

By using the result of our analysis, along with the Funder Registry, we have gotten results in the 90%'s, and anticipate being able to improve on that.

In addition to improving automated results, there is further work we are doing to disambiguate mnemonics, in which a designated funder could be one of many, and to better identify grant recipients, who are not often not fully identified in funding acknowledgment.

The goal going forward will be to increase yield further, and with better precision, while reducing false positives. Our plans are to achieve better calibration with larger test sets, improvements in machine learning and in our parsing algorithms, improvements in the Funder Registry and authority files.