Balisage Paper: White Hat Web Crawling: Industrial-Strength Web Crawling for Serious Content Acquisition

Overview

Vast amounts of business-critical information appear only on public websites that are constantly updated to present both new and modified content. While the information on many of these websites is extremely valuable, no standards exist today for the way content is organized, presented and formatted, or for how individual websites are constructed or accessed.

This creates a significant challenge for companies that require data sourced from these websites in a timely manner, which they need downloaded and structured to support business practices and downstream systems.

The paper will focus on specific impediments that we typically encounter and tactics we’ve adopted to overcome them in the process of creating a streamlined, automated process to crawl websites, scrape content and metadata, and transform the content into a standardized XML format. Our comments and recommendations are based on having successfully traversed hundreds of variated, multilingual, multi-platform, global websites.

We will elaborate on our methodology and bots used to facilitate high-volume data retrieval in a variety of source formats (HTML, RTF, DOCX, TXT, XML, etc.), in English, European and Asian languages, with varying organizational approaches.

White Hat versus Black Hat - Good Guys or the Bad Guys?

While Web Crawling sounds somewhat nefarious, there is an important white-hat side to it. Much original source material today appears only on the Web. For many government agencies and various NGO’s, the web version is the “document of record,” the most current version available, and where you are referred when you make inquiries regarding reports, articles, whitepapers, etc.

While there are many tools available to handle the basic crawling and scraping of websites, they mostly work on one website at time. Analyzing and traversing volumes of complex websites - somewhat like developing autonomous vehicles - requires the ability to adapt to changing conditions, across websites and over time. The presentation will examine the thought processes behind our approaches, including website analysis, techniques to detect and deal with website and content anomalies, methods to detect meaningful content changes, and approaches to verifying results.

Why Do We Need This Information?

There is a vast amount of data available on websites, from informative to entertaining to legal, with critical content that services a wide variety of purposes, depending on the business need. The most common endgame is the normalization, decomposition, and transformation of the information into a structured format to power derivative databases, data analytics platforms, and other downstream systems.

Why All the Fuss?

It is estimated that there are 4.52 billion webpages out in the wild (http://www.worldwidewebsize.com/). Many of these are maintained by webmasters who are certain that their architecture for running a web website is the best one, as opposed to the guy one page over.

Of course, it would be nice if all websites offered a convenient, reliable method to download and monitor their content, but most do not do so. It would also be helpful if the different websites complied with standards to make new and modified content easier to find and extract, but no such luck; the variations are endless and often at the whim of the developer and content owner. It would also be helpful if once the website was in place, its design and structure remained static; that does not happen either. Compounding it, in our hack-worried world, some restrict or limit access to prevent malicious intruders, at the expense of legitimate users. Finally, software bugs introduced inadvertently by developers add to the challenge.

It is Not “One Size Fits All”

“We want the content from www.very_important_content.com.” These marching orders launch our focused functional and technical analysis of each new website.

Understanding the design of each individual website is a prerequisite for successful crawler automation and content harvesting. Our methodology guides both the website analysts, and later, the developers, through a series of questions designed to derive the best approach for each unique website and content set.

Some critical questions to ask are the following:

One Best Approach? Wishful Thinking….

One learns quickly that one size does not fit all. It is not feasible to design one approach to intelligently crawl even a small subset of these websites. Even within the same department, the web page layout and backend technology often varies, requiring frequent customizations.

Modern websites have progressed far beyond simple html pages to interactive, database driven applications using logic residing both on the client page and server based code. This forced a transition from NCSA Mosaic (last released OSX version size 1.7 mb) to the current version of Google Chrome, weighing in at a hefty 554 mb at the time of this writing. This growth in application size represents the ever-expanding feature set supported in modern browsers.

Our Methodology - How to Focus on What Matters

In order to attack these problems, we’ve identified a focused series of questions that guide the developer through the decision process and determine an optimized approach to extract content and metadata from each specific website. These include:

Crawler Magic - From Their Website to Ours

Rarely are two websites alike. A viable crawl solution must accommodate the unique aspects of a website without starting from scratch each time we face a new nuance. Our toolchain approach, in which a set of components are assembled into a crawler, is our preferred method for crawling large numbers of diverse websites in an efficient, timely manner and has proven very effective.

Some of the components we configure in our toolchain approach include:

No plan survives contact with a webmaster

Sites change, pages are updated with new character sets, update notification pages are frequently wrong, links die or are changed. We couple our automated crawling with automated validation to ensure that we have all the required files and metadata. When we find discrepancies, alerts are issued and our website analysts often reach out to the webmasters. From then on, it is uncertain as to whether we will get a response or resolution, and we often have to implement workarounds.

What is Next? Beyond 2018….

Over the past two and a half years, we have developed a series of best practices for web crawling and harvesting technologies, achieving fully automated processing against a wide range of diverse, complex and often poorly structured websites. Our methodology has been iteratively refined to accommodate the ever-changing landscape of internet content and facilitate a model of continuous improvement.

We are far from done. While not there yet, we are well on our way to eliminate manual intervention and further automate website analysis, reducing greatly the manual effort to research and resolve problems. We are starting to leverage our volumes of training data to create training sets for machine learning based troubleshooting and information extraction, and it is already demonstrating significant potential.

Our current road map includes utilizing TensorFlow, NLP and supervised machine learning to classify sections of text, extract references and metadata and to supplement our quality control, all targeted to improve the consistency and reliability of our results - and to do it faster and better.