Foundations of Python Network Programming 978-1-4302-3004-5

CHAPTER 10 ■ SCREEN SCRAPING
Condition: Fair
Temperature: 54 F
Humidity: 28 %
$ python weather.py Grand Canyon, AZ
Condition: Fair
Temperature: 67°F
Humidity: 28 %
Condition: Fair
Temperature: 67 F
Humidity: 28 %
You will note that some cities have spaces between the temperature and the F, and others do not.
No, I have no idea why. But if you were to parse these values to compare them, you would have to learn
every possible variant and your parser would have to take them into account.
I leave it as an exercise to the reader to determine why the web page currently displays the word
“NULL”—you can even see it in the browser—for the temperature in Elk City, Oklahoma. Maybe that
location is too forlorn to even deserve a reading? In any case, it is yet another special case that you would
have to treat sanely if you were actually trying to repackage this HTML page for access from an API:
$ python weather.py Elk City, OK
Condition: Fair and Breezy
Temperature: NULL
Humidity: NA
Condition: Fair and Breezy
Temperature: NULL
Humidity: NA
I also leave as an exercise to the reader the task of parsing the error page that comes up if a city
cannot be found, or if the Weather Service finds it ambiguous and prints a list of more specific choices!
Summary
Although the Python Standard Library has several modules related to SGML and, more specifically, to
HTML parsing, there are two premier screen-scraping technologies in use today: the fast and powerful
lxml library that supports the standard Python “ElementTree” API for accessing trees of elements, and
the quirky BeautifulSoup library that has powerful API conventions all its own for querying and
traversing a document.
If you use BeautifulSoup before 3.2 comes out, be sure to download the most recent 3.0 version; the
3.1 series, which unfortunately will install by default, is broken and chokes easily on HTML glitches.
Screen scraping is, at bottom, a complete mess. Web pages vary in unpredictable ways even if you
are browsing just one kind of object on the site—like cities at the National Weather Service, for example.
To prepare to screen scrape, download a copy of the page, and use HTML tidy, or else your screenscraping
library of choice, to create a copy of the file that your eyes can more easily read. Always run
your program against the ugly original copy, however, lest HTML tidy fixes something in the markup
that your program will need to repair!
Once you find the data you want in the web page, look around at the nearby elements for tags,
classes, and text that are unique to that spot on the screen. Then, construct a Python command using
your scraping library that looks for the pattern you have discovered and retrieves the element in
question. By looking at its children, parents, or enclosed text, you should be able to pull out the data that
you need from the web page intact.
177