Foundations of Python Network Programming 978-1-4302-3004-5

CHAPTER 10 ■ SCREEN SCRAPING
BeautifulSoup lets you choose the first child element with a given tag by simply selecting the
attribute .tagname, and lets you receive a list of child elements with a given tag name by calling an
element like a function—you can also explicitly call the method findAll()—with the tag name and a
recursive option telling it to pay attention just to the children of an element; by default, this option is set
to True, and BeautifulSoup will run off and find all elements with that tag in the entire sub-tree beneath
an element!
Anyway, two lessons should be evident from the foregoing exploration.
First, both lxml and BeautifulSoup provide attractive ways to quickly grab a child element based on
its tag name and position in the document.
Second, we clearly should not be using such primitive navigation to try descending into a real-world
web page! I have no idea how code like the expressions just shown can easily be debugged or
maintained; they would probably have to be re-built from the ground up if anything went wrong with
them—they are a painful example of write-once code.
And that is why selectors that each screen-scraping library supports are so critically important: they
are how you can ignore the many layers of elements that might surround a particular target, and dive
right in to the piece of information you need.
Figuring out how HTML elements are grouped, by the way, is much easier if you either view HTML
with an editor that prints it as a tree, or if you run it through a tool like HTML tidy from W3C that can
indent each tag to show you which ones are inside which other ones:
$ tidy phoenix.html > phoenix-tidied.html
You can also use either of these libraries to try tidying the code, with a call like one of these:
lxml.html.tostring(html)
soup.prettify()
See each library’s documentation for more details on using these calls.
Selectors
A selector is a pattern that is crafted to match document elements on which your program wants to
operate. There are several popular flavors of selector, and we will look at each of them as possible
techniques for finding the current-conditions tag in the National Weather Service page for
Phoenix. We will look at three:
• People who are deeply XML-centric prefer XPath expressions, which are a
companion technology to XML itself and let you match elements based on their
ancestors, their own identity, and textual matches against their attributes and text
content. They are very powerful as well as quite general.
• If you are a web developer, then you probably link to CSS selectors as the most
natural choice for examining HTML. These are the same patterns used in
Cascading Style Sheets documents to describe the set of elements to which each
set of styles should be applied.
• Both lxml and BeautifulSoup, as we have seen, provide a smattering of their own
methods for finding document elements.
Here are standards and descriptions for each of the selector styles just described— first, XPath:
http://www.w3.org/TR/xpath/
http://codespeak.net/lxml/tutorial.html#using-xpath-to-find-text
http://codespeak.net/lxml/xpathxslt.html
173