Foundations of Python Network Programming 978-1-4302-3004-5

CHAPTER 10 ■ SCREEN SCRAPING
And here are some CSS selector resources:
http://www.w3.org/TR/CSS2/selector.html
http://codespeak.net/lxml/cssselect.html
And, finally, here are links to documentation that looks at selector methods peculiar to lxml and
BeautifulSoup:
http://codespeak.net/lxml/tutorial.html#elementpath
http://www.crummy.com/software/BeautifulSoup/documentation.html#Searching the Parse Tree
The National Weather Service has not been kind to us in constructing this web page. The area that
contains the current conditions seems to be constructed entirely of generic untagged elements; none of
them have id or class values like currentConditions or temperature that might help guide us to them.
Well, what are the features of the elements that contain the current weather conditions in
Listing 10–3? The first thing I notice is that the enclosing element has the class "big". Looking at the
page visually, I see that nothing else seems to be of exactly that font size; could it be so simple as to
search the document for every with this CSS class? Let us try, using a CSS selector to begin with:
>>> from lxml.cssselect import CSSSelector
>>> sel = CSSSelector('td.big')
>>> sel(tree)
[]
Perfect! It is also easy to grab elements with a particular class attribute using the peculiar syntax of
BeautifulSoup:
>>> soup.find('td', 'big')
A Few Clouds
71°F(22°C)
Writing an XPath selector that can find CSS classes is a bit difficult since the class="" attribute
contains space-separated values and we do not know, in general, whether the class will be listed first,
last, or in the middle.
>>> tree.xpath(".//td[contains(concat(' ', normalize-space(@class), ' '), ' big ')]")
[]
This is a common trick when using XPath against HTML: by prepending and appending spaces to
the class attribute, the selector assures that it can look for the target class name with spaces around it
and find a match regardless of where in the list of classes the name falls.
Selectors, then, can make it simple, elegant, and also quite fast to find elements deep within a
document that interest us. And if they break because the document is redesigned or because of a corner
case we did not anticipate, they tend to break in obvious ways, unlike the tedious and deep procedure of
walking the document tree that we attempted first.
Once you have zeroed in on the part of the document that interests you, it is generally a very simple
matter to use the ElementTree or the old BeautifulSoup API to get the text or attribute values you need.
Compare the following code to the actual tree shown in Listing 10–3:
>>> td = sel(tree)[0]
>>> td.find('font').text
'\nA Few Clouds'
>>> td.find('font').findall('br')[1].tail
u'71°F'
174