Foundations of Python Network Programming 978-1-4302-3004-5

CHAPTER 10 ■ SCREEN SCRAPING
content = response.read()
open('phoenix.html', 'w').write(content)
Many mechanize users instead choose to select forms by the order in which they appear in the
page—in which case we could have called select_form(nr=1). But I prefer not to rely on the order, since
the real identity of a form is inherent in the action that it performs, not its location on a page.
You will see immediately the problem with using mechanize for this kind of simple task: whereas
Listing 10–1 was able to fetch the page we wanted with a single HTTP request, Listing 10–2 requires two
round-trips to the web site to do the same task. For this reason, I avoid using mechanize for simple form
submission. Instead, I keep it in reserve for the task at which it really shines: logging on to web sites like
banks, which set cookies when you first arrive at their front page and require those cookies to be present
as you log in and browse your accounts. Since these web sessions require a visit to the front page
anyway, no extra round-trips are incurred by using mechanize.
The Structure of Web Pages
There is a veritable glut of online guides and published books on the subject of HTML, but a few notes
about the format would seem to be appropriate here for users who might be encountering the format for
the first time.
The Hypertext Markup Language (HTML) is one of many markup dialects built atop the Standard
Generalized Markup Language (SGML), which bequeathed to the world the idea of using thousands of
angle brackets to mark up plain text. Inserting bold and italics into a format like HTML is as simple as
typing eight angle brackets:
The very strange book Tristram Shandy.
In the terminology of SGML, the strings and are each tags—they are, in fact, an opening
and a closing tag—and together they create an element that contains the text very inside it. Elements
can contain text as well as other elements, and can define a series of key/value attribute pairs that give
more information about the element:
I am reading Hamlet.
There is a whole subfamily of markup languages based on the simpler Extensible Markup Language
(XML), which takes SGML and removes most of its special cases and features to produce documents that
can be generated and parsed without knowing their structure ahead of time. The problem with SGML
languages in this regard—and HTML is one particular example—is that they expect parsers to know the
rules about which elements can be nested inside which other elements, and this leads to constructions
like this unordered list , inside which are several list items :
FirstSecondThirdFourth
At first this might look like a series of elements that are more and more deeply nested, so that
the final word here is four list elements deep. But since HTML in fact says that elements cannot
nest, an HTML parser will understand the foregoing snippet to be equivalent to this more explicit XML
string:
FirstSecondThirdFourth
And beyond this implicit understanding of HTML that a parser must possess are the twin problems
that, first, various browsers over the years have varied wildly in how well they can reconstruct the
document structure when given very concise or even deeply broken HTML; and, second, most web page
authors judge the quality of their HTML by whether their browser of choice renders it correctly. This has
resulted not only in a World Wide Web that is full of sites with invalid and broken HTML markup, but
167