Core Python Programming (2nd Edition)

20.3. Advanced Web Clients
Web browsers are basic Web clients. They are used primarily for searching and downloading documents
from the Web. Advanced clients of the Web are those applications that do more than download single
documents from the Internet.
One example of an advanced Web client is a crawler (aka spider, robot). These are programs that
explore and download pages from the Internet for different reasons, some of which include:
● Indexing into a large search engine such as Google or Yahoo!
● Offline browsingdownloading documents onto a local hard disk and rearranging hyperlinks to
create almost a mirror image for local browsing
● Downloading and storing for historical or archival purposes, or
● Web page caching to save superfluous downloading time on Web site revisits.
The crawler we present below, crawl.py, takes a starting Web address (URL), downloads that page and
all other pages whose links appear in succeeding pages, but only those that are in the same domain as
the starting page. Without such limitations, you will run out of disk space! The source for crawl.py
appears in Example 20.2.
Line-by-Line (Class-by-Class) Explanation
Lines 111
The top part of the script consists of the standard Python Unix start-up line and the importation of
various module attributes that are employed in this application.
Lines 1349
The Retriever class has the responsibility of downloading pages from the Web and parsing the links
located within each document, adding them to the "to-do" queue if necessary. A Retriever instance
object is created for each page that is downloaded from the net. Retriever consists of several methods
to aid in its functionality: a constructor (__init__()), filename(), download(), and parseAndGetLinks().
The filename() method takes the given URL and comes up with a safe and sane corresponding filename
to store locally. Basically, it removes the "http://" prefix from the URL and uses the remaining part as
the filename, creating any directory paths necessary. URLs without trailing file-names will be given a
default filename of "index.htm". (This name can be overridden in the call to filename()).
The constructor instantiates a Retriever object and stores both the URL string and the corresponding file
name returned by filename() as local attributes.
The download() method, as you may imagine, actually goes out to the net to download the page with the
given link. It calls urllib.urlretrieve() with the URL and saves it to the filename (the one returned by
filename()). If the download was successful, the parse() method is called to parse the page just copied
from the network; otherwise an error string is returned.
If the Crawler determines that no error has occurred, it will invoke the parseAndGetLinks() method to