Foundations of Python Network Programming 978-1-4302-3004-5

CHAPTER 9 ■ HTTP
HTTP Caching
Many elements of a typical web site design are repeated on every page you visit, and your browsing
would slow to a crawl if every image and decoration had to be downloaded separately for every page you
viewed. Well-configured web servers therefore add headers to every HTTP response that allow browsers,
as well as any proxy caches between the browser and the server, to continue using a copy of a
downloaded resource for some period of time until it expires.
You might think that adding a simple expiration date to each resource that could be cached and
redisplayed would have been a sufficient innovation. However, given the real-world behaviors of servers,
caches, and browsers, it was prudent for the HTTP specification to detail a much more complicated
scheme involving several interacting headers. Several pages are expended, for example, on the specific
question of how to determine how old a cached copy of a page is. I refer you to RFC 2616 for the real
details, but I will cover a few of the most common cases here.
There are two basic mechanisms by which servers can support client caching.
In the first approach, an HTTP response includes an Expires: header that formats a date and time
using the same format as the standard Date: header:
Expires: Sun, 21 Jan 2010 17:06:12 GMT
However, this requires the client to check its clock—and many computers run clocks that are far
ahead of or behind the real current date and time.
This brings us to a second, more modern alternative, the Cache-Control header, that depends only
on the client being able to correctly count seconds forward from the present. For example, to allow an
image or page to be cached for an hour but then insist that it be refetched once the hour is up, a cache
control header could be supplied like this:
Cache-Control: max-age=3600, must-revalidate
When the time comes to validate a cached resource, HTTP offers a very nice shortcut: the client can
ask the server to retransmit the resource only if a new version has indeed been released. There are two
fields that the client can supply. Either content type is sufficient to convince most servers to answer with
only an HTTP header, but no content type or body, if the cached resource is still current. One possibility
is to send back the value that the Last-modified: header had in the HTTP response that first requested
the item:
If-Modified-Since: Sun, 21 Jan 2010 14:06:12 GMT
Alternatively, if the server tagged the resource version with a hash or version identifier in an Etag:
header—either approach will work, so long as the value always changes between versions of the
resource—then the client can send that value back:
Etag: BFDS2Cpq/BM6w
Note that all of this depends on getting some level of cooperation from the server. If a web server
fails to provide any caching guidelines and also does not supply either a Last-modified: or Etag: header
for a particular resource, then clients have no choice but to fetch the resource every time it needs to be
displayed to a user.
Caching is such a powerful technology that many web sites go ahead and put HTTP caches like
Squid or Varnish in front of their server farms, so that frequent requests for the most popular parts of
their site can be answered without loading down the main servers. Deploying caches geographically can
also save bandwidth. In a celebrated question-and-answer session with the readers of Reddit about The
Onion’s then-recent migration to Django, the site maintainers—who use a content delivery network
(CDN) to transparently serve local caches of The Onion’s web site all over the world—indicated that they
were able to reduce their server load by two-thirds by asking the CDN to cache 404 errors! You can read
155