php|architect's Guide to Web Scraping with PHP - Wind Business ...

HTTP ” 13
i
URL Encoding
One trait of query strings is that parameter values are encoded using percent-encoding
or, as it’s more commonly known, URL encoding. The PHP functions urlencode and
urldecode are a convenient way to handle string values encoded in this manner. Most
HTTP client libraries handle encoding request parameters for you. Though it’s called
URL encoding, the technical details for it are actually more closely associated with the
URI as shown in section 2.1 of RFC 3986.
HEAD Requests
Though not common when accessing target web applications, HEAD requests are
useful in web scraping applications in several ways. They function in the same way
as a GET request with one exception: when the server delivers its response, it will
not deliver the resource representation that normally comprises the response body.
The reason this is useful is that it allo ws a client to get at the data present in the
response headers without having to do wnload the entire response, which is liable
to be significantly larger. Such data can include whether or not the resource is still
available for access and, if it is, when it was last modified.
HEAD /wiki/Main_Page HTTP/1.1
Host: en.wikipedia.org
Speaking of responses, no w would be a good time to investigate those in more detail.
R esponses
Aside from the first response line, called the status line, responses are formatted very
similarly to requests. While different headers are used in requests and responses,
they are formatted the same way. A blank line separates the headers and the body in
both requests and responses. The body may be absent in either depending on what
the request operation is. Belo w is an example response.
HTTP/1.0 200 OK
Date: Mon, 21 Jul 2008 02:32:52 GMT