php|architect's Guide to Web Scraping with PHP - Wind Business ...

HTTP ” 19
disabling a primary site feature, such as an e-commerce checkout page that uses
ActiveX (a technology specific to Windo ws and Internet Explorer).
One well-kno wn application of this technique is the robots exclusion standard,
which is used to explicitly instruct crawlers to avoid accessing individual
resources or the entire web site. M ore information about this is available at
http://www.robotstxt.org. The guidelines detailed there should definitely be accounted
for when developing a web scraping application so as to prevent it from
exhibiting behavior inconsistent with that of a normal user.
In some cases, a client practice called user agent spoofing involving the specification
of a false user agent string is enough to circumvent user agent sniffing, but not
always. An application may have platform-specific requirements that legitimately
warrant it denying access to certain user agents. In any case, spoofing the user agent
is a practice that should be avoided to the fullest extent possible.
Ranges
The Range request header allo ws the client to specify that the body of the server’s
response should be limited to one or more specific byte ranges of what it would normally
be. Originally intended to allo w failed retrieval attempts to resume from their
stopping points, this feature can allo w you to minimize data transfer between your
application and the server to reduce bandwidth consumption and runtime of your
web scraping application.
This is applicable in cases where you have a good rough idea of where your target
data is located within the document, especially if the document is fairly large and
you only need a small subset of the data it contains. H o wever, using it does add one
more variable to the possibility of your application breaking if the target site changes
and you should bear that in mind when electing to do so .
While the format of the header value is being left open to allo w for other range
units, the only unit supported by HTTP/1.1 is bytes. The client and server may both
use the Accept-Ranges header to indicate what units they support. The server will
include the range (in a slightly different format) of the full response body in which
the partial response body is located using the Content-Range header.
In the case of bytes, the beginning of the document is represented by 0. Ranges use
inclusive bounds. F or example, the first 500 bytes of a document would be specified