php|architect's Guide to Web Scraping with PHP - Wind Business ...

Chapter 2
HTTP
The first task that a web scraping application must be capable of performing is the
retrieval of documents containing the information to be extracted. If you have used
a web bro wser without becoming aware of all that it does “ under the hood” to render
a page for your viewing pleasure, this may sound trivial to you. H o wever, the complexity
of a web scraping application is generally proportional to the complexity of
the application it targets for retrieving and extracting data.
F or targets consisting of multiple pages or requiring retention of session or authentication
information, some level of reverse-engineering is often required to develop
a corresponding web scraping application. Like a complex mathematics problem
with a very simple answer, the development of web scraping applications can
sometimes involve more analysis of the target than work to implement a script capable
of retrieving and extracting data from it.
This sort of reconnaisance requires a decent working kno wledge of the HyperText
Transfer Protocol or HTTP, the protocol that po wers the internet. The majority of
this chapter will focus on familiarization with that protocol. The end goal is that you
become capable of performing the necessary research to learn ho w a target application
works such that you are capable of writing an application to extract the data you
want.