untangling_the_web

DOClD: 4046925
UNCLASSIFIEDJJFOR OFFlCI,lcL \:ISE ot4L't
Beyond Search Engines-Specialized
Research Tools
Search engines are a good and natural starting place for performing research on the
Internet, but they represent only a small portion of the data available on the web and
only one way of tapping into that data. It is also important to understand that search
engine spiders do not access (and therefore search engines do not index) most
data contained in many databases or websites that require registration or
payment to enter. For example, search engines do not normally index the data in
PeopleData.com (a database) or any information beyond the first page or so of the
Chicago Tribune (which requires registration) . These types of sites require users to
access them directly. The information at these sites is part of the invisible, hidden, or
deep web.
The types of sites and information that are not generally accessible to search
engines include:
~ information in databases: phone and email directories, Whois registration &
DNS data, dictionaries , encyclopedia articles, statistics , legal and medical
data, financial information .
~ rapidly changing information: news, airline flight information , stock, bond,
currency market data, auctions.
~ for-fee and subscription services.
~ information behind a firewall (corporate , government, educational).
To give you a better idea just how vast the deep web is, consider these points from
"The Deep Web: Surfacing Hidden Value,,112 by Michael K. Bergman .
112 Michael K. Bergman , "The Deep Web: Surfacing Hidden Value," BrightPlanet, August 2001,
(14 November 2006).
306 UNCLASSIFIEDNFOR OFFlCI,lcL \:ISE ONLY