untangling_the_web

DOCID: 4046925
UNCLASSIFIEOHFOR OFFIGIAL I:JSE ONLY
respects these mechanisms. Password protection, firewalls, and other
measures will generally keep spiders from crawling a website and indexing it.
The Web Robots Pages ...
Robots Exclusion
Sometimes people find they have been indexed by an indexing robot, or that a resource discovery robot has
visited part of a site that for some reason shouldn't be visited by robots
In recognition of this problem, many Web Robots offer facilities for Web site administrators and content
providers to limit what the robot does. This is achieved through two mechanisms:
The Robots Exclusion Protocol
A Web site administrator can indicate which parts of the site should not
be vistsed by a robot, by providing a specially formatted file on their
site, inhttpll./robotstxt
TIle Robots META tag A Web author can indicate if a page mayor may not be indexed, or
analysed for links, through the use of a special HTIvfi.., META tag
The remainder ofthis pages provides full details on these facilities.
Note that these methods rely on cooperation from the Robot, and are by no means guaranteed to work for every
Robot. If you need stronger protection from robots and other agents, you should use alternative methods such as
password protection
Robots Exclusion Page
http://www.robotstxt.org/wc/exclusion.html
Not every search engine has its own proprietary search program but instead relies
upon another company's search service for its results. Most of these strategic
alliances now involve Yahoo, Google, and Windows Live Search. All these
partnerships are subject to change without notice; for more on these strategic
alliances, see:
Search Engine Alliances
http://searchenginewatch.com/reports/alliances.html
Knowing that Yahoo, for example, is the search tool behind a search engine can
save you time because you can be pretty sure that using AltaVista Will get you
similar (although not identical) results to the other search engines also powered by
Yahoo. It is critical to remember that each service powered by a particular search
engine produces different results even though they may all use the same core
database. Why is this? Because the search interfaces have their own algorithms that
decide how queries are run, how results are returned, or even if they query the entire
database (most do not). In short, go to the primary search engine-Google, Yahoo,
or Live Search for best results.
UNCLASSIFIEOHFOR OFFIOIAL I:JSE ONLY 21