untangling_the_web

DID: 4046925
UNCLASSIFIEDNFOR OFFIGIAb l:IS~ O~lb¥
engine spiders. Indeed, without them, we would have little or no idea what is "out
there" and available to us. The problem for webmasters is that it is their
responsibility to keep the search engine spiders out of any parts of their websites
they do not want to be accessed and indexed by a search engine. The spider is not
smart; it simply knows that if a "door" is open, it can-and will-go in and crawl
around. Webmasters must tell spiders "do not enter" (primarily) by the use of the
Robots Exclusion Protocol.
Robots Exclusion 62 comes in two basic flavors: either a metatag that can be inserted
into the HTML of a web page (usually used by an individual) or a Robots Exclusion
Protocol (robots.txt) file, a specially formatted file inserted by the website
administrator to tell the spider which parts of the website may and may not be
indexed by the spider. If a robots exclusion is missing or improperly configured, the
spider will index pages that the website owner may not have wished to have been
accessed.
The whole problem of keeping information on the Internet private dramatically
worsened almost overnight a couple of years ago when Google quietly started
indexing whole new types of data. Originally, most of what got spidered and indexed
was HTML webpages and documents, with some plain text thrown in for good
measure. However, the ever-innovative Google decided this wasn't good enough
and started to index PDF, PostScript, and-most importantly-a whole range of
Microsoft file types: Word, Excel, PowerPoint, and Access. Problem was, lots of
folks had assumed these file types were "immune" to spidering not because it
couldn't be done but because no one had yet done it. As a result, many companies,
organizations, and even governments had quite a lot of egg on their faces when
sensitive documents began turning up in the Google database.
That was then, this is now. You might think people would have learned, but judging
by the amount of "sensitive" information still available, many have not. Even though
search engines now routinely index many non-HTML file types, many individuals and
organizations still do not protect these files from the long reach of search engine
spiders. Furthermore, there are many ways for sensitive information to end up in
search engine databases. An improperly configured server, security holes, and
unpatched software can give search engine spiders unintended access. Quite
frankly, most of the problems boil down to one thing: human error, either through
ignorance or neglect.
What kinds of sensitive information can routinely be found using search engines?
. The types of data most commonly discovered by Google hackers usually falls into
one of these categories:
62 For additional information, see: (14 November 2006).
176 UNCLASSIFIEDft'JiOK OJiJilehltL USE Ot4LY