untangling_the_web

OOClO: 4046925
UNCLASSIFIEONFOR OFFl61AL l:JSE m.LY
Uncovering the "Invisible" Internet
One of the most frustrating things about Internet search tools is the fact that even
the best index only a portion of the web, much less the entire Internet. The deep
(aka hidden or invisible) web continues to elude most search services and users
seeking to plumb its depths. We are still, for the most part, dependent upon specialty
tools and sites to help us find and exploit deep web resources. The challenge is how
to access that part of the web that remains invisible to search engines. It is important
to understand that search engines are generally designed to index a certain subset
of the Internet: web pages and, in some cases, certain types of files, e.g., video,
audio, PDF 91 . Furthermore, most search engines limit their web page and
document indexing. For example, Google used to index approximately the first
100KS of HTML, and reportedly the first megabyte of PDF documents, but in
October 2005, Google dramatically increased the size of its cache limit, although no
one knows for sure what that limit is. Yahoo indexes at least the first 500KS of HTML
and PDF documents. In any event, long documents usually are partially invisible to
these and other search engines. You cannot rely upon a search engine spider to
index long documents in their entirety.
A9 Search
At the end of September 2006, A9, the Amazon.com-owned search property, made
sweeping changes, some good and' some bad. Contrary to what some search
bloggers said, A9 is not "dead" (at least not yet). But some of A9's best features are
gone. As I feared, not enough people used the wonderful "street view" map resource
and now it is gone. As of September 29, 2006, A9 "discontinued A9 Maps and the
A9 Yellow Pages (including BlockView) ... [and] discontinued the A9 Instant
Reward program, and the A9 Toolbar and personalized services such as history,
bookmarks, and diary." Other changes include "a new continuous scrolling feature,
so you no longer have to bother with next and previous buttons to move from one
page of results to the next. You can now also drag-and-drop the columns to change
91 Google was the first major search engine to routinely index the contentof many file types, including
pdf, ps, xis, doc, ppt, and others. See "Google's Frequently Asked Questions - File Types ,"
(14 November 2006).
UNCLASSIFIEOMI='OR Qj;j;ISIAL l:JSE ONLY 239