General Computer Science 320201 GenCS I & II Lecture ... - Kwarc

engine extracts salient features from the documents and stores them in a special data structure
(usually tree-like) that can be queried instead of the documents themselves. In the prevalent
information retrieval algorithms, the salient feature is a word frequency vector.
Searching for Documents Efficiently: Indexing
Problem: We cannot search the WWWeb linearly (even with 10 6 computers: ≥ 10 15 B)
Idea: Write an “index” and search that instead. (like the index in a book)
Definition 569 Search engine indexing analyzes data and stores key/data pairs in a special
data structure (the search index to facilitate efficient and accurate information retrieval.
Idea: Use the words of a document as index (multiword index) The key for a document is
the vector of word frequencies.
term 2
term 1
D1(t1,1, t1,2, t1,3)
D2(t2,1, t2,2, t2,3)
term 3
c○: Michael Kohlhase 375
Note: The word frequency vectors used in the “vector space model” for information retrieval are
very high-dimensional; the dimension is the number of words in the document corpus. Millions
of dimensions are usual. However, linguistic methods like “stemming” (reducing words to word
stems) are used to bring down the number of words in practice.
Once an answer set has been determined, the results have to be sorted, so that they can be
presented to the user. As the user has a limited attention span – users will look at most at three
to eight results before refining a query, it is important to rank the results, so that the hits that
contain information relevant to the user’s information need early. This is a very difficult problem,
as it involves guessing the intentions and information context of users, to which the search engine
has no access.
Ranking Search Hits: e.g. Google’s Pagerank
Problem: There are many hits, need to sort them by some criterion (e.g. importance)
Idea: A web site is important, . . . if many other hyperlink to it.
Refinement: . . . , if many important web pages hyperlink to it.
215