2 Debian Code Search: An Overview

3.8 The trigram index
3.8 The trigram index
In order to perform a regular expression match, it is impractical to search through the whole
corpus for each query. Instead, an inverted n-gram index is required:
“[…] [an] n-gram is a contiguous sequence of n items from a given sequence of
text or speech.” [23]
In our case, the text is either the program source code when indexing, or the user’s search
terms when processing a query. An n-gram of size 3 is called a trigram, a good size for
practical use:
“This sounds more general than it is. In practice, there are too few distinct
2-grams and too many distinct 4-grams, so 3-grams (trigrams) it is.” [2]
As an example, consider the following simple example of a search query:
/Google.*Search/ (matching Google, then anything, then Search)
Obviously, only files which contain both the terms “Google” and “Search” should be considered
for the actual regular expression search. Therefore, these terms can be translated into
the following trigram index query:
Goo AND oog AND ogl AND gle AND Sea AND ear AND arc AND rch
This process can be performed with any query, but for some, the translation might lead to
the trigram query matching more documents than the actual regular expression. Since the
trigram index does not contain the position of each trigram, these kinds of false positives are
to be expected anyway.
The full transformation process is described in Russ Cox’ “Regular Expression Matching
with a Trigram Index” [2] .
15