2 Debian Code Search: An Overview

3.8.2 Looking up trigrams in the index
3.8 The trigram index
In order to modify the index data structure in such a way that it is more suited for our purposes,
we first need to understand how it works. Figure 3.4 is an illustration of the different parts
of the index. On the left, you can see each section within the index file. On the right, an
example entry for each of the different sections is provided (except for list of paths, which
is unused in our code). Each section of the index is sorted, and the number of entries of the
name index and posting list index are stored in the trailer.
codesearch index
Header
List of Paths
List of Names g t k
List of Posting Lists s n a 2 5 1 1
Name Index
trigram
0 0 0 8
file IDs
0 0 0 8
Posting List Index
file ID
s n a
name offset
0 0 0 8 0 4 3 0
Trailer
trigram file count post. list offset
- 3 / c o n s t . c \0
filename
Figure 3.4: The Codesearch index format. Trigram lookups are performed as described below.
Assuming the list of all files which contain the trigram “sna” needs to be obtained, the
following steps have to be performed:
1. Seek to the posting list index and perform a binary search to find the entry for trigram
“sna”. The entry reveals the file count and a byte offset (relative to the first byte of the
index) pointing at the entry for “sna” in the list of posting lists.
2. Seek to the entry for “sna” in the list of posting lists and decode the varint [7] -encoded
list of file IDs.
3. For each file ID, seek to the appropriate position in the name index. The byte offset
pointing to the filename in the list of names is now known.
4. For each name offset, seek to the appropriate position in the list of names and read
the filename (NUL-terminated).
17