2 Debian Code Search: An Overview

3 Architecture
3.8.1 Trigram index limitations
This work is based on Russ Cox’ Codesearch tools [2] (see also 3.2, page 7), which define an
index data structure, index source code and support searching the previously indexed source
code using regular expressions.
Google Code Search was launched in 2006, but Russ Cox’ Codesearch tools are implemented
in Go, a language which was created in late 2007 [6] . Therefore, they cannot be the original
Google Code Search implementation, and have not been used for a large-scale public search
engine.
On the contrary, Russ recommends them for local repositories, presumably not larger than
a few hundred megabytes:
“If you miss Google Code Search and want to run fast indexed regular expression
searches over your local code, give the standalone programs a try.” [2]
One obvious limitation of the index implementation is its size limitation of 4 GiB, simply
because the offsets it uses are 32-bit unsigned integers which cannot address more than 4 GiB.
Another not so obvious limitation is that even though an index with a size between 2 GiB
and 4 GiB can be created, it cannot be used later on, since Go’s mmap implementation only
supports 2 GiB 9 .
To circumvent this limitation while keeping modifications to Russ Cox’ Codesearch tools to
a minimum 10 , a custom indexing tool to replace Codesearch’s cindex has been implemented.
This replacement, called dcsindex, partitions the index into multiple files. Additionally, it
tries to reduce the index size in the first place by ignoring some files entirely.
Partitioning does not only work around the index size limitation, it also comes in handy
when distributing the index across multiple machines. Instead of running six index shards
on one machine, one could run three index shards on each of two machines.
9 This is because len() always returns an int to avoid endless loops in a for i:=0; i