2 Debian Code Search: An Overview

3.4 Architecture overview
HTTP frontend dcs-web
(nginx)
delivers static assets
load-balances requests
3.4 Architecture overview
queries backends
generates response
index backends (sharded) PostgreSQL source-backend
Figure 3.2: Architecture overview, showing which different processes are involved in handling
requests to Debian Code Search. 1
Debian Code Search consists of three different types of processes (dcs-web, index-backend,
source-backend) running “behind” an nginx webserver and accessing a PostgreSQL database
when starting up.
When a new request comes in to http://codesearch.debian.net/, nginx will deliver
the static index page. However, when the request is not for a static page but an actual search
query, say http://codesearch.debian.net/search?q=XCreateWindow, the request will
be forwarded by nginx to the dcs-web process.
dcs-web first parses the search query, meaning it handles special keywords contained in
the query term, e.g. “filetype:perl”, and stores parameters for pagination. Afterwards,
dcs-web sends requests to every index-backend process and gets back a list of filenames
which possibly contain the search query from the index-backends. See section 3.5 on why
there are multiple index-backend instances. See section 3.8, “The trigram index”, on page 15
for details of the index-backend lookup process.
These filenames are then ranked with ranking data loaded from PostgreSQL in such a way
that the filename which is most likely to contain a good result comes first. Afterwards, the
list of ranked filenames is sent to the source-backend process, which performs the actual
searching using a regular expression matcher, just like the UNIX tool grep(1).
As soon as the source-backend has returned enough results, dcs-web ranks them again with
the new information that was obtained by actually looking into the files and then presents
the results to the user.
1 This figure has been created with dia. The icons are gnomeDIAicons, licensed under the GPL.
9