Information und Wissen: global, sozial und frei? - Fachhochschule ...

26 Pavel Sirotkin
Formula 1. MAP formula with queries Q, relevant documents R, documents D at
rank r and n returned results. rel is a relevance function assigning 1 to relevant results.
Another metric which has enjoyed wide popularity since its introduction
is Discounted Cumulated Gain or DCG for short (Järvelin and Kekäläinen
2002). The more basic measure upon which it is constructed is the Cumulated
Gain, which is a simple sum of the relevance judgements of all results
up to a certain rank. DCG enhances this rather simple method by introducing
“[a] discounting function [...] that progressively reduces the document score
as its rank increases but not too steeply (e.g., as division by rank) to allow for
user persistence in examining further documents” (Järvelin and Kekäläinen
2002, p. 425). In practice, the authors suggest a logarithmic function, which
can be adjusted (by selecting its base) to provide a more or less strong discount,
depending on the expectations of users’ persistence. DCG can be
modified to allow for better inter-query comparison; to this end, a perfect
ranking for known documents is constructed. The DCG of a result list is then
divided by the ideal DCG, producing normalized DCG (nDCG) in the 0...1
range.
Formula 2. DCG with logarithm base b (based on Järvelin and Kekäläinen 2002).
Metric Evaluations
When a new evaluation metric is introduced, it is usually explained what its
advantage over existing metrics is. Mostly, this happens in theoretical terms;
more often than not, an experimental metric evaluation is also given. There
are many studies comparing one metric to another; however, this has the
disadvantage of being a circular confirmation, indicating at best correlation
between metrics.
Another method was used for evaluating different CG metrics (Järvelin
and Kekäläinen 2000; Järvelin and Kekäläinen 2002). Those were used to
evaluate different IR systems, where one was hypothesized to outperform the