Berry

Memory-Based Reasoning and Collaborative Filtering 273
MEASURING THE EFFECTIVENESS OF ASSIGNING CODES:
RECALL AND PRECISION
Recall and precision are two measurements that are useful for determining the
appropriateness of a set of assigned codes or keywords. The case study on
coding news stories, for instance, assigns many codes to news stories. Recall
and precision can be used to evaluate these assignments.
Recall answers the question: “How many of the correct codes did MBR
assign to the story?” It is the ratio of codes assigned by MBR that are correct
(as verified by editors) to the total number of correct codes on the story. If MBR
assigns all available codes to every story, then recall is 100 percent because the
correct codes all get assigned, along with many other irrelevant codes. If MBR
assigns no codes to any story, then recall is 0 percent.
Precision answers the question: “How many of the codes assigned by MBR
were correct?” It is the percentage of correct codes assigned by MBR to the
total number of codes assigned by MBR. Precision is 100 percent when MBR
assigns only correct codes to a story. It is close to 0 percent when MBR assigns
all codes to every story.
Neither recall nor precision individually give the full story of how good the
classification is. Ideally, we want 100 percent recall and 100 percent precision.
Often, it is possible to trade off one against the other. For instance, using more
neighbors increases recall, but decreases precision. Or, raising the threshold
increases precision but decreases recall. Table 8.5 gives some insight into these
measurements for a few specific cases.
Table 8.5 Examples of Recall and Precision
CODES BY MBR CORRECT CODES RECALL PRECISION
A,B,C,D
A,B
A,B,C,D
A,B,C,D
100% 100%
50% 100%
A,B,C,D,E,F,G,H A,B,C,D
100% 50%
E,F A,B,C,D 0% 0%
A,B,E,F
A,B,C,D
50% 50%
The original codes assigned to the stories by individual editors had a recall
of 83 percent and a precision of 88 percent with respect to the validated set of
correct codes. For MBR, the recall was 80 percent and the precision 72 percent.
However, Table 8.6 shows the average across all categories. MBR did
significantly better in some of the categories.
(continued)