The Annoyance Filter.pdf - Fourmilab

158 CLASSIFY MESSAGE ANNOYANCE-FILTER §188
188. Once we’ve obtained a list of tokens in the message, we now wish to filter it by the significance of
the probability that a token appears in junk or legitimate mail. This is simply the absolute value of the
difference of the token’s junkProbability from 0.5—the probability for a token equally likely to appear
in junk and legitimate mail. We construct a multimap called rtokens which maps this significance
value to the token string; since any number of tokens may have the same significance, we must use a
multimap as opposed to a map.
We count on multimap being an ordered collection class which, when traversed by its reverse iterator,
will return tokens in order of significance. This assumption may be unwarranted, but it’s valid for all
the STL implementations I’m aware of (and is essentially guaranteed since the fact that multimap
requires only the < operator for ordering effectively mandates a binary tree implementation).
〈 Classify message tokens by probability of significance 188 〉 ≡
multimap〈double, string〉 rtokens ;
for (set〈string〉::iterator t = utokens .begin ( ); t ≠ utokens .end ( ); t++) {
double pdiff ;
}
dictionary ::iterator dp;
if (fd ⃗ isDictionaryLoaded ( )) {
pdiff = fd ⃗ find (∗t);
if (pdiff < 0) {
pdiff = unknownWordProbability ;
}
pdiff = abs (pdiff − 0.5);
}
else {
if (((dp = d ⃗ find (∗t)) ≠ d ⃗ end ( )) ∧ (dp ⃗ second .getJunkProbability ( ) ≥ 0)) {
pdiff = abs (dp ⃗ second .getJunkProbability ( ) − 0.5);
}
else {
pdiff = abs (unknownWordProbability − 0.5);
}
}
rtokens .insert (make pair (pdiff , ∗t));
This code is cited in section 256.
This code is used in section 185.