The Annoyance Filter.pdf - Fourmilab

§189 ANNOYANCE-FILTER CLASSIFY MESSAGE 159
189. Given the list of most signficant tokens, we now use Bayes’ theorem to compute the aggregate
probability the message is junk. If p i is the probability word i of the most significant n (nExtremal)
words in a message appears in junk mail, the probability the message as a whole is junk is:
n∏
p i
i=1
n∏ ∏
p i + n (1 − p i )
i=1
〈 Compute probability message is junk from most significant tokens 189 〉 ≡
unsigned int n = min (static cast〈multimap〈double, string〉::size type 〉(nExtremal ),
rtokens .size ( ));
multimap〈double, string〉::const reverse iterator rp = rtokens .rbegin ( );
double probP = 1, probQ = 1;
i=1
if (verbose ) {
cerr ≪ "Rank␣␣␣Probability␣␣␣Token" ≪ endl ;
}
for (unsigned int i = 0; i < n; i++) {
double p;
if (fd ⃗ isDictionaryLoaded ( )) {
p = fd ⃗ find (rp ⃗ second );
if (p < 0) {
p = unknownWordProbability ;
}
}
else {
dictionary ::iterator dp = d ⃗ find (rp ⃗ second );
p = ((dp ≡ d ⃗ end ( )) ∨ (dp ⃗ second .getJunkProbability ( ) < 0)) ? unknownWordProbability :
dp ⃗ second .getJunkProbability ( );
}
if (verbose ) {
cerr ≪ setw (3) ≪ setiosflags (ios ::right ) ≪ (i + 1) ≪ "␣␣␣␣␣␣" ≪ setw (9) ≪ setprecision (5) ≪
setiosflags (ios ::left ) ≪ p ≪ "␣␣" ≪ rp ⃗ second ≪ endl ;
}
probP ∗= p;
probQ ∗= (1 − p);
rp ++;
}
junkProb = probP /(probP + probQ );
if (verbose ) {
cerr ≪ "ProbP␣=␣" ≪ probP ≪ ",␣ProbQ␣=␣" ≪ probQ ≪ endl ;
}
This code is used in section 185.