The Annoyance Filter.pdf - Fourmilab

§228 ANNOYANCE-FILTER MAIN PROGRAM 183
228. If −−autoprune is specified, the memory consumed by the dictionary is estimated as tokens
are added and, if the threshold is exceeded, all unique words are pruned from the dictionary. If, after
the prune is complete, the dictionary still exceeds 90of beginning to thrash, pruning over and over to
no effect. If this is the case, we automatically increase the −−autoprune setting by 25% to stave off
thrashing (while, of course, running the risk of paging thrashing if physical memory is exceeded.
〈 Prune unique words from dictionary if autoPrune threshold is exceeded 228 〉 ≡
if ((autoPrune ≠ 0) ∧ (dict .estimateMemoryRequirement ( ) > autoPrune )) {
if (verbose ) {
cerr ≪ "Dictionary␣size␣" ≪ dict .estimateMemoryRequirement ( ) ≪
";␣starting␣automatic␣prune." ≪ endl ;
}
dict .purge (1);
if (dict .estimateMemoryRequirement ( ) > ((autoPrune ∗ 9)/10)) {
cerr ≪ "Dictionary␣size␣after␣−−autoprune␣is␣larger␣than␣90%" ≪ endl ;
cerr ≪ "of␣−−autoprune␣setting␣of␣" ≪ autoPrune ≪ "␣bytes." ≪ endl ;
autoPrune = static cast〈unsigned int〉(autoPrune ∗ 1.25);
cerr ≪ "Increasing␣−−autoprune␣threshold␣25%␣to␣" ≪ autoPrune ≪
"␣to␣avoid␣thrashing." ≪ endl ;
}
}
This code is used in section 227.
229. The updateProbability function recomputes word probabilities in the dictionary. It should be
called after any changes are made to the contents of the dictionary. Any operation which recomputes the
probabilities makes us ineligible for optimising out probability computation loading the first dictionary,
so we clear the singleDictionaryRead flag.
〈 Global functions 184 〉 +≡
static void updateProbability (void)
{
dict .computeJunkProbability (messageCount [dictionaryWord ::Mail ],
messageCount [dictionaryWord ::Junk ], mailBias , minOccurrences );
singleDictionaryRead = false ;
}
230. The printDictionary function dumps the dictionary in human-readable form to a specified output
stream,
〈 Global functions 184 〉 +≡
static void printDictionary (ostream &os = cout )
{
updateProbability ( );
os ≪ "Dictionary␣contains␣" ≪ dict .size ( ) ≪ "␣unique␣tokens." ≪ endl ;
for (dictionary ::iterator dp = dict .begin ( ); dp ≠ dict .end ( ); dp ++) {
dp ⃗ second .describe (os );
}
}