The Annoyance Filter.pdf - Fourmilab

§31 ANNOYANCE-FILTER DICTIONARY 33
31. Exporting or importing a dictionary to or from a binary file is more or less a matter of iterating
through the dictionary and delegating the matter to each individual word. One detail we must deal
with, however, is adding a pseudo-word at the head of the dictionary to record the number of mail and
junk messages which contributed the words to the dictionary. These counts are needed to subsequently
recompute the probability for each word.
When loading a dictionary with importFromBinaryFile this pseudo-word is recognised and the values
it contains are added to the messageCount for each category. Note that importing a file is logically an
addition to an existing dictionary—you may import any number of binary dictionary files, just as you
can add mail folders with the −−mail and −−junk options.
#define pseudoCountsWord "␣COUNTS␣"
〈 Class implementations 11 〉 +≡
void dictionary ::exportToBinaryFile (ostream &os )
{
if (verbose ) {
cerr ≪ "Exporting␣dictionary␣to␣binary␣file." ≪ endl ;
}
}
dictionaryWord pdw ;
pdw .set(pseudoCountsWord , messageCount [dictionaryWord ::Mail ],
messageCount [dictionaryWord ::Junk ], −1);
pdw .exportToBinaryFile (os );
for (dictionary ::iterator p = begin ( ); p ≠ end ( ); p++) {
p ⃗ second .exportToBinaryFile (os );
}
void dictionary ::importFromBinaryFile (istream &is )
{
if (verbose ) {
cerr ≪ "Importing␣dictionary␣from␣binary␣file." ≪ endl ;
}
}
dictionaryWord dw ;
if (dw .importFromBinaryFile (is )) {
assert(dw .get ( ) ≡ pseudoCountsWord );
messageCount [dictionaryWord ::Mail ] += dw .n mail ( );
messageCount [dictionaryWord ::Junk ] += dw .n junk ( );
while (dw .importFromBinaryFile (is )) {
include (dw ) ;
}
}