The Annoyance Filter.pdf - Fourmilab

204 DEVELOPMENT LOG ANNOYANCE-FILTER §256
256. Development log.
2002 August 28
Created development tree and commenced implementation.
2002 September 1
Release 0.1 circulated for review.
2002 September 6
Added the ability to compute descriptive statistics of the dictionary built by parsing the −−mail and
−−junk folders, using the facilities of the statlib.w program. Statistics are written to standard output.
Added a −−plot option to plot a histogram of words in a newly parsed dictionary (not a lookup
dictionary loaded with −−read). Creating the plot requires the GNUPLOT and PBMPlus utilities to be
installed.
2002 September 7
Well, after a huge amount of hunkering down and twiddling, parsing of MIME multi-part messages and
decoding of parts encoded in Base64 and Quoted−Printable encoding now seems to be working. This
drastically improves the quality of parsing, particularly for junk where these forms of encoding are used
as “stealth” to evade other content-based filters.
2002 September 8
Added the ability to read mail folders compressed with gzip or other compressors detected by the
Autoconf script. This saves a lot of space when you’re keeping large training archives around. This will
work only on systems with suitable decompressors and the popen facility.
2002 September 9
Added the −−pdiag option to write the parser diagnostics to a designated file.
controlled by a gnarly # define.
Previously this was
Added a “X−Annoyance−Filter−Decoder” line to the −−pdiag output to indicate the activation of
decoders (including the sink) for MIME parts in the message. These lines are not seen by the token
parser.
Fixed a bug in parsing of tokens including ISO accented characters. . .signed characters strike again.
2002 September 10
Added a −−ptrace option to include the actual tokens parsed as indented, quoted lines following each
line of parser input in the −−pdiag file.
Added code to classifyMessage which appends lines to the message header in the −−pdiag file giving
the aggregate junk probability and the most significant words and their individual probabilities.
Separated the mail and junk thresholds, which may now be set independently by the −−threshjunk
and −−threshmail options. The −−classify command now writes “INDT” (for “indeterminate”) if a
message falls between the two thresholds and exits with a return status of 4.
Added the −−binwrite and −−binread options to export and import a dictionary as a portable
(assuming IEEE floating point on all platforms) binary file. This will permit easier distribution of
dictionary databases and may be faster to load than the lookupDictionary .