The Annoyance Filter.pdf - Fourmilab

§256 ANNOYANCE-FILTER DEVELOPMENT LOG 205
Added the −−clearjunk and −−clearmail options to clear counts of junk and mail. This can be used,
in conjunction with the binwrite option, to prepare databases for use by folks who do not wish to
prepare their own.
2002 September 11
Added the ability to enforce minimum and maximum length constraints on tokens returned by tokenParser.
The limits are set to accept tokens from 1 to 255 characters in the tokenParser constructor, and may
be changed at any time with the setTokenLengthLimits method. Note that the length limits are not
reset by a call to setSource .
Set the default token parser length limits to accept tokens between 1 and 64 characters.
doubtless be the subject of yet more command line options before long.
This will
Modified the code which decides whether a mail folder is compressed to check for the argument being a
symbolic link. If so, the link target is tested for the extension indicating a compressed file. I only follow
links one level—if this poses a problem, your life is probably too complicated.
Fixed computation of probability to avoid crashes if no words are present in a category. Probabilities
don’t make any sense in such circumstances, but you may wish to create such a database for use with
−−binread.
Added logic to dictionary ::exportToBinaryFile and dictionary ::exportToBinaryFile to save and
restore the count of messages contributing to the dictionary in the messageCount array in a pseudo-word
called “␣COUNTS␣” (obligatorily) at the start of the dictionary. These counts are required should we
need to recompute the probability subsequent to loading the dictionary.
Added the −−newword and.–sigwords options to specify the probability given to words in a message
which don’t appear in the dictionary and the number of “most significant” words whose probabilities
are used to determine the aggregate probability a given message is junk.
2002 September 12
Added logic to cope with the body of a message being encoded in a Content−TransferEncoding.
While processing the header, this and the Content−Type are parsed as in MIME headers, with their
arguments saved in bodyContentType , bodyContentTypeCharset , and bodyContentTransferEncoding . At
the end of the header, if a bodyContentTransferEncoding has been specified, the values are transferred
to the corresponding mime . . . variables and multiPart is set with an end terminator of the null string.
The latter disables the decoder’s test for a part end sentinel and the warning for an unterminated part.
Messages with Subject lines which contain ISO 8859 encoded characters employ a form of Quoted−Printable
encoding to permit these characters to appear in a mail header where only 7 bit ASCII is permitted.
I added code to mailFolder to detect these lines and call a new decodeEscapedText method of
quotedPrintableMIMEdecoder to decode them if properly formed. This will permit parsing of ISO
subject lines, which may prove critical in discriminating among messages with very short body copy.
Yikes! As far as I can determine from the RFCs, what we’re supposed to do with continued header
lines is just concatenate them, discarding all white space on the continuation even if this runs together
tokens on adjacent lines. At least, if you don’t do this, encoded words split across continued Subject
lines end up with nugatory white space in the middle. So, I fixed 〈 Check for continuation of mail header
lines 143 〉 to “work this way”. Given our definition of tokens, it’s likely to fix more things than it breaks
anyway.
Added documentation to the CWEB file for yesterday’s new options.
2002 September 13