The Annoyance Filter.pdf - Fourmilab

§256 ANNOYANCE-FILTER DEVELOPMENT LOG 207
fixed a bug in updating the message counts which applied to both −−csvread and the existing −−read
code, but which only manifested itself when loading multiple dictionaries.
Wheels within wheels. . .MIME multipart messages can, of course, be nested. You can be blithely parsing
your way through a message when you trip over a part with a Content−type of “multipart/alternative”,
which pushes a new part boundary onto the stack, to be popped when the end sentinel of that nested
section is encountered. What fun. We consequently introduce a new partBoundaryStack to keep track
of the nested part boundary sentinels, along with all of the defensive code needed to cope with the
realities of real world mail.
2002 September 15
Loosened up the test for multipart Content−type so that “multipart/related” types will be recognised.
Added the long-awaited −−transcript option. (Thanks, Kern, for suggesting it!) A transcript of the input
message for a −−test or −−classify operation is written to the argument file name (standard output
if the argument is “−”, with X−Annoyance−Filter−Junk−Probability and X−Annoyance−Filter−Classification
items appended to the header indicating the calculated junk probability and classification according to
the thresholds.
Finished the first cut of multiple byte character set decoders and interpreters. A decoder scans the mail
body (encoded or not), and parses the byte stream into logical characters up to 32 bits in width. An
interpreter expresses these characters in a form suitable for analysis. Ideographic languages are typically
interpreted as one word per character, other languages as one letter per character. These components
must, of course, be utterly bullet-proof as they will be subjected to every possibly kind of garbage in the
course of parsing real-world mail. At the moment, we have decoders for EUC and Big5, and interpreters
for GB2312 and Big5.
Added a decoder for EUC-encoded Korean (euc−kr) as an example of how to handle an alphabetic
language with a non-Western character set.
2002 September 16
Modified EUC MBCSdecoder to discard the balance of any encoded line in which an invalid EUC
second byte is encountered. After encountering such garbage, the rest of the line is usually junk and
there’s no profit in blithering through it.
Added logic to scan application binary byte streams for possible embedded tokens. The new −−binword
option sets the shortest sequence of contiguous ASCII alphanumeric characters or dollar signs (with possible
embedded hyphens and apostrophes, but not permitting these character at the start or end of a
token—the default is 5 characters, which is a tad more discriminating than the UNIX strings which
defaults to 4 printable characters. You can disable the scanning of binary streams entirely by setting
−−binword to zero. Scanning binary streams might seem to be a curious endeavour, but it’s highly
effective at percolating text embedded in viruses and worm attachments to junk mail to the top of the
junk probability hit parade, then screening them out when the arrive in incoming mail.
Although the Subject line is the most important, any line in a mail header may actually contain quoted
sequences specifying a character set and Quoted−Printable or Base64 encoded characters. I modified
〈 Check for encoded header line and decode 147 〉 to no longer restrict decoding to the subject line.
Once decoded, if the charset specification in a header line quoted sequence is a character set we
understand, it is not decoded and interpreted. ISO−8859 sets of all flavours are decoded but not
processed further.
Fixed a few gcc −Wall quibbles in tokenDefinition which popped up on Solaris compiler but didn’t
seem to perturb the almost identical version of gcc on Linux.