The Annoyance Filter.pdf - Fourmilab

210 DEVELOPMENT LOG ANNOYANCE-FILTER §256
Added a name method to MIMEdecoder and all its children, then took advantage of that to dispense
with the horrific duplication of decoder diagnostic code in 〈 Verify Content-Transfer-Encoding and
activate decoder if necessary 160 〉. What was previously dispersed among the several branches of the
decoder activation is now collected together in a single case after the decoder has been chosen.
Modified Makefile.in to delete the fussy core.process files Linux has taken to produce.
Fixed configure.in to specify −Wall if we’re building with GCC.
2002 September 20
On Solaris, GCC is prone to hang if invoked with −O2 (at least as of version 2.95.3). I twiddled the
configure.in to change the compile option to −O for Solaris builds.
ctangle and cweave spewed copious warnings on a GCC −Wall build. To avoid modifying these
programs, which are prefectly compliant ANSI C, I changed Makefile.in to suppress the −Wall option
for them when the compiler is detected as GCC.
make dist didn’t do a make distclean before generating the distribution archive, which could result
in build-specific files being included in the archive. Fixed.
2002 September 21
Added documentation on how to integrate annoyance−filter into a .forward pipeline to Procmail,
and build a .procmailrc rule set for a typical user-level filtering. It’s 03:40 and I’m going to get some
sleep before proofing this text—at the moment it’s something between a random scribble and a first
draft.
Okay, I just couldn’t stand it. . .I just had to take another crack at the infernal dictionary ::purge
method. One of the many bees in my bonnet buzzed the idea into my ear that I could avoid both the
extra memory consumption of yesterday’s scheme and the risk of instability in the container by testing
the probability of the first item in the map, adding it to the queue of survivors if its probability is
significant, then performing an erase (begin ( )). Cool, huh? No iterators, no mess, no two copies of any
word in memory.
The hits just keep on coming. . .the stupid built-in purge in dictionary ::resetCat also ran afoul of the
“stale iterator” problem. I blew it away—henceforth, it’s up to you to do a −−purge after a −−clearmail
or −−clearjunk. With the new tolerance for un-purged dictionaries, no great harm will be done if you
forget.
Added a \subsection macro to create subheads within documentation sections. The section number
is automatically grabbed from the cwebmac.tex definition, but lower level numbering is manual, permitting
you to add additional levels of hierarchy with a specification like:
\subsection{4.2.1}{Twiddling little details}.
It turns out that all the cheesy mess I put in to patch the user’s home directory into the annoyance−filter−run
script wasn’t necessary after all since sendmail is kind enough to change to the user’s home directory
before piping a message to a program. This means we can just cd to .annoyance−filter relative to
the home directory. This also means one can remove the absolute path name from the .forward file,
which cleans up the documentation on integration with Procmail.
Added a rather tacky check target to the Makefile.in to serve as a “sanity check” that doesn’t
require an extensive training databases. The scheme is to train the program with the source code
for annoyance−filter.w serving as the mail collection and statlib.w the junk bucket. Then those
programs themselves are tested, and the transcripts verified to confirm they were correctly classified.
Astute observers will ask where I get off using something which isn’t a well-formed mail folder to train
the program. Well, it works thanks to a gimmick I put into the probability calculation to keep it from
dividing by zero if one or both of the message counts were zero. That keeps anything untoward from