The Annoyance Filter.pdf - Fourmilab

208 DEVELOPMENT LOG ANNOYANCE-FILTER §256
Modified the −−test option so that if the −−transcript option has been previously specified with
standard output as the destination (“−”), the junk probability is not written to standard output at the
end of the transcript.
2002 September 17
The Base64 decoder could hang if one of the lines it was decoding contained white space. Fixed.
Added logic to detect and discard header items which begin with our own Xfile sentinel. This shouldn’t
happen in the normal course of things, but somebody may try to spoof a downstream filter by sending
mail which contains a sentinel purporting to be a classification by of of its legitimacy. Deleting our
own header items also allow us to process our own transcripts containing them and reproduce the same
results as if they hadn’t been added.
Cleaned up the horrific 〈 Activate MIME decoder if required 153 〉 section which “jes’ grew” in mailFolder ::nextLine
as more and more complexities were cranked in to MIME part decoding, multiple byte character sets,
parsing ASCII strings out of binary data streams, etc.
2002 September 18
Cleaned up documentation of command line options, clarifying that they are logically commands which
must be specified in the order in which they are to be executed. In the process, I added an example
of invoking annoyance−filter as a pre-processor for a mail sorting program such as Procmail to the
“Quick and dirty user guide”.
Added a new annoyance−filter−run shell script to execute the program in default filter mode with
the executable and dictionary installed in the default “$HOME/.annoyance−filter” directory. Oh, you
haven’t hear about that. . .well, stay tuned. . .details in the next episode.
Incremental refinement of the README and INSTALL files, with many keystrokes to go before we put these
documents to sleep.
Added −−verbose tell-tales for the −−plot and −−statistics options.
Replaced the annoyance−filter.1 manual page with a cop-out which directs the esteemed reader to
the PDF program documentation. This thing is changing so rapidly that the last thing I need is to
maintain four copies of the bloody command line option documentation. Four? Think about it: the
program (CWEB), its embedded −−help option text, a Web page (nonexistent at the moment, thank
Bob), and a manual page. Keeping all four simultaneously in sync is something which could appeal
only to an accountant. I’m a programmer, not an accountant—I drink their blood, but I don’t do their
work.
The code which discards header lines we’ve generated attempted to remove lines from the transcript
even when no transcript was being generated, for example, when adding a message we’d previously
processed to the −−mail or −−junk database. This caused a Λ pointer reference in 〈 Check for lines with
our sentinel already present in the header 144 〉—fixed.
Hours of patient, unremunerated toil cleaning up Makefile.in to bash things into a distributable
form. I added an install target which installs the program in the default $HOME/.annoyance−filter
directory, creating a customised run program (annoyance−filter−run in the build directory) which
supplies the home directory which sendmail doesn’t. Massive clean-up of Makefile.in, yielding a
template which is far more generic for our next foray into software land.
2002 September 19
Further testing revealed that the segmentation fault in dictionary ::purge which I thought I fixed a
week or so ago was still lurking to bite the unwary soul whose dictionary contained a large number of
words eligible for purging. As far as I can determine, when you erase an item from a set, not only does