The Annoyance Filter.pdf - Fourmilab

216 DEVELOPMENT LOG ANNOYANCE-FILTER §256
modified the clean target in the actual Makefile to leave cweave.c around. I also modified our own
clean target to clean the cweb directory as well.
Attempting to build .dvi or pdf targets after you’d cleaned the cweb directory failed for lack of cweave;
I added a dependency to Makefile.in to ensure it’s rebuilt when needed.
Since certain recent versions of gcc libraries have begun to natter if C++ include files specify the .h
extension (which, for years, was required by those self-same libraries), I eliminated them from our list
of includes, which finally seems to work on gcc 2.96. Doubtless this will torpedo somebody using an
earlier version.
Broke up the unreadably monolithic list of include files into sections which explain what’s what.
Dooooh! Forgot to disable the declaration of the pdfTextExtractor in mailFolder when HAVE_PDF_DECODER
was not defined, which was the undoing of the Win32 build; fixed.
Release 0.1-RC5.
2002 October 19
Added a check in classifyMessages to verify that a dictionary has been loaded before attempting to
classify a message. If no dictionary is present, a warning is written to standard error and the junk
probability is returned as 0.5.
Added a warning if command line are specified after a −−classify command. Since this command
always exits with an exit code indicating the classification, specifying subsequent arguments is always
an error.
Added a bunch of consistency checking for combinations of options which don’t make any sense and
suggest the user doesn’t understand in which order they should be specified. To facilitate this, I modified
the code for the −−classify option to set a new lastOption flag to bail out of the option processing
loop and set exitStatus to the classification rather than exiting directly before the option consistency
checks are performed. This cleans up the control structure in any case.
In the process of adding the above code, I discovered that the any ( ) method of bitset seems to be
broken in the glibc which accompanies gcc 2.96. I tested count ( ) against zero and that seems to work
OK.
Implemented phrase tokens. You can consider phrases of consecutive tokens as primitive tokens by
specifying the minimum and maximum words composing a phrase with the −−phrasemin and phrasemax
options. These default to 1 and 1, which suppresses all phrase-related flailing around. If set otherwise,
tokens are assembled into a queue and all phrases within the length bounds are emitted as tokens. How
well this works is a research question we may now address with the requisite tool in hand.
2002 October 20
Added code to import a binary dictionary file with the −−read option using memory-mapped I/O if
./configure detects that facility and defines HAVE_MMAP. This isn’t a big win on individual runs of
the program, but if you’re installing it on a high volume server, multiple read-only references to the
dictionary file (be sure to make the file read-only, by the way) can simply bring the file into memory
where it is re-used by multiple instances of the program. (Of course, if the system has an efficient file
system cache, that may work just as well, but there’s no harm in memory mapping in any case.) Thanks
to the C++ theologians who deprecated the incredibly useful strstream facility, which is precisely what
you need to efficiently access a block of memory mapped data as a stream, I included a copy of the
definition of this facility in mystrstream.h so we don’t have to depend on the C++ library providing it.
I was a little worried about writing phrases in CSV format without quoting the fields, but I did an
experiment with Excel and discovered it doesn’t quote such fields either—it only uses quotes if the cell