The Annoyance Filter.pdf - Fourmilab

§256 ANNOYANCE-FILTER DEVELOPMENT LOG 215
COMPRESSED_FILES was not defined.
definition.
We now unconditionally define isc in the mailFolder class
With these fixes, the makew32.bat build on Win32 now works once again.
Added a testw32.bat file which runs a rudimentary test of the Win32 build similar to the check
target in Makefile.in. I added this file to both the dist and winarch archive generation targets in
Makefile.in.
Modified Makefile.in to replace the hard-coded /ftp/annoyance−filter destination with a PUBDEST
declaration at the top of the file which defaults to the same directory. This permits overriding the default
publication destination for use at another site or for nondestructive testing of new releases simply by
editing the Makefile. Some day, it might make sense to permit overriding this with an option at
./configure time, but this is not that day.
Release 0.1-RC4.
2002 October 11
Integrated the application string parsers for Flash and PDF formats, which were developed in a
separate stand-alone test program. These include the classes applicationStringParser (mother of all
application parsers), flashStream, flashTextExtractor, and pdfTextExtractor, the latter compiled
in only if all the utilities it needs to decode PDF via a pipe to pdftotext are present. At the moment,
these aren’t hooked up to the mail folder, but are merely exercised by code in the −−jig.
Integrated Knuth and Levy’s CWEB version 3.64 in the cweb directory. The CWEAVE and CTANGLE
programs are built with a change file, common−bigger.ch which increases the input line length limit to
400 characters as I did in the earlier 3.63 release.
Added plumbing to invoke Flash and PDF parsers for attachments with those application types. Thanks
to the inability to take a class member function as an unqualified function pointer, this is somewhat
tacky, requiring a pointer to the mailFolder to obtain decoded data.
2002 October 12
Added decoders and interpreters for Shift-JIS and Unicode (UCS-2, UTF-8, and UTF-16 encodings).
These are used to decode and interpret these character sets in Flash animations whose fonts are so
tagged.
Added logic to invoke the new Unicode UTF-8 decoder when a MIME part’s charset= designates it so
encoded.
2002 October 13
In the process of testing UTF-8 decoding of Unicode messages, I stumbled over a bug in ignoring HTML
comments embedded within tokens, a common trick in junk mail to evade naïve filters, for example,
“remove␣yourself”. (Yes, I know a valid HTML comment is supposed to contain a
space after the initial and before the final sentinel, but junk mail often violates this rule, counting on
sloppy browsers not to enforce the standard, so we must comply in the interest of “seeing what the user
would”.) HTML comments are now completely discarded, even when embedded within tokens.
The dist target in Makefile.in failed to clean the cweb directory before including it in the source
archive, which could have the result of leaving objects and binaries not compatible with the system on
which the user is installing. I modified the target to descend into the cweb directory and make␣clean.
This promptly ran into another problem because the CWEB Makefile deletes the C source for CWEAVE,
using the bootstrapped CTANGLE to re-build it. This is clean, but runs afoul of my rebuilding both
programs directly in the outer Makefile. I saved the original CWEB makefile as Makefile.ORIG and