The Annoyance Filter.pdf - Fourmilab

§256 ANNOYANCE-FILTER DEVELOPMENT LOG 225
Added a parser diagnostic to mailFolder ::nextLine to indicate when an applicationStringParser is
closed.
The close method of pdfTextExtractor failed to close the input stream it used to read the output
from the pipe connected to pdftotext, which caused (for some bizarre reason), the raw binary PDF
file to be returned, not the decoded text. I added the requisite close of the stream.
When pdfTextExtractor was transcribing the decoded attachment to the temporary file to be read
by pdftotext, it checked for end of file but not error conditions. I modified it to use isOK ( ) to govern
the copy loop.
The flashTextExtractor and its parent flashStream were not restartable because they did not
propagate the close up to the applicationStringParser from which all are derived, and because
flashTextExtractor did not reset its own initialised and textOnly at end of file. Fixed.
Because the flashStream decoder usually terminates upon seeing a stagEnd tag in the input stream,
it failed to read from the MIME decoder until end of file was encountered. This caused an extraneous
blank line to be inserted in the transcript at the end of the MIME-encoded data and before the part
end sentinel. I added logic to flashTextExtractor ::nextString to call get8 ( ) until an end of file is
reported before returning the logical end of file for the flash stream.
The input stream close I added to pdfTextExtractor ::close ran afoul of the fdistream logic used
to cope with gcc 3 which, helpfully, does not define a close method. I made the close conditional on
HAVE_FDSTREAM_COMPATIBILITY not being defined.
This time, our attempt to rebuild the Win32 version was torpedoed by getopt in yet another innovative
way. This time, the care we took to avoid including our own getopt.h stabbed us in the back, because
the library’s version (which I still haven’t figured out the reason it’s being included) doesn’t define the
long version of getopt , and wants a different symbol to do so than our include file. So, I added WIN32
conditional code before the include of our version to force it to be included and define the long option
version of getopt . This GCC/Cygwin “compatibility” is turning out to be a running bad joke.
Release 1.0a.
2003 September 23
A file whose name contained the string “.gz” (or whatever other compressed file extension was configured)
would be fed through the decompressor even if the sequence was embedded in the middle of the
file name. I modified the tests to deem a file compressed only if the Compressed file type string appears
at the end of the file name. This applies both to files named directly on the command line and files
within directories.
A PDF file which has been marked by its creator as view-only will not be processed by pdftotext—
no output is generated and the message “Error: Copying of text from this document is not
allowed.” is sent to standard output. There’s nothing we can do about this, absent making a version
of pdftotext which bypasses the PDF file security mechanisms. While there’s something to be said for
this, it’s well beyond the mandate of annoyance−filter.
An assertion added to flashStream ::ignoreTag in the process of debugging problems due to multiple
flash attachments could fail when −−bsdfolder mode was used to scan a mail or junk folder. I
commented out the assertion.
2003 September 24
Phil Karn (KA9Q) reported that on the latest Debian distribution, compilations failed due to a missing
definition of assert. As far as I can determine, assert.h was pulled in by other includes in earlier
libraries, but now must be included explicitly. I added the requisite includes to annoyance−filter.w
and statlib.w.