The Annoyance Filter.pdf - Fourmilab

16 A BRIEF HISTORY OF ANNOYANCE-FILTER ANNOYANCE-FILTER §9
deep enough for high diving. No more laps round the worry room for me!
I must confess to some inside information in this regard. While working for a regrettable employer in an
odious swamp, I’d twigged to the fact that network television advertisers tagged their commercials with a
signature in the vertical retrace interval to permit audit bureaux to measure how many network affiliates
actually broadcast each commercial. This tag appeared to me the Achilles’ heel of television advertising.
As long as one could distinguish tagged commercials from an un-tagged program, it would be more or less
straightforward to detect when a commercial was being transmitted and pause the VCR until the program
resumed.
If only. . . . In reality, only nationally broadcast commercials bore the tag, and only some of them. Local
commercials were never tagged. This created a difficult marketing dilemma for my grand scheme. While it
might have been possible to block some of the most ubiquitous and irritating commercials on mass-market
network series, the bottom feeders who watch those shows probably enjoyed the commercials and wouldn’t
be prospects for my gadget, while those like myself, infuriated by incessant commercials interrupting late
night movies, would find the device ineffective since local commercials on independent stations were never
tagged. Real-time analysis of video or even audio in the 1970’s and early 80’s was technologically out of
the question for a product aimed at a mass consumer market. So, I put the idea of an annoyance filter for
television aside and occupied myself with other endeavours.
We now arrive at the late 1980’s. I’d spent the last decade or so filling up the money bin more or less flat
out, and having reached a level I judged more than adequate, I began to turn my attention to matters I’d
neglected during those laser-focused years.
Writing science fiction, for one thing. There was something about the advertising filter which had dug its
way into my brain so deeply that nothing could dislodge it. The year is 1989; the Berlin Wall is about to
tumble; and I’m scribbling a story about two programmers spending the downtime between Christmas and
New Year’s Day (the period when I’d accomplished about half of my own productive work over the previous
half decade) prowling the nascent Internet for evidence of an extraterrestrial message already received, but
not recognised as such. In
We’ll Return, After this Message,
it is an annoyance filter which recognises an extraterrestrial message for what it is, advertising, and as
von Mises observed, distinguishable by its own strident clamouring for attention.
A decade later, in the very years in which I set my science fiction story, I launched my own search for a
message from our Creator hidden in the most obvious of locations—no results so far. Yet still I scour the
Net.
Which brings us, more or less, to the present. The idea of an annoyance filter continued to intermittently
occupy my thoughts, especially as the volume of junk arriving in my mailbox incessantly mounted despite
ongoing efforts to filter it with increasingly voluminous and clever Procmail rules. Then, in August 2002, my
friend and colleague Kern Sibbald brought to my attention Paul Graham’s brilliant design for an adaptable,
Bayesian filter to discriminate junk and legitimate mail by word frequencies measured in actual samples of
mail pre-sorted into those categories. Now that sounded promising! Here was a design which was simple in
concept, theoretically sound, and best of all, it seemed to work. Graham implemented his prototype filter in
the “Arc” Lisp dialect used in his research. I decided to build a deployable tool in industrial-strength C++,
founded on his design, and handling all the details required so the filter could, as much as possible, interpret
mail the same way a human would—decoding, translating, and extracting wherever necessary to defeat the
techniques junk mailers adopt to hide their content from nave filtering utilities.
This is not a simple task. Consider—you can probably sort out a message you’re interested in reading
from unsolicited junk in a fraction of a second, but that assumes it’s presented to you after all of the mail
transfer and content encodings have been peeled away to reveal the true colours of the content. Long gone
are days when E-mail was predominantly ASCII text. Today, it’s more than likely to be HTML (if not a
Flash animation or some other horror), often transmitted in Quoted−Printable or Base64 encodings largely
in the interest of “stealth”—to hide the content from filters not equipped with the decoding facilities of a
full-fledged mail client.