The Annoyance Filter.pdf - Fourmilab

§3 ANNOYANCE-FILTER GETTING STARTED 3
3. Getting started.
The Annoyance Filter is organised as a toolbox which can be used to explore content-based mail filtering.
It includes diagnostic tools and output which will eventually be little used once the program is tuned and
put into production.
The program is normally run in two phases. In the training phase, collections of legitimate and junk mail
stored in UNIX mail folders are read and used to build a dictionary in which the probability of a word’s
identifying a message as junk is computed. This dictionary is then exported to be used in subsequent runs
to classify incoming messages based on the word probabilities determined from prior messages.
3.1. Building
If you have a more or less standard present-day UNIX system, you should be able to build and install the
program with the commands:
./configure
make
make check
make install
3.2. Training
Now you must train the program to discriminate legitimate junk and mail by showing it collections of
such mail you’ve hand sorted into a pile of stuff you want to receive and another which you don’t. Assuming
you have mail folders containing collections of legitimate mail and junk named “m−good” and “m−junk”
respectively, you can perform the training phase and create a binary dictionary file named “dict.bin” and
a fast dictionary “fdict.bin” for classifying messages with the command:
annoyance−filter --mail m-good --junk m-junk --prune \
--write dict.bin --fwrite fdict.bin
The arguments to the −−mail and −−junk options can be either UNIX “mail folders” consisting of one
or more E-mail messages concatenated into a single file, or the name of a directory containing messages in
individual files. In either case, the files may be compressed with gzip—annoyance−filter will automatically
expand them. You can supply as many −−mail and −−junk options as you like on a command line; the
contents added cumulatively to the dictionary.
It is absolutely essential that the collections of legitimate and junk mail used to train annoyance−filter
be completely clean—no junk in the −−mail collection or vice versa. Pollution of either collection by messages
belonging in the other is very likely to corrupt the calculation of probabilities, resulting in messages which
belong in one category being assigned to the other. The utilities/splitmail.pl program can help in
manually sorting mail into the required two piles, and I hope some day I will have the time to adequately
document it.
You may find it worthwhile to add an archive of mail you’ve sent to the legitimate category with −−mail.
In many cases, the words you use in mail you send are an excellent predictor of how worthy an incoming
message is of your attention. I’ve found this works well with my own archives, but I haven’t tested how
effective it is for a broader spectrum of users.
When you compile the collections of junk and legitimate mail to train annoyance−filter, it’s important to
include all the copies of similar or identical messages you’ve received in either category. annoyance−filter
bases its classifications on the frequency of indicative words in the entire set of mail you receive. An obscure
string embedded in a mail worm spewed onto the net may not filter it out if you train annoyance−filter
with only one copy, but will certainly consign it to the junk heap if you train annoyance−filter with the
twenty or thirty you receive a day.
3.3. Scoring
Dictionary in hand, you can now proceed to the scoring phase, where the dictionary is used, along with
the list of words appearing in a message, to determine its overall probability of being junk. If you have a
mail message in a file “mail.txt”, you can compute and display its junk probability with:
annoyance−filter --fread fdict.bin --test mail.txt