The Annoyance Filter.pdf - Fourmilab

§5 ANNOYANCE-FILTER PHRASE-BASED CLASSIFICATION 9
5. Phrase-based classification.
annoyance−filter has the ability to classify messages based upon occurrences of multiple-word phrases
as well as individual words. Here are results from an empirical test of classifying messages by single word
frequencies compared to considering both individual words, phrases of 1–2 and 1–3 words, and phrases of two
to three words. With this test set (compiled by hand sorting three years of legitimate and junk mail), adding
classification by two word phrases reduces the number of false negatives (junk mail erroneously classified as
legitimate) by more than 90%, while preserving 100% accuracy in identifying legitimate mail.
Folder −−phrasemin −−phrasemax Total Mail Junk Prob
Junk 1 1 8957 37 8920 0.9970
Mail 1 1 2316 2316 0 0.0000
Junk 1 2 8957 3 8954 0.9997
Mail 1 2 2316 2316 0 0.0000
Junk 1 3 8957 9 8948 0.9983
Mail 1 3 2316 2316 0 0.0000
Junk 2 3 8957 9 8948 0.9981
Mail 2 3 2316 2316 0 0.0000
There’s no need to overdo it, however. Note that extending classification to phrases of up to three words
actually slightly reduced the accuracy with which junk was recognised. In most circumstances, classifying
based on phrases of one and two words will yield the best results.