The Annoyance Filter.pdf - Fourmilab

§173 ANNOYANCE-FILTER TOKEN PARSER 145
173. Token parser.
A tokenParser reads lines from a mailFolder and returns tokens as defined by its active tokenDefinition.
Separate tokenDefinitions can be defined for use while parsing regular text and binary byte streams,
respectively. A tokenParser has the ability to save the lines parsed from a message in a messageQueue ,
permitting further subsequent analysis. Note that what is saved is “what the parser saw”—after MIME
decoding or elision of ignored parts.
〈 Class definitions 10 〉 +≡
class tokenParser {
protected:
mailFolder ∗source ;
string cl ;
string ::size type clp;
bool atEnd , inHTML, inHTMLcomment ;
tokenDefinition ∗td ; /∗ Token definition for text mode ∗/
tokenDefinition ∗btd ; /∗ Token definition for byte stream parsing ∗/
bool saveMessage ; /∗ Save current message in messageQueue ? ∗/
bool assemblePhrases ; /∗ Are we assembling phrases ? ∗/
deque〈string〉 phraseQueue ; /∗ Phrase assembly queue ∗/
deque〈string〉 pendingPhrases ; /∗ Queue of phrases awaiting return ∗/
public:
list〈string〉 messageQueue ; /∗ Current message ∗/
tokenParser( )
{
td = Λ;
}
void setSource (mailFolder &mf )
{
source = &mf ;
cl = "";
clp = 0;
atEnd = inHTML = inHTMLcomment = false ;
saveMessage = false ;
messageQueue .clear ( );
phraseQueue .clear ( );
pendingPhrases .clear ( );
〈 Check phrase assembly parameters and activate if required 179 〉;
}
void setTokenDefinition (tokenDefinition &t, tokenDefinition &bt )
{
td = &t;
btd = &bt ;
}
void setTokenLengthLimits (unsigned int lMax , unsigned int lMin = 1, unsigned int
blMax = 1, unsigned int blMin = 1)
{
assert(td ≠ Λ);
td ⃗ setLengthLimits (lMin , lMax );
assert(btd ≠ Λ);
btd ⃗ setLengthLimits (blMin , blMax );
}