The Annoyance Filter.pdf - Fourmilab

152 TOKEN PARSER ANNOYANCE-FILTER §178
178. If the item being read from the mailFolder has been identified as a binary byte stream, read
it character by character and parse for probable strings. We use the byte stream tokenDefinition
btd to determine token composition, permitting stricter construction of plausible tokens in binary byte
streams.
We get here only when our source identifies itself as chewing through a byte stream with isByteStream .
While in a byte stream, the mailFolder permits calls to its nextByte method, which returns bytes
directly from the active stream decoder. At the end of the stream (usually denoted by the end sentinel
of the MIME part containing the stream), nextByte returns −1 and clears the byte stream indicator.
We escape from here when that happens, and go around the main loop in nextToken again, which will,
now that byte stream mode is cleared, resume dealing with the mail folder at the nextLine level, where
all of the housekeeping related to the end of the byte stream will be dealt with.
This code is so similar to the main loop it’s embedded in it should probably be abstracted out as
a token recogniser engine parameterised by the means of obtaining bytes and the token definition it
applies. I may get around to this when I’m next in clean freak mode, but for the nonce I’ll leave it as-is
until I’m sure no additional special pleading is required when cracking byte streams.
〈 Parse plausible tokens from byte stream 178 〉 ≡
int b;
while ((b = source ⃗ nextByte ( )) ≥ 0) {
/∗ Ignore non-token characters until start of next token ∗/
if (¬(btd ⃗ isTokenMember (b))) {
continue;
} /∗ Check for characters we don’t accept as the start of a token ∗/
if (btd ⃗ isTokenNotAtEnd (b)) {
continue;
} /∗ First character of token recognised; store and scan balance ∗/
if (btd ⃗ isTokenNotExclusively (b)) {
necount ++;
}
token += static cast〈char〉(b);
while (((b = source ⃗ nextByte ( )) ≥ 0) ∧ btd ⃗ isTokenMember (b)) {
if (btd ⃗ isTokenNotExclusively (b)) {
necount ++;
}
token += static cast〈char〉(b);
} /∗ Prune characters we don’t accept at the end of a token ∗/
while ((token .length ( ) > 0) ∧ btd ⃗ isTokenNotAtEnd (ChIx (token [token .length ( ) − 1]))) {
token .erase (token .length ( ) − 1);
} /∗ Verify that the token meets our minimum and maximum length constraints ∗/
if (¬(btd ⃗ isTokenLengthAcceptable (token ))) {
token = "";
continue;
} /∗ Verify that the token isn’t composed exclusively of characters permitted in a token but
not allowed to comprise it in entirety. ∗/
if (necount ≡ token .length ( )) {
token = "";
continue;
}
d.set(token );
d.toLower ( ); /∗ Convert to canonical form ∗/
〈 Check for phrase assembly and generate phrases as required 180 〉;
if (pTokenTrace ∧ saveMessage ) {
messageQueue .push back (string("␣␣\"") + d.text + "\"");