The Annoyance Filter.pdf - Fourmilab

144 TOKEN DEFINITION ANNOYANCE-FILTER §171
171. Initialise a tokenDefinition for parsing ISO-8859 text with our chosen defaults for punctuation
embedded in such tokens. Any pre-existing definitions are cleared.
〈 Class implementations 11 〉 +≡
void tokenDefinition ::setISO 8859defaults (unsigned int lmin , unsigned int lmax )
{
clear ( );
setLengthLimits (lmin , lmax );
for (unsigned int c = 0; c < 256; c++) {
isToken [c] = (isascii (c) ∧ isdigit (c)) ∨ isISOalpha (c) ∨ (c ≡ ’−’) ∨ (c ≡ ’\’’) ∨ (c ≡ ’$’);
notExclusively [c] = (isdigit (c) ∨ (c ≡ ’−’)) ? 1 : 0;
}
#define CI(x)static cast〈int〉 (x)
notAtEnd [CI(’−’)] = notAtEnd [CI(’\’’)] = true ;
#undef CI
}
172. Initialise a tokenDefinition for parsing US-ASCII text with our chosen defaults for punctuation
embedded in such tokens. Any pre-existing definitions are cleared.
〈 Class implementations 11 〉 +≡
void tokenDefinition ::setUS ASCIIdefaults (unsigned int lmin , unsigned int lmax )
{
clear ( );
setLengthLimits (lmin , lmax );
for (unsigned int c = 0; c < 128; c++) {
isToken [c] = isalpha (c) ∨ isdigit (c);
notExclusively [c] = (isdigit (c) ∨ (c ≡ ’−’)) ? 1 : 0;
}
#define CI(x)static cast〈int〉 (x)
isToken [CI(’_’)] = notExclusively [CI(’_’)] = true ;
notAtEnd [CI(’−’)] = notAtEnd [CI(’\’’)] = true ;
#undef CI
}