Modern compiler design [PDF]

Once a name is defined, it can be used in a regular expression for a lex rule by enclosing it in braces { and
}. For instance, if the line
DIGIT [0-9]
appeared in the Lex Definitions of a .lex file, then {DIGIT} would be a symbolic name for the regular
expression [0-9]. Anywhere [0-9] appears in a regular expression, we could use {DIGIT} instead. Thus the
rule:
{DIGIT}+ { return INTEGER_LITERAL; }
would be equivalent to the rule:
[0-9]+ { return INTEGER_LITERAL; }
An example of using named regular expressions is in Figure 2.2.
2.1.3 Tokens With Values
For tokens that don’t have values – ELSE, FOR, SEMICOLON, etc – we only care about which rule was
matched. No other information is needed. Some tokens, such as INTEGER LITERAL and IDENTIFIER,
have values. 57, for instance, is an INTEGER LITERAL token that has the value 57. foobar is an IDEN-
TIFIER token that has the value “foobar”. How can we have yylex return both which token was matched,
and the value of the token? We can use a global variable to communicate the extra information. yylex() can
set the value of the global variable before returning the token type. The function that calls yylex() can then
examine the value of this global variable to determine the value of the token. Let’s take a look at a slightly
more complicated lex file, simple2.lex (Figure 2.2). In the C declarations portion of the file, we have defined
a global variable yylval 1 .
union {
int integer_value;
char *string_value;
} yylval;
When we wish to return the value of a token, we will set the value of this global variable. Consider the
rule for INTEGER LITERAL:
[0-9]+ { yylval.integer_value = atoi(yytext);
return INTEGER_LITERAL; }
When we match an integer in the input file, we first set the integer value field of the global variable yylval.
To set the integer value field to the correct value, we need to know the actual string that was matched to the
regular expression [0-9]+. The variable yytext is automatically set to the value of the last string matched
by lex. Thus, all we need to do is convert that string to an integer value through the ascii-to-integer function
atoi defined in , and set the integer value field of the variable yylval. We can then return an
INTEGER LITERAL token.
The other token in this example that has a value is the IDENTIFIER token. The rule for an IDENTIFER
is:
[a-zA-Z][a-zA-Z0-9]* { yylval.string_value = malloc(sizeof(char) *
(strlen(yytext)+1));
strcpy(yylval.string_value,yytext);
return IDENTIFIER; }
1 Why do we call this variable yylval instead of a more meaningful name like tokenValue? For the final project, we will use
lex in conjunction with another tool, yacc. Yacc has a rather odd naming convention, and we have to follow it for the two tools
to work together.
7