Introduction to Lex - Faculty of Computer Science

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Introduction to Lex
• General Description
• Input file
• Output file
• How matching is done
• Regular expressions
• Local names
• Using Lex
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
General Description
• Lex is a program that automatically
generates code for scanners.
• Input: a description of the tokens in the
form of regular expressions, together
with the actions to be taken when each
expression is matched.
• Output: a text file with C source code
defining a procedure yylex() that is a table
implementing the DFA for the regular
expressions.
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Input File
• Lex input file is divided in three
parts
/* declarations*/
…
%%
/* rules */
…
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
%%
/* auxiliary functions*/
…

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
Input File – Declaration
• The declaration part includes the
assignment of names to regular
expressions in the form:
• It can also include C code external to the
definition of yylex() within %{ and %} in
the first column.
• Also, it is possible to specify some
options with the sintax %option

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Input File - Rules
• The rules part specifies what to do
when a regular expression is
matched
• Actions are normal C sentences
(can be a complex C sentence
between {}).
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
• %{ %} enclosed text appearing
before the first rule may be used to
declare local variables.

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
Input File – Aux Functions
• The auxiliary functions part is only
C code.
• It includes function definitions for
every function needed in the rule
part
• It can also contain the main()
function if the scanner is going to
be used as a standalone program.
• The main() function must call the
function yylex()

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
Code Generated by Lex
• The output of Lex is a file called
lex.yy.c
• It is a C function that returns an
integer, i.e., a code for a Token
• If it contains the main() function
definition, it must be compiled to
run.
• Otherwise, the code can be an
external function declaration for the
function int yylex()

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
How matching is done
• By running the generated scanner, it
analyses its input looking for strings that
match any of its patterns, and then
executes the action.
• If more than one match is found, it selects
the regular expression matching the
longest string.
• If it finds two or more matches of the same
length, the one listed first is selected.
• If no match is found, then the default rule
is executed: i.e., the next character in the
input is copied to the output.

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
Regular expressions
• ::= a the character a
• ::= sthe string s, even if s contains metacharacters
• ::= \a the character a when a is a metacharacter (e.g., *)
• ::= . any character except newline
• ::= [+] any of the character
• ::= [-] any character from to
• ::= [^+] any character except those
• ::= * zero or more repetitions of
• ::= + one or more repetitions of
• ::= ? zero or one repetitions of
• ::= | or
• ::= followed by
• ::= () same as
• ::= {} the named regular expression in the
definitions part

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
Internal names
• The rules, inside the action
definition, can refer to the following
variables:
– yytext, the string being matched
(lexeme)
– yyin, the input file
– yyout, the output file
– ECHO, the default rule action
– yyval, the global variable for
communicating the Attribute for a
Token to the Parser

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Using Lex
• There are several lex versions.
We're going to use flex.
• In order to maximize compatibility,
use -l option when compiling, and
%option noyywrap in the definition
part of the Lex input file.
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Example 1
%{
#include
%}
%%
[0-9]+ { printf("%s\n", yytext); }
.|\n ;
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
%%
main()
{
yylex();
}

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Example 2
%{
int c=0, w=0, l=0;
%}
word [^ \t\n]+
eol \n
%%
{word} {w++; c+=yyleng;};
{eol} {c++; l++;}
. {c++;}
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
%%
main()
{
}
yylex();
printf("%d %d %d\n", l, w, c);

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Example 3
%{
int tokenCount=0;
%}
%%
[a-zA-Z]+ { printf("%d WORD \"%s\"\n",
++tokenCount, yytext); }
[0-9]+ { printf("%d NUMBER \"%s\"\n",
++tokenCount, yytext); }
[^a-zA-Z0-9]+ { printf("%d OTHER \"%s\"\n",
++tokenCount, yytext); }
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
%%
main() { yylex(); }

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Example 4
%{
#include
int lineno =1;
%}
line .*\n
%%
{line} {printf("%5d %s", lineno++, yytext); }
%%
int main() {
yylex(); return 0; }
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Example 5
%{
#include
%}
comment_line \/\/.*\n
%%
{comment_line} { printf("%s\n", yytext); }
.*\n ;
%%
int main() {
yylex(); return 0;
}
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Example 7
%{
#include
%}
digit [0-9]
number {digit}+
%%
{number}
. {;}
%%
{ int n = atoi(yytext);
printf("%x", n); }
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
int main() {
yylex(); return 0;
}

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Example 8 (1/4)
%{
#include "globals.h"
#include "util.h"
#include "scan.h"
int lineno=0;
FILE *listing;
FILE *code;
FILE *source;
code
int TraceScan = 1;
int EchoSource = 1;
// used to output source code listing
// used to output assembly code
// used to input tiny program source
/* lexeme of identifier or reserved word */
char tokenString[MAXTOKENLEN+1];
%}
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
digit [0-9]
number {digit}+
letter [a-zA-Z]
identifier {letter}+
newline \n
whitespace [ \t]+

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
Example 8 (2/4)
%%
"if" {return IF;}
"then" {return THEN;}
"else" {return ELSE;}
"end" {return END;}
"repeat" {return REPEAT;}
"until" {return UNTIL;}
"read" {return READ;}
"write" {return WRITE;}
":=" {return ASSIGN;}
"=" {return EQ;}
"

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Example 8 (3/4)
{number} {return NUM;}
{identifier} {return ID;}
{newline} {lineno++;}
{whitespace} {/* skip whitespace */}
"{" { char c;
do
{ c = input();
if (c == '\n') lineno++;
} while (c != '}');
}
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science

Principles of Compilers - 2/03/2006
Pablo R. Fillottrani
Example 8 (4/4)
%%
TokenType getToken(void)
{ static int firstTime = TRUE;
TokenType currentToken;
if (firstTime)
{ firstTime = FALSE;
lineno++;
yyin = source=stdin;
yyout = listing=stdout; }
currentToken = (TokenType)yylex();
strncpy(tokenString,yytext,MAXTOKENLEN);
if (TraceScan) {
fprintf(listing,"\t%d: ",lineno);
printToken(currentToken,tokenString);
}
return currentToken; }
FREE
UNIVERSITY OF
BOZEN–BOLZANO
Faculty of
Computer Science
int main(){
TraceScan = TRUE;
while( getToken() != ENDFILE);
return 0; }