Lecture 3: Syntax: Grammars, Derivations, Parse Trees. Scanning.

Lecture 3: Syntax: Grammars, Derivations, Parse Trees. 

Scanning. 

September 1st, 2010 

1

Lecture Outline 

Programming Languages—Syntactic Specifications and Analysis 

Formal Grammars 

Backus-Naur Form 

Classification of Formal Languages 

Syntactic Analysis of Programs 

Derivations 

Syntax Trees 

Ambiguity 

Avoiding Ambiguity 

Scanning 

Summary 

2









Ambiguity 



Summary 

3

Formal Grammars contd 

How to use a grammar to generate sentences? 

1. Let σ be a sequence containing just the start variable: σ = v s . 

4




2. While σ contains any non-terminals, do: 

4





2.1 Choose one non-terminal (say, v) in σ. 

4






2.2 From R choose a rule (say, r) in which v appears on the left-hand side. 

4







2.3 Replace the chosen occurence of v in σ with the right-hand side of r. 

4








3. Return σ. 

4








3. Return σ. 

What if σ contains a non-terminal v for which there is no rule in R that would 

have v at its left-hand side? 

4








3. Return σ. 

What if σ contains a non-terminal v for which there is no rule in R that would 

have v at its left-hand side? 

◮ The grammar is incomplete. 

4


Example (Formal grammar) 

◮ V = {c} 

◮ S = {a,b} 

◮ R = {(c, ǫ), (c,aca), (c,bcb)} 

◮ v s = c 

5



◮ V = {c} 

◮ S = {a,b} 

◮ R = {(c, ǫ), (c,aca), (c,bcb)} 

◮ v s = c 

◮ Is the string abacaba valid in L? 

5



◮ V = {c} 

◮ S = {a,b} 

◮ R = {(c, ǫ), (c,aca), (c,bcb)} 

◮ v s = c 

◮ Is the string abacaba valid in L? No; c /∈ S. 

◮ Is ababbbaba valid in L? 

5



◮ V = {c} 

◮ S = {a,b} 

◮ R = {(c, ǫ), (c,aca), (c,bcb)} 

◮ v s = c 


◮ Is ababbbaba valid in L? No; this string’s length is not even. 

◮ What is the language L generated by the grammar? 

5



◮ V = {c} 

◮ S = {a,b} 

◮ R = {(c, ǫ), (c,aca), (c,bcb)} 

◮ v s = c 


◮ Is ababbbaba valid in L? No; this string’s length is not even. 

◮ What is the language L generated by the grammar? The set of all 

even-length palindromes over the alphabet S. 

5









Ambiguity 



Summary 

6


BNF Notation 

◮ Grammars are usually written using a special notation: the Backus-Naur 

Form (BNF). 

◮ BNF is often extended with convenience symbols to shorten the notation: 

the Extended BNF (EBNF). 

◮ BNF (and EBNF) is a metalanguage, a language for talking about 

languages. 

◮ We will use EBNF extensively during the course. 

7

Backus-Naur Form contd 

Elements of BNF 

Terminals are distinguished from non-terminals (variables) by some 

typographical convention, for example: 

◮ non-terminals are written in italics, using angle brackets, etc.; 

◮ terminals are written in a monotype font, enclosed in quotation marks, 

etc. 

Rules are written as strings which contain: 

◮ a non-terminal, 

◮ a special ‘production’ symbol (typically, ‘::=’), 

◮ a sequence of terminals and non-terminals, or the symbol ‘ǫ’. 

By convention, 

◮ the terminals and non-terminals of the grammar are those, and only 

those, included in at least one of the rules; 

◮ the left-hand side (the first element) of the topmost rule is the start 

variable v s . 

8


Example (BNF representation of a grammar, Γ 1 ) 

〈c〉 ::= ǫ 

〈c〉 ::= a〈c〉a 

〈c〉 ::= b〈c〉b 

In this Γ 1 , 

◮ V = {〈c〉}, 

◮ S = {a,b}, 

◮ R = {(〈c〉, ǫ), (〈c〉,a〈c〉a), (〈c〉,b〈c〉b)}, 

◮ v s = 〈c〉. 

The specified language L(Γ 1 ) is: 

◮ L(Γ 1 ) = {ǫ,aa,bb,aaaa,baab,abba,bbbb,aaaaaa,baaaab, . . . } 

9


Example (EBNF representation of a grammar, Γ 1 ) 

The grammar can be also written as 

or as 

〈c〉 ::= ǫ 

| a〈c〉a 

| b〈c〉b 

〈c〉 ::= ǫ | a〈c〉a | b〈c〉b 

◮ The special symbol ‘|’ has the meaning of ‘or’, and is an element of the 

metalanguage, not the language specified by the grammar. 

10


Metasyntactic extensions 

Convenient extensions to the metalanguage inlcude: 

◮ the special symbols ‘[’ and ‘]’ used to enclose a subsequence that appears 

in the string at most once; 

◮ the special symbols ‘{’ and ‘}’ used to enclose a subsequence that appears 

in the string any number of times. 1 

Alternatively, we can use only the symbols ‘{’ and ‘}’ together with a 

superscript to specify the number of occurences: 

◮ ‘{ 〈sequence〉 } 2 ’ means two subsequent occurences of 〈sequence〉; 

◮ ‘{ 〈sequence〉 } + ’ means at least one occurence of 〈sequence〉; 

◮ ‘{ 〈sequence〉 } ∗ ’ means any number of occurences of 〈sequence〉; 

Further extensions are possible (and are sometimes used). 

1 The Kleene closure. 

11









Ambiguity 



Summary 

12

Chomsky’s Hierarchy of Languages 

Noam Chomsky defined four classes of languages: 

Type 0: Unconstrained Languages 

13




Type 1: Context-Sensitive Languages 

13





Type 2: Context-Free Languages 

13





Type 2: Context-Free Languages 

Type 3: Regular Languages 

13

Chomsky’s Hierarchy of Languages contd 

Note: 

◮ All regular languages are context-free, but not all context-free languages 

are regular. 

14


Note: 

◮ All regular languages are context-free, but not all context-free languages 

are regular. 

◮ All context-free languages are context-sensitive [sic], but not all 

context-sensitive languages are context-free. 

etc. 

This may sound unintuitive, but it follows a well-established convention. 

14

Regular Grammars 

What is a regular language? 

A regular language is a language generated by a regular grammar. 

◮ In a regular grammar, all rules are of one of the forms: 2 

v ::= s v ′ 

v ::= s 

v ::= ǫ 

where s ∈ S; v, v ′ ∈ V; and it is not required that v ≠ v ′ . 

2 These are right-regular grammars. In left-regular grammars, the first rule form above is replaced 

by ‘v ::= v ′ s’. 

15

Regular Grammars 

What is a regular language? 

A regular language is a language generated by a regular grammar. 

◮ In a regular grammar, all rules are of one of the forms: 2 

v ::= s v ′ 

v ::= s 

v ::= ǫ 

where s ∈ S; v, v ′ ∈ V; and it is not required that v ≠ v ′ . 

Example (A regular grammar) 

〈string〉 ::= a〈substring〉 | b〈substring〉 

〈substring〉 ::= ǫ | c〈substring〉 

Regular grammars are conveniently expressed with regular expressions. The 

above could be written as ‘(a|b)c*’, ‘(?:a|b)c*’, or ‘[ab]c*’, etc. 

2 These are right-regular grammars. In left-regular grammars, the first rule form above is replaced 

by ‘v ::= v ′ s’. 

15

Context-Free Grammars 

What is a context-free language? 

A context-free language is a language generated by a context-free grammar. 

◮ In a context-free grammar, all rules are of the form: 

v ::= γ 

where v ∈ V and γ ∈ (V ∪ S) ∗ (the set of all sequences of variables from 

V and symbols from S). 3 

3 (V ∪ S) ∗ is the Kleene closure of V ∪ S. 

16

Context-Free Grammars 

What is a context-free language? 

A context-free language is a language generated by a context-free grammar. 

◮ In a context-free grammar, all rules are of the form: 

v ::= γ 

where v ∈ V and γ ∈ (V ∪ S) ∗ (the set of all sequences of variables from 

V and symbols from S). 3 

Example (A non-regular context-free grammar) 

〈expression〉 ::= 〈number〉 

| 〈expression〉〈operator〉〈expression〉 

| ( 〈expression〉 ) 

| . . . 

3 (V ∪ S) ∗ is the Kleene closure of V ∪ S. 

16

Context-Free Grammars contd 

17

Context-Sensitive and Unconstrained Languages 

What is a context-sensitive language? 

A context-sensitive language is a language generated by a context-sensitive 

grammar. 

◮ In a context-sensitive grammar, all rules are of the form: 

where v ∈ V, and α,β, γ ∈ (V ∪ S) ∗ . 

αvβ ::= αγβ 

18

Context-Sensitive and Unconstrained Languages 

What is a context-sensitive language? 

A context-sensitive language is a language generated by a context-sensitive 

grammar. 

◮ In a context-sensitive grammar, all rules are of the form: 

where v ∈ V, and α,β, γ ∈ (V ∪ S) ∗ . 

αvβ ::= αγβ 

What is an unconstrained language? 

An unconstrained language is a language generated by an unrestricted 

grammar. 

◮ In an unrestricted grammar, all rules are of the form: 

α ::= β 

where α, β ∈ (V ∪ S) ∗ and α is non-empty. 

18


Why care about the hierarchy of languages? 

◮ Different grammars have different computational complexity: 

unconstrained ≻ context-sensitive ≻ context-free ≻ regular 

4 CTMCP uses the term ‘lexical syntax’ rather than ‘microsyntax’; others use the term ‘lexical 

structure’. 

5 Macrosyntax is usually referred to as ‘syntactic structure’. 

6 The less restrictive the metalanguage used to define the grammar, the more restrictive can the 

grammar be wrt. to the specified language. 

19





◮ Regular grammars are commonly used to define the microsyntax of 

programming languages—the syntax of lexemes as sequences of symbols 

from the alphabet of characters. 4 


structure’. 




19








◮ Context-free grammars are used to define (macro)syntax of programming 

languages—the syntax of programs as sequences of symbols from the 

alphabet of tokens (classified lexemes). 5 


structure’. 




19








◮ Context-free grammars are used to define (macro)syntax of programming 

languages—the syntax of programs as sequences of symbols from the 

alphabet of tokens (classified lexemes). 5 

◮ Additional constraints may be needed to further constrain the syntax, 

e.g., by specifying that variable identifiers can be used only after they 

have been declared, etc. 6 


structure’. 




19









Ambiguity 



Summary 

20


How are programs processed? 

◮ The initial input is linear—it is a sequence of symbols from the alphabet 

of characters. 

◮ A lexical analyzer (scanner, lexer, tokenizer) reads the sequence of 

characters and outputs a sequence of tokens. 

◮ A parser reads a sequence of tokens and outputs a structured (typically 

non-linear) internal representation of the program—a syntax tree (parse 

tree). 

◮ The syntax tree is further processed, e.g., by an interpreter or by a 

compiler. 

We have seen some of these steps implemented in the mdc interpreter. 7 

7 There, both the microsyntax and the syntax were trivial, no parsing was really needed as the 

intermediate representation was linear and colinear with the list of tokens, and no compilation was 

developed. 

21

Syntactic Analysis of Programs contd 

How are programs processed? contd 

Program: if X == 1 then . . . 

22




Input: ‘i’ ‘f’ ‘ ’ ‘X’ ‘ ’ ‘=’ ‘=’ ‘ ’ ‘t’ ‘h’ ‘e’ ‘n’ . . . 

22




Input: ‘i’ ‘f’ ‘ ’ ‘X’ ‘ ’ ‘=’ ‘=’ ‘ ’ ‘t’ ‘h’ ‘e’ ‘n’ . . . 

Lexemization: ‘if’ ‘X’ ‘==’ ‘1’ ‘then’ . . . 

22




Input: ‘i’ ‘f’ ‘ ’ ‘X’ ‘ ’ ‘=’ ‘=’ ‘ ’ ‘t’ ‘h’ ‘e’ ‘n’ . . . 


Tokenization: key(‘if’) var(‘X’) op(‘==’) int(1) key(‘then’) . . . 

22




Input: ‘i’ ‘f’ ‘ ’ ‘X’ ‘ ’ ‘=’ ‘=’ ‘ ’ ‘t’ ‘h’ ‘e’ ‘n’ . . . 



Parsing: program(ifthenelse(eq(var(‘X’) 

int(1)) 

. . . 

. . . ) 

. . . ) 

22




Input: ‘i’ ‘f’ ‘ ’ ‘X’ ‘ ’ ‘=’ ‘=’ ‘ ’ ‘t’ ‘h’ ‘e’ ‘n’ . . . 



Parsing: program(ifthenelse(eq(var(‘X’) 

int(1)) 

. . . 

. . . ) 

. . . ) 

Interpretation: actions according to the program and language semantics 

Compilation: code generation according to the program and language 

semantics 

22


Example (Partial microsyntax of Oz, using Perl-style regexes) 

〈variable〉 ::= [A..Z][A..Za..z0..9_]* 

A variable (a variable name) consists of an uppercase letter followed by any 

number of word characters. 

◮ Variable is valid as a variable name, atom and 123 are not. 

23


Example (Partial microsyntax of Oz, using Perl-style regexes) 

〈variable〉 ::= [A..Z][A..Za..z0..9_]* 

A variable (a variable name) consists of an uppercase letter followed by any 

number of word characters. 

◮ Variable is valid as a variable name, atom and 123 are not. 

Example (Partial microsyntax of Oz, using POSIX classes) 

〈atom〉 ::= [[:lower:]][[:word:]]* 

additional constraint: no keyword is an atom 

An atom consists of a lowercase letter followed by any number of word 

characters. 

◮ variable is valid as an atom, Atom and 123 are not. 

23


Example (Partial syntax of Oz) 

〈statement〉 ::= skip 

| if 〈variable〉 then 〈statement〉 else 〈statement〉 end 

| . . . 

where skip, if, then, else, and end are symbols from the alphabet of 

lexemes. 

◮ ‘if X then skip else if Y then skip else skip end end’ is a valid 

statement in Oz; 

◮ ‘if X then skip end’ and ‘if x then skip else skip end’ are not. 8 

8 The former is not valid in the Oz kernel language, but is valid in the syntactically extended 

version. 

24


Note: It is convenient to use indentation to make the structure of a program 

clear to the programmer, but (in Oz) this is inessential for the syntactic 

and semantic validity of programs. 

Example (Indentation in Oz) 

if A then 

skip 

else 

if B then 

if C then 

skip 

else 

skip 

end 

else 

skip 

end 

end 

25


Note: In some programming languages indentation is essential for the 

syntactic and semantic validity of programs. 

Example (Indentation in Python) 

# valid function definition 

def foo(bar): 

print bar 

return foo 

# invalid 

def foo(bar): print bar 

return foo 

# invalid 

def foo(bar): 

print bar 

return foo 

26


Note: In some programming languages the programmer has control of 

whether indentation is essential for the syntactic and semantic validity 

of programs or not. 

Example (Indentation in F#) 

(* valid, no indentation required *) 

let hello = 

fun name -> printf "hello, %a" name 

(* invalid, 4-space indentation required *) 

#light 

let hello = 

fun name -> printf "hello, %a" name 

27









Ambiguity 



Summary 

28



Following the recipe for using a grammar explained earlier, we can derive 

sentences in the language L(Γ) specified by a grammar Γ in a sequence of 

steps. 

◮ In each step we transform one sentential form (a sequence of terminals 

and/or non-terminals) into another sentential form by replacing one 

non-terminal with the right-hand side of a matching rule. 

◮ The first sentential form is the start variable v s alone. 

◮ The last sentential form is a valid sentence, composed only of terminals. 

Sequences of sentential forms starting with v s and ending with a sentence in 

L(Γ) obtained as specified above are called ‘derivations’. 

29

Derivations contd 

The following are two of infinitely many derivations possible to obtain with 

the previously defined grammar Γ 1 . 9 

Example (Derivation using Γ 1 ) 

1. 〈c〉 

2. a〈c〉a 

3. ab〈c〉ba 

4. abba 

Example (Derivation using Γ 1 ) 

1. 〈c〉 

2. 

9 〈c〉 ::= ǫ | a〈c〉a | b〈c〉b . 

30


Rightmost and leftmost derivations 

A derivation is a sequence of sentential forms beginning with a single 

nonterminal and ending with a (valid) sequence of terminals. 

◮ A derivation such that in each step it is the leftmost non-terminal that is 

replaced is called a ‘leftmost derivation’. 

◮ A derivation such that in each step it is the rightmost non-terminal that is 

replaced is called a ‘rightmost derivation’. 

◮ There can be derivations that are neither leftmost nor rightmost. 

31


Rightmost and leftmost derivations 

A derivation is a sequence of sentential forms beginning with a single 

nonterminal and ending with a (valid) sequence of terminals. 

◮ A derivation such that in each step it is the leftmost non-terminal that is 

replaced is called a ‘leftmost derivation’. 

◮ A derivation such that in each step it is the rightmost non-terminal that is 

replaced is called a ‘rightmost derivation’. 

◮ There can be derivations that are neither leftmost nor rightmost. 

Given a start variable v and a sequence s of terminals, there can be 

◮ no derivation of s from v (if s is not valid in the defined language); 

◮ exactly one derivation of s from v; 

◮ more than one derivation. 

31


Example (A leftmost derivation) 

1. 〈statement〉 

32




2. if 〈variable〉 then 〈statement〉 else 〈statement〉 end 

32





3. if A then 〈statement〉 else 〈statement〉 end 

32






4. if A then skip else 〈statement〉 end 

32







5. if A then skip else 

if 〈variable〉 then 〈statement〉 else 〈statement〉 end 

end 

32









end 

. . . 


if B then 

if C then else skip end 

else skip end 

end 

32


Example (A rightmost derivation) 



3. if 〈variable〉 then 〈statement〉 else 


end 

. . . 


if B then 

if C then else skip end 

else skip end 

end 

33









Ambiguity 



Summary 

34


Syntax tree 

A parse tree (a syntax tree) is a structured representation of a program. 

◮ Parse trees are generate in the process of parsing programs. 

◮ A parser is a function (a program) that takes as input a sequence of 

tokens (the output of a lexer) and returns a nested data structure 

corresponding to a parse tree. 

The data structure returned by the parser is an internal (intermediate) 

representation of the program. A parse tree can be used to: 

◮ interpret the program (in interpreted langagues); 

◮ generate target code (in compiled languages); 

◮ optimize the intermediate code (in both interpreted and compiled 

languages). 

35


Example (Syntax tree) 

Let Γ have the following rule(s): 

〈v〉 ::= ǫ | a〈v〉 | 〈v〉b | 〈v〉〈v〉 

Does the sequence ‘ba’ belong to L(Γ)? 

36




〈v〉 ::= ǫ | a〈v〉 | 〈v〉b | 〈v〉〈v〉 

Does the sequence ‘ba’ belong to L(Γ)? Yes, it has the following parse tree: 

〈v〉 

〈v〉 

〈v〉 

〈v〉 

b 

a 

〈v〉 

ǫ 

ǫ 

How many distinct derivations lead from 〈v〉 to ‘ba’? 

36




〈v〉 ::= ǫ | a〈v〉 | 〈v〉b | 〈v〉〈v〉 

Does the sequence ‘ba’ belong to L(Γ)? Yes, it has the following parse tree: 

〈v〉 

〈v〉 

〈v〉 

〈v〉 

b 

a 

〈v〉 

ǫ 

ǫ 

How many distinct derivations lead from 〈v〉 to ‘ba’? 

◮ There are six such derivations (check this!). 

36

Syntax Trees contd 

Example (A simple syntax tree for Oz) 

The Oz grammar includes the following rules: 



with the microsyntactic definition of 〈variable〉 given earlier. What is the 

parse tree for ‘if A then skip else if B then skip else skip end end’? 

37


Example (A simple syntax tree for Oz) 

The Oz grammar includes the following rules: 



with the microsyntactic definition of 〈variable〉 given earlier. What is the 

parse tree for ‘if A then skip else if B then skip else skip end end’? 

〈statement〉 

if 〈variable〉 

then 〈statement〉 

else 


end 

A 

skip 



else 〈statement〉 

end 

B 

skip 

skip 

37


Suppose we rewrite the grammar above as 


| if 〈variable〉 then 〈statement〉 else 〈statement〉 

| if 〈variable〉 then 〈statement〉 

How many syntax trees does ‘if A then if B then skip else skip’ have, 

given this grammar? 

38


Suppose we rewrite the grammar above as 


| if 〈variable〉 then 〈statement〉 else 〈statement〉 

| if 〈variable〉 then 〈statement〉 

How many syntax trees does ‘if A then if B then skip else skip’ have, 

given this grammar? There are two parse trees for this sequence—see the next 

slide. 

38


Example (Parse tree for ‘if A then if B then skip else skip’) 



then 


A 




B 

skip 

skip 

39





then 


A 




B 

skip 

skip 





else 


A 



skip 

B 

skip 

39


Does it matter that a sentence has more than one parse tree? 

40


Does it matter that a sentence has more than one parse tree? 

◮ For a sentence like 

if A then if B then skip else skip 

where all the conditional actions are skip (‘do nothing’, ‘noop’), it does 

not matter much. 

◮ In general, it does matter, since what actions will be taken and in which 

order depends on how the program is ‘understood’ by the interpreter (or 

compiler), which in turn depends on how the program is parsed. 

It is therefore essential that 

◮ the specification of the syntax is unambiguous, and 

◮ the programmer does not make false assumptions about how the code 

will be parsed. 

40


Example (The if-then-else construct in Python) 

Given these two pieces of code, what is the output for each possible 

combination of values if both a and b can have a value from {True,False}? 

1. if a: 

if b: print 1 

else: print 2 

2. if a: 

if b: print 1 

else: print 2 

41


Example (The if-then-else construct in Python) 

Given these two pieces of code, what is the output for each possible 

combination of values if both a and b can have a value from {True,False}? 

1. if a: 

if b: print 1 

else: print 2 

2. if a: 

if b: print 1 

else: print 2 

◮ a = True,b = True: both print ‘1’ 

◮ a = True,b = False: the first prints ‘2’, the second nothing 

◮ a = False,b = True: the second prints ‘2’, the first nothing 

◮ a = False,b = False: the second prints ‘2’, the first nothing 

The lack of ‘end’ would add to the grammar ambiguity which is resolved by 

involving whitespace in the specification. 

41


Example (Multistatement lines in Python) 

In Python, colon (‘;’) can be used to separate multiple statements within one 

line. 10 Which of the following are equivalent? 

1. if a: print 1; print 2 

2. if a: 

print 1 

print 2 

3. if a: 

print 1 

print 2 

10 Multistatement lines are considered bad practice in Python. 

42






2. if a: 

print 1 

print 2 

3. if a: 

print 1 

print 2 

◮ 1. is equivalent to 2. 

◮ What about ‘if a: if b: print 1; else print 2’? 11 


42






2. if a: 

print 1 

print 2 

3. if a: 

print 1 

print 2 

◮ 1. is equivalent to 2. 

◮ What about ‘if a: if b: print 1; else print 2’? 11 


11 Invalid syntax. 

42









Ambiguity 



Summary 

43

Ambiguity 

Ambiguity 

A grammar is ambiguous if a sentence can be parsed in more than one way: 

◮ the program has more than one parse tree, that is, 

◮ the program has more than one leftmost derivation. 12 

Note: The fact that a program has more than one derivation is not sufficient 

to consider the grammar ambiguous. 

◮ In practice, most programs have more than one derivation, but all 

these derivations correspond to the same parse tree—the grammar 

is unambiguous. 

◮ Two distinct leftmost derivations for the same program must 

correspond to two distinct parse trees—the grammar must be 

ambiguous in this case. 

12 Or more than one rightmost derivation. 

44

Ambiguity contd 

Example (An ambiguous grammar) 

Let Γ exp be a grammar including the following rules: 

〈expression〉 ::= 〈integer〉 


〈operator〉 ::= - | + | * | / 

where 〈integer〉 may generate any integer numeral (a sequence of digits). 

Why is Γ exp ambiguous? 

45


Example (An ambiguous grammar) 

Let Γ exp be a grammar including the following rules: 



〈operator〉 ::= - | + | * | / 

where 〈integer〉 may generate any integer numeral (a sequence of digits). 

Why is Γ exp ambiguous? 

◮ Sentences like ‘1 + 2 + 3’ have more than one parse tree. 

◮ Worse, sentences like ‘1 + 2 * 3’ have more than one parse tree. 

Should ‘1 + 2 * 3’ evaluate to 9 or to 7? 

◮ In Smalltalk, the result would be 9. 

◮ In general, we would like it to be 7. 

45


Example (An ambiguous grammar contd) 

The expression ‘1 + 2 * 3’ has two parse trees: 

〈expression〉 


〈operator〉 


〈integer〉 

- 




1 

〈integer〉 

* 

〈integer〉 

2 

3 








* 

〈integer〉 

〈integer〉 

- 

〈integer〉 

3 

1 

2 

46









Ambiguity 



Summary 

47


There are a number of ways to avoid ambiguity in grammars. Here, we 

consider four alternative solutions. 

Solution 1: Obligatory parentheses 

We can modify Γ exp by enforcing parentheses around complex expressions: 


| (〈expression〉〈operator〉〈expression〉) 

〈operator〉 ::= - | + | * | / 

Benefit: Ambiguity has been resolved. 

48


There are a number of ways to avoid ambiguity in grammars. Here, we 

consider four alternative solutions. 

Solution 1: Obligatory parentheses 

We can modify Γ exp by enforcing parentheses around complex expressions: 


| (〈expression〉〈operator〉〈expression〉) 

〈operator〉 ::= - | + | * | / 

Benefit: Ambiguity has been resolved. 

Drawback: Expressions such as ‘1 + 2 * 3’, or even ‘1 + 2’, are no longer 

legal. (We must type ‘(1 + (2 * 3))’ and ‘(1 + 2)’ instead.) 

48


Solution 2: Precedence of operators 

We can modify Γ exp by distinguishing operators of high and low priority: 

〈expression〉 ::= 〈term〉 

| 〈expression〉〈lp-operator〉〈expression〉 

〈term〉 ::= 〈integer〉 

| (〈expression〉) 

| 〈term〉〈hp-operator〉〈term〉 

〈hp-operator〉 ::= * | / 

〈lp-operator〉 ::= + | - 

where 〈hp-operator〉 and 〈lp-operator〉 are high-priority and low-priority 

operators, respectively. 

Benefit: Expressions such as ‘1 + 2 * 3’ can be (partially) parsed as 

‘1 + 〈expression〉’ but not as ‘〈expression〉 * 3’. 

49


Solution 2: Precedence of operators 

We can modify Γ exp by distinguishing operators of high and low priority: 


| 〈expression〉〈lp-operator〉〈expression〉 

〈term〉 ::= 〈integer〉 


| 〈term〉〈hp-operator〉〈term〉 



where 〈hp-operator〉 and 〈lp-operator〉 are high-priority and low-priority 

operators, respectively. 

Benefit: Expressions such as ‘1 + 2 * 3’ can be (partially) parsed as 

‘1 + 〈expression〉’ but not as ‘〈expression〉 * 3’. 

Drawback: An expression like ‘1 - 2 - 3’ is still ambiguous: it can be 

(partially) parsed both as ‘〈expression〉 - 3’ and as 

‘1 -〈expression〉’. 

49


Solution 3: Associativity of operators 

We can modify Γ exp by introducing associativity of operators: 

〈expression〉 ::= 〈integer〉 | 〈expression〉〈operator〉〈integer〉 

〈operator〉 ::= * | / | + | - 

Benefit: The operators in this grammar are left-associative; the expression 

‘1 - 2 - 3’ can only be (partially) parsed as ‘〈expression〉 - 3’, 

and not as ‘1 - 〈expression〉’. 

50


Solution 3: Associativity of operators 

We can modify Γ exp by introducing associativity of operators: 

〈expression〉 ::= 〈integer〉 | 〈expression〉〈operator〉〈integer〉 

〈operator〉 ::= * | / | + | - 

Benefit: The operators in this grammar are left-associative; the expression 

‘1 - 2 - 3’ can only be (partially) parsed as ‘〈expression〉 - 3’, 

and not as ‘1 - 〈expression〉’. 

Drawback: All operators have equal precedence; an expression like 

1 - 2 * 3 can only be (partially) parsed as ‘〈expression〉 * 3’, and 

not as ‘1 - 〈expression〉’. 

50


Solution 4: Combine associativity, precedence, and parentheses 

We can modify Γ exp by adding all of the above: 


| 〈expression〉〈hp-operator〉〈term〉 

〈term〉 ::= 〈factor〉 

| 〈term〉〈lp-operator〉〈factor〉 

〈factor〉 ::= 〈integer〉 




51









Ambiguity 



Summary 

52


What is scanning? 

Scanning is the process of translating programs from the string-of-characters 

input format into the sequence-of-tokens intermediate format. 

We have seen scanning in action in the mdc example: 

◮ the lexemizer took as input a string of characters and returned a 

sequence of lexemes; 

◮ the tokenizer took as input a sequence of lexemes and returned a 

sequence of tokens. 

These two steps are usually merged into one pass, called ‘scanning’ (but 

sometimes even ‘lexing’, or ‘tokenization’ is used about both operations, and 

‘scanning’ may be used for only creating the lexemes). 

53

Scanning contd 

How do we design and implement a scanner? 

Building a scanner requires a number of steps: 

1. Specification of the microsyntax (the lexical structure) of the language, 

typically using regular expressions (regexes). 

2. Based on the regexes, a nondeterministic finite automaton (NFA) is built 

that recognizes lexemes of the language. 

3. A deterministic finite automaton (DFA) equivalent to the NFA is built. 

4. The DFA is implemented using a nested control stucture that processes 

the input one character at a time. 

All steps can be realized manually, but there exist tools which 

◮ allow one to specify the lexical structure using regular expressions, and 

◮ build an implementation of the DFA automatically. 

We shall revisit the mdc example and build a scanner both manually and using 

a scanner-building tool. 

54


Before we implement an mdc scanner, we first have a look at a recognizer for 

mdc lexemes. 

◮ A scanner processes an input string and returns a list of lexemes (or 

tokens). 

◮ A recognizer checks whether the whole input string is a single lexeme. 

Example (A recognizer for mdc lexemes) 

Step 1: The microsyntax of mdc is trivially specified with the following regular 

expressions: 

〈command〉 ::= [pf] 

(exactly one p or one f) 

〈operator〉 ::= [\+\-\*\/] 

(analogously, symbols escaped with \) 

〈integer〉 ::= [0..9]+ 

(one or more digits) 

55


Example (A recognizer for mdc lexemes contd) 

Step 3: The regex specification is realized by the following DFA: 13 

p, f 

cmd 

start 

+, -, *, / 

0, . . . , 9 

op 

int 

0, . . . , 9 

13 We skip Step 2; see the further reading section for references if you need more details. 

56


Example (A recognizer for mdc lexemes contd) 

Step 4: An algorithm for the mdc recognizer DFA: 14 

input: string of characters; output: boolean 

state ← start; char ← next() 

while char ≠ EOF: 

if state = start: 

if char ∈ {p,f}: state ← cmd 

else if char ∈ {+,-,*,/}: state ← op 

else if char ∈ {0, . . . ,9}: state ← int 

else: return false 

else if state ∈ {cmd,op}: return false 

else if state = int: 

if char /∈ {0, . . . ,9}: return false 

char ← next() 

if state ∈ {cmd,op,int}: return true 

else: return false 

14 Notation varies. EOF means end of file (input). Each call to next() returns the next character 

from the input. 

57


The recognizer checks whether the whole string is a single lexeme, but we 

want more: 

◮ process strings that include more than one lexeme; 

◮ return a sequence of classified lexemes rather than a yes/no answer. 

58


The recognizer checks whether the whole string is a single lexeme, but we 

want more: 

◮ process strings that include more than one lexeme; 

◮ return a sequence of classified lexemes rather than a yes/no answer. 

In the previous implementation of mdc, all lexemes in a program had to be 

separated by whitespace. This leads to a tradeoff: 

◮ it is more convenient to implement the lexemizer—just split the input by 

whitespace; 

◮ it is less convenient to use the language—the programmer must separate 

all lexemes with whitespace. 

We shall now develop a scanner that makes whitespace between lexemes 

optional (unless we want to separate two numerals). 

58


Try it! The file code/mdc-recognizer.oz contains an implementation of the 

mdc recognizer and a few simple test cases. 

◮ Open the file in the OPI (oz &, then C-x C-f). 

◮ Execute the code (C-. C-b). 

What happens? 

59


Try it! The file code/mdc-recognizer.oz contains an implementation of the 

mdc recognizer and a few simple test cases. 

◮ Open the file in the OPI (oz &, then C-x C-f). 

◮ Execute the code (C-. C-b). 

What happens? 

◮ {MDCRecognizer "p"} evaluates to true, because the input is a 

command. 

◮ {MDCRecognizer "123"} evaluates to true, because the input is 

an integer. 

◮ {MDCRecognizer "1 2 +"} evaluates to false, because the input 

is not a valid lexeme, even though it is a valid sentence (legal 

sequence of valid lexemes) in mdc. 

59


Example (A scanner for mdc) 

Step 4: An algorithm for the mdc scanner DFA: 15 

input: string of characters; output: sequence of tokens 

tokens ← (); state ← start; char ← next(); seen ← ǫ 

while char ≠ EOF: 

if state = start: 

if char ∈ {p,f}: append 〈cmd, char〉 to tokens 

else if char ∈ {+,-,*,/}: append 〈op, char〉 to tokens 

else if char ∈ {0, . . . ,9}: state ← int; seen ← char 

else if char /∈ S: error(char) 

char ← next() 

else if state = int: 

if char ∈ {0, . . . ,9}: concatenate char to seen; char ← next() 

else: append 〈int, seen〉 to tokens; seen ← (); state ← start 

if state = int: append 〈int, seen〉 to tokens 

return tokens 

15 tokens maintains a list of tokens recognized so far. seen maintains a string of characters seen 

since the most recently recognized token. Angle brackets (‘〈’ and ‘〉’) denote tokens (class-lexeme 

pairs). 

60









Ambiguity 



Summary 

61

Summary 

This time 

◮ syntax, grammars, derivations, parse trees, ambiguity 

◮ recognizing, scanning 

◮ design and implementation of an mdc scanner 

Note! The code examples are used as an illustration; we will return to 

(some parts of) them when you learn more about the syntax and 

semantics of Oz. 

Next time 

◮ syntax and semantics of the declarative kernel language 

62

Summary contd 

Homework 

Pensum 

Further reading 

Questions ◮ . . . ? 

◮ . . . ? 

◮ . . . ? 

◮ Examine and try out today’s code, read Mozart/Oz 

documentation if necessary. 

◮ Most of today’s slides, except for implementational 

details of mdc scanners and the recognizer and scanner 

DFA. 

◮ See, e.g., Ch. 3 in Sebesta Concepts of Programming 

Languages; Ch. 2 in Scott Programming Language 

Pragmatics; Ch. 2–4 in Copper and Torczon Engineering 

a Compiler (a detailed, in-depth but readable 

presentation). 

63

Lecture 3: Syntax: Grammars, Derivations, Parse Trees. Scanning.

Transform your PDFs into Flipbooks and boost your revenue!

Delete template?

Save as template ?