Semantic Role Labeling

Acknowledgement Motivation Background Resources Machine Learning for SRL Evaluation Summary 

Semantic Role Labeling 

Wilker Aziz 

w.aziz@wlv.ac.uk 

January 31, 2012 

Wilker Aziz w.aziz@wlv.ac.uk 

Semantic Role Labeling


Table of Contents 

1 Motivation 

2 Background 

3 Resources 

4 Machine Learning for SRL 

5 Evaluation 

6 Summary 


Semantic Role Labeling


Acknowledgement 

Much of these slides are inspired on (Palmer et al., 2010) 


Semantic Role Labeling 1 / 34



Representing the meaning of text 






Identifying events and participants 







Highlighting 







Highlighting 

Who did What to Whom 







Highlighting 


How 







Highlighting 


How 

When 







Highlighting 


How 

When 

Where 







Highlighting 


How 

When 

Where 

Why 




Task 

Input 

John broke the window 




Task 

Syntactic Parse 

[John ]subject [broke ]verb [the window ]object 




Task 



Semantic Parse 

[John ]breaker [broke ]event [the window ]broken thing 




Task 





Input 

The window broke 




Task 






[The window ]subject 

[broke ]verb 




Task 







[broke ]verb 


[The window ]broken thing 

[broke ]event 




Task 







[broke ]verb 


[The window ]broken thing 

[broke ]event 

“Disassociate” the semantic role from the syntactic realization 




Linguistic Background 

Cases as semantically typed verb arguments (Fillmore, 1968) 






Case relations 







Agentive 







Agentive 

Marks the agent of an action, e.g singer 







Agentive 


morphologically marked at the surface level 







Agentive 

Dative 









Agentive 

Dative 



Marks the recipient of an action, e.g to him 







Agentive 

Dative 




syntactically marked at the surface-structure level 







Agentive 

Dative 




syntactically marked at the surface-structure level 

Syntactic relations are cues to underlying case assignments 




Semantic Types 

Debate 

1 What are the semantic roles 





Debate 


2 How do we group them in categories 





Debate 



Associating different types of nouns with different types of cases 





Debate 









Debate 





Mary is cooking the potatoes 





Debate 






Animate things are capable of performing actions 





Debate 







Agentive arguments are likely to be of type ANIMATE 





Debate 







Agentive arguments are likely to be of type ANIMATE 

Later known as Selectional Preferences 




Semantic Generalization 

The semantics of the verb determine the number and type of roles 

associated with it 





The semantics of the verb determine the number and type of roles 

associated with it 

blush Dative 1 She 1 blushed 

give Agentive 1 , Objective 2 and Dative 3 I 1 gave you 3 food 2 

open Agentive opt , Objective 1 and The door 1 opened 

Instrument opt 





Some NP 1 s have the same case assignments 

Conjunction test 

John and Mary broke the window 

1 Noun Phrase 








*John and a hammer broke the window 

1 Noun Phrase 









Both John and Marie are breakers 

1 Noun Phrase 










While John is the breaker 

1 Noun Phrase 










While John is the breaker 

the hammer is the instrument used to break 

1 Noun Phrase 





Some verbs are semantically equivalent 






[She ]subject 

likes [ reading ]object 






[She ]subject 


[Reading ]subject 

pleases [ her ]object 






[She ]subject 




[ dative 

She ] likes [objective reading ] 






[She ]subject 




[ dative 


[ objective 

Reading ] pleases [dative her ] 






[She ]subject 




Similarly to break and break 

[ dative 


[ objective 

Reading ] pleases [dative her ] 




Systematizing 

Why is generalizing semantics desirable 




Systematizing 


1 Explicit commonalities helps NLP 




Systematizing 



2 Fewer types are needed for describing the lexicon 




Systematizing 




Formalisms 




Systematizing 




Formalisms 

1 Deep cases (failed to form a consensus) 




Systematizing 




Formalisms 

1 Deep cases (failed to form a consensus) 

2 Semantic/Thematic/Theta- roles (generally agreed upon) 




Theta-Grids 

Syntactically a verb has a subcategorization frame 

2 verb 




Theta-Grids 


Syntax: Subcategorization frame 

[She ]NP/subject [likes ]VP/verb [reading ]NP/object 

[Reading ]NP/subject [pleases ]VP/verb [her ]NP/object 

2 verb 




Theta-Grids 





Theta-grid is the set of semantic roles associated with the 

argument positions of a target predicate 2 

2 verb 




Theta-Grids 





Theta-grid is the set of semantic roles associated with the 

argument positions of a target predicate 2 

Semantics:Theta-grid 

[ Experiencer 

She ] [Target likes ] [Theme reading ] 

[ Theme 

Reading ] [Target pleases ] [Experiencer her ] 

2 verb 



Semantic Roles 

Agent 

Patient 

Theme 

Experiencer 

Beneficiary 

Instrument 

Initiator of action, capable 

of volition 

Affected by action, undergoes 

change of state 

Entity moving, or being 

located 

Perceives action but not 

in control 

For whose benefit action 

is performed 

Intermediary/means 

used to perform an 

action 

The batter smashed the pitch into left field. The 

pilot landed the plane as lightly as a feather. 

David trimmed his beard. John broke the window. 

Paola threw the Frisbee. The picture hangs 

above the fireplace. 

He tasted the delicate flavour of the baby lettuce. 

Chris noticed the cat slip through the partially 

open door. 

He sliced me a large chunk of prime rib, and I 

could hardly wait to sit down to start in on it. 

The Smiths rented an apartment for their son. 

He shot the wounded buffalo with a rifle. The 

surgeon performed the incision with a scalpel. 

Location Place of object or action There are some real monsters hiding in the anxiety 

closet. The band played on the stage. 

Source Starting point The jet took off from Nairobi. We heard the 

rumour from a friend. 

Goal Ending point The ball rolled to the other end of the hall. 

Laura lectured to the class. 

(Saeed, 2003)


Agreement 

We still observe substantial disagreement on the thematic roles 




Agreement 


Jackendoff (1972) brings clarifications 




Agreement 



1 The Agent is the initiator of the action, typically acts 

deliberately 




Agreement 




deliberately 

2 The Patient is being acted upon, it is likely to change state as 

a result of the Agent’s actions 




Agreement 




deliberately 

2 The Patient is being acted upon, it is likely to change state as 

a result of the Agent’s actions 

3 Patients undergo a change of state whereas Themes simply 

change location 




Assimilating 

Annotate with thematic roles 

1 The ball flew into the outfield 




Assimilating 


1 

[ Theme 

The ball ] flew [Goal into the outfield ] 




Assimilating 


1 

[ Theme 


2 Jim gave the book to the professor 




Assimilating 


1 

2 

[ Theme 


[ Agent 

Jim ] gave [Patient the book ] [Goal to the professor ] 




Assimilating 


1 

2 

[ Theme 


[ Agent 


3 Laura talked to the class about history 




Assimilating 


1 

2 

3 

[ Theme 


[ Agent 


[ Agent 

Laura ] talked [Goal to the class ] [Theme about history ] 




Assimilating 


1 

2 

3 

[ Theme 


[ Agent 


[ Agent 


4 Laura scolded the class 




Assimilating 


1 

2 

3 

4 

[ Theme 


[ Agent 


[ Agent 


[ Agent 

Laura ] scolded [Patient the class ] 




Assimilating 


1 

2 

3 

4 

[ Theme 


[ Agent 


[ Agent 


[ Agent 


5 Bill cut his hair with a razor 




Assimilating 


1 

2 

3 

4 

5 

[ Theme 


[ Agent 


[ Agent 


[ Agent 


[ Agent 

Bill ] cut [Patient his hair ] [Instrument with a razor ] 




Assimilating 


1 

2 

3 

4 

5 

[ Theme 


[ Agent 


[ Agent 


[ Agent 


[ Agent 


6 Gina crashed the car with a resounding boom 




Assimilating 


1 

2 

3 

4 

5 

6 

[ Theme 


[ Agent 


[ Agent 


[ Agent 


[ Agent 


[ Agent 

Gina ] crashed [Patient the car ] with a resounding boom 




Levin’s Verb Classes and Alternations 

Levin (1993) argues that syntactic variations are a direct reflection 

of underlying semantics 

Class Diathesis Alternations Verbs 

break 45.1 John broke the jar The jar broke Jars break easily break, chip, 

crack, crash, 

. . . 

cut 21.1 John cut the bread *The bread cut Bread cuts easily chip, chop, 

clip, cut, . . . 

hit 18.1 John hit the wall *The wall hit *Walls hit easily bang, bash, 

click, dash,. . . 









crack, crash, 

. . . 





The set of syntactic frames associated with a verb reflect underlying 

semantic components that constrain allowable arguments 









crack, crash, 

. . . 





Verbs are grouped into classes based on their ability to occur or not 

occur in pairs of syntactic frames that are meaning preserving 




Frame Semantics 

Fillmore (1982) 






Frame is a description of a type of event 







Frame Elements are the concepts involved in the event 








Lexical Units are the words that evoke the frame 









Example 









Example 

Frame Apply-heat 






Example 





Frame Elements Cook Agent , Food Theme , Heating 

Instrument Instrument 






Example 





Frame Elements Cook Agent , Food Theme , Heating 

Instrument Instrument 

Lexical Units bake, barbecue, blanch, braise, broil, brown, . . . 





Example 





Example 

Apply-heat 

[ Cook 

The boys ] grill [Food their catches ] [Heating instrument 

on an open fire ] 





Example 

Apply-heat 

[ Cook 



Frame elements are classified in terms of how central they are 





Example 

Apply-heat 

[ Cook 




1 core: conceptually necessary for the frame 





Example 

Apply-heat 

[ Cook 





2 peripheral: not central, but providing additional information 





Example 

Apply-heat 

[ Cook 






3 extra-thematic: not specific to the frame, but situating the 

frame with respect to a broader context 





Example 

Apply-heat 

[ Cook 








Lexical items (senses of a lemma) are grouped together 





Example 

Apply-heat 

[ Cook 









based solely on having the same frame semantics 





Example 

Apply-heat 

[ Cook 









based solely on having the same frame semantics 

regardless of syntactic similarities 




Available Resources 

Lexical Resources 






FrameNet (Baker et al., 1998) 







VerbNet (Kipper et al., 2008) 








Annotation 








Annotation 

Propositional Bank (Palmer et al., 2005) 




FrameNet 

Based on Filmore’s Frame Semantics 




FrameNet 


1126 Frames 




FrameNet 


1126 Frames 

9,686 Frame Elements 




FrameNet 


1126 Frames 

9,686 Frame Elements 

12,317 Lexical Units 




VerbNet 

Extended form Levin’s verb classes 




VerbNet 


440 classes 




VerbNet 


440 classes 

3769 verb lemmas 




VerbNet 


440 classes 

3769 verb lemmas 

5257 verb senses 




PropBank 

Meant to be used as training data for supervised machine learning 




PropBank 


Generic semantic roles consistently annotated across syntactic 

variations 




PropBank 



variations 

Semantic roles defined on a verb by verb basis 




PropBank 



variations 


Semantic arguments are numbered as they exhibit features of 




PropBank 



variations 



A0 a prototypical Agent 




PropBank 



variations 




A1 a prototypical Patient or Theme 




PropBank 



variations 





Additional argument modifiers 




PropBank 



variations 






AM-LOC location 




PropBank 



variations 







AM-ADV adverbial 




PropBank 



variations 








AM-TMP temporal 




PropBank 



variations 








AM-TMP temporal 

Arguments were made to be consistent with VerbNet whenever 

possible 




PropBank 

Annotation 




PropBank 

Annotation 

Frameset a roleset and its syntactic realizations 




PropBank 

Annotation 


Roleset a distinct use of a verb 




PropBank 

Annotation 



Example 




PropBank 

Annotation 



Example 

Roleset bake 01 - VerbNet Class: 1 “create via heat” 




PropBank 

Annotation 



Example 


Roles A0:baker, A1:creation, A2:source, A3:benefactive 




PropBank 

Annotation 



Example 


Roles A0:baker, A1:creation, A2:source, A3:benefactive 

[ AM-TMP 

Today ] [A2 whole grains ] are freshly ground every day and backed [A1 into 

bread ] 




PropBank 

Wrapping up 




PropBank 

Wrapping up 

+ Theory-agnostic labels 




PropBank 

Wrapping up 


+ Labels have verb-specific meanings 




PropBank 

Wrapping up 



+ Avoid making claims about how any one verb’s arguments 

relate to other verb’s arguments 




PropBank 

Wrapping up 





+ In practice about 85% of the arguments are 

Proto-Agent/Proto-Patient 




PropBank 

Wrapping up 







- Less generalization power for inferences 




PropBank 

Wrapping up 








- Less generalization power across verb classes 




PropBank 

Wrapping up 









- Arguments A2-A5 are highly overloaded, therefore 

performance drops on them 




PropBank 

Wrapping up 









- Arguments A2-A5 are highly overloaded, therefore 

performance drops on them 

- Genre specific corpus: Wall Street Journal 




Supervised Machine Learning for SRL 

Task of identifying and classifying 






verb/noun predicate arguments 







Relying on 







Relying on 

PropBank annotation and other lexical-semantics resources 







Relying on 

PropBank annotation and other lexical-semantics resources 

WordNet 

FrameNet 

VerbNet 




Overview 

Generalize from training examples 

Apply knowledge to unseen examples 




Overview 

1 Observe example 

[ A1 

W. Ed Tyler ] was elected [A2 president of his technology 

group ] . 




Overview 

2 Extract features 

ID Form Lemma POS Head Dependency Target 

1 W. w. NNP 3 NAME - 

2 Ed ed NNP 3 NAME - 

3 Tyler tyler NNP 4 SBJ - 

4 was w be VBD 0 ROOT - 

5 elected elect VBN 4 VC elect.01 

6 president president NN 5 OPRD - 

7 of of IN 6 NMOD - 

8 his his PRP$ 10 NMOD - 

9 technology technology NN 10 NMOD - 

10 group group NN 7 PMOD - 

11 . . PUNC 7 P - 




Overview 

3 Gold-standard annotation 

ID Form Lemma POS Head Dep Target Args 

1 W. w. NNP 3 NAME - - 

2 Ed ed NNP 3 NAME - - 

3 Tyler tyler NNP 4 SBJ - A1 

4 was w be VBD 0 ROOT - - 

5 elected elect VBN 4 VC elect.01 - 

6 president president NN 5 OPRD - A2 

7 of of IN 6 NMOD - - 

8 his his PRP$ 10 NMOD - - 

9 technology technology NN 10 NMOD - - 

10 group group NN 7 PMOD - - 

11 . . PUNC 7 P - - 




Overview 

4 Generalize from example 




Overview 

5 Reuse knowledge 

The people elected Barack Obama president of The United States 

this monday . 




Overview 

5 Reuse knowledge 

[ A0 

The people ] elected [A1 Barack Obama ] [A2 president of The 

United States ] [AM-TMP this monday ] . 




Features 

Features 




Features 

Features 

Shallow syntactic analysis (Hacioglu et al., 2004) 




Features 

Features 


Deep syntactic analysis (Pradhan et al., 2008) 




Features 

Features 


Deep syntactic analysis (Pradhan et al., 2008) 

Lexical features (Zapirain et al., 2010) and (Aziz et al., 2011) 




Shallow Syntactic Features 

Shallow syntax is a reliable and easy-to-obtain resource 






POS tags 






POS tags 

Phrase type 






POS tags 

Phrase type 

Prepositional attachment 






POS tags 

Phrase type 

Prepositional attachment 

Shallow Paths 




Deep Syntactic Features 

Deep syntax is a very informative resource 






Syntactic paths 







Syntactic relations 








Voice 








Voice 

Headword 








Voice 

Headword 

Subcategorization 








Voice 

Headword 


Constituency parse 








Voice 

Headword 


Constituency parse 

Dependency parse 




Lexical Features 

Usually sparser, although accounting for semantics 






Surface 

Semantic resources 

Wordform 

WordNet senses 

Lemma 

VerNet classes 

NE categories 

FrameNet rolesets 






Surface 


Wordform 


Lemma 


NE categories 


Corpus-based 

Word classes: automatic clustering 






Surface 


Wordform 

Lemma 

NE categories 

Corpus-based 





Word similarity: strength of association between the predicate 

and its arguments 






Surface 


Wordform 

Lemma 

NE categories 

Corpus-based 





Word similarity: strength of association between the predicate 

and its arguments 

Additional evidence: automatically extracted similar words 

and selectional preferences 




Frameworks 

Machine Learning Framework 




Frameworks 


SVM 




Frameworks 


SVM 

Argument Identification 




Frameworks 


SVM 


Argument Classification 




Frameworks 


SVM 



CRF 




Frameworks 


SVM 



CRF 

Sequence labeling 




Frameworks 


SVM 



CRF 

Sequence labeling 

Jointly identifies and classifies arguments 




SVM 

Given a particular target predicate 




SVM 


1 Identify candidate arguments 




SVM 



2 Classify them accordingly 




SVM 




Imbalance 




SVM 




Imbalance 

positive samples constituents that are arguments (minority) 




SVM 




Imbalance 


negative samples constituents that are not arguments (majority) 




SVM 




Imbalance 



Heuristic pruning prior to identification 




SVM 




Imbalance 




certain syntactic paths are too unlikely to be considered 




SVM 




Imbalance 





prune out the majority of negative samples 




SVM 




Imbalance 





prune out the majority of negative samples 

widely used syntax-driven algorithm (Xue and Palmer, 2004) 




Unsupervised Machine Learning for SRL 

An unsupervised approach for Frame Semantics 




Unsupervised Machine Learning for SRL 

An unsupervised approach for Frame Semantics 

LU 

like 

like 


Subject Object 

I you 

mother child 

people artists 

. . . . . . 

Person Person 

I meat 

kids candies 

baby milk 

. . . . . . 

Person Food 




Clustering 

Typically a clustering task 




Clustering 


Lexical Unit is a target predicate (e.g verbs) 




Clustering 



Subcategorization Frame is a syntactic frame commonly 

observed with the LU 




Clustering 





Syntactic Realization is a possible filler of a syntactic relation in 

the subcategorization frame 




Clustering 







Semantic Role is a cluster of syntactic realizations 




Clustering 







Semantic Role is a cluster of syntactic realizations 

Semantic Frame is the tuple of semantic roles 




Evaluation Campaigns 

CoNLL 2005-2009 (Carreras and Màrquez, 2005) 






In-domain: Wall Street Journal 







Out-of-domain: Brown Corpus 








Relaxed Boundaries 









Reward systems when they correctly label the headword even 

though they miss the boundaries 











F-measure 











F-measure 

precision percentage of labels output by the system which are 

correct 











F-measure 

precision percentage of labels output by the system which are 

correct 

recall percentage of the true labels correctly assigned by 

the system 




Wrap up 

1 Semantics is hard to systematize 




Wrap up 


2 Different formalisms bring different benefits at different costs 




Wrap up 



3 Agreement is generally hard to achieve 




Wrap up 




4 Pragmatic approaches towards supervised machine learning 

has guided recent research 




Wrap up 






5 Propbank-like semantic parsing is now achievable 




Wrap up 







6 Performance lies within 50-85% depending on the nature of 

the task 




Wrap up 








the task 

7 State-of-the-art SRL systems are heavily dependent on syntax 




Wrap up 








the task 


8 Robust, reliable parsing is not a reality for many languages 




Wrap up 








the task 



9 Recent efforts towards alternative resources: shallow syntax 

and lexical features 




Wrap up 








the task 



9 Recent efforts towards alternative resources: shallow syntax 

and lexical features 

10 Unsupervised SRL mostly concerns clustering 

subcategorization realizations depending on their fillers’ 

lexical-semantics 



References I 

Wilker Aziz, Miguel Rios, and Lucia Specia. 

2011. 

Improving Chunk-based Semantic Role Labeling with Lexical 

Features. 

In Recent Advances in Natural Language Processing 

Conference, RANLP ’11, Hissar, Bulgaria, September. RANLP. 

Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 

1998. 

The Berkeley FrameNet Project. 

In Proceedings of the 36th Annual Meeting of the Association 

for Computational Linguistics and 17th International 

Conference on Computational Linguistics - Volume 1, ACL ’98, 

pages 86–90, Stroudsburg, PA, USA. 



References II 

Xavier Carreras and Lluís Màrquez. 

2005. 

Introduction to the conll-2005 shared task: semantic role 

labeling. 

In Proceedings of the Ninth Conference on Computational 

Natural Language Learning, CONLL ’05, pages 152–164, 

Stroudsburg, PA, USA. 

Charles J. Fillmore. 

1968. 

The Case for Case. 

Universals in Linguistic Theory, pages 1–88. 



References III 

Charles J. Fillmore. 

1982. 

Frame Semantics. 

Linguistics in the Morning Calm, pages 96–138. 

Kadri Hacioglu, Sameer Pradhan, Wayne Ward, James H. 

Martin, and Daniel Jurafsky. 

2004. 

Semantic role labeling by tagging syntactic chunks. 

In Eighth Conference on Natural Language Learning, CONLL 

’04. 



References IV 

Ray Jackendoff. 

1972. 

Semantic Interpretation in Generative Grammar. 

MIT Press, Cambridge, Massachussets. 

Karin Kipper, Anna Korhonen, Neville Ryant, and Martha 

Palmer. 

2008. 

A large-scale classification of English verbs. 

Language Resources and Evaluation, 42:21–40, March. 



References V 

Beth Levin. 

1993. 

English Verb Classes and Alternations: A Preliminary 

Investigation. 

University of Chicago Press, Chicago. 

Martha Palmer, Daniel Gildea, and Paul Kingsbury. 

2005. 

The Proposition Bank: An Annotated Corpus of Semantic 

Roles. 

Computational Linguistics, pages 71–106, March. 



References VI 

Martha Palmer, Daniel Gildea, and Nianwen Xue. 

2010. 

Semantic Role Labeling. 

Mogan & Claypoole. 

Sameer S. Pradhan, Wayne Ward, and James H. Martin. 

2008. 

Towards robust semantic role labeling. 

Computational Linguistics, pages 289–310, June. 

John Saeed. 

2003. 

Semantics. 

Blackwell Publishing, Malden, Massachusetts. 



References VII 

Nianwen Xue and Martha Palmer. 

2004. 

Calibrating feature for semantic role labeling. 

In Proceedings of the 2004 Conference on Empirical Methods 

in Natural Language Processing, EMNLP ’04. 

Benat Zapirain, Eneko Agirre, Lluís Màrquez, and Mihai 

Surdeanu. 

2010. 

Improving semantic role classification with selectional 

preferences. 

In Human Language Technologies: The 2010 Annual 

Conference of the North American Chapter of the Association 



References VIII 

for Computational Linguistics, HLT ’10, pages 373–376, 

Stroudsburg, PA, USA.

Semantic Role Labeling

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?