The SenSem Project: syntactico-semantic annotation of ... - cs@famaf

The SenSem Project: syntactico-semantic annotation of sentences inSpanishAlonso,Laura*Capilla, JoanAntoni†Castellón,Irene*Fernández-Montraveta,Ana**Vázquez,Gloria†*Department of Linguistics, Universitat de Barcelona, Spain{lalonso,icastellon}@ub.edu**Department of English and German Philology, Universtitat Autònoma deBarcelona, Spainana.fernandez@uab.es†Department of English and Linguistics, Universitat de Lleida, Spain{jcapilla,gvazquez}@dal.udl.esAbstractThis paper presents SenSem, a project 1 thataims to systematize the behavior of verbs inSpanish at the lexical, syntactic and semanticlevel. As part of the project, two resources arebeing built: a corpus where sentences areassociated to their syntactico-semanticinterpretation and a lexicon where each verbsense is linked to the corresponding annotatedexamples in the corpus. Some tendencies thatcan be observed in the current state ofdevelopment are also discussed.1 IntroductionThe SenSem project 2 aims to build a databank ofSpanish verbs based on a lexicon that links each verbsense to a significant number of manually analyzedcorpus examples. This databank will reflect thesyntactic and semantic behavior of Spanish verbs innaturally occurring text.We analyze the 250 verbs that occur mostfrequently in Spanish. Annotation is carried out atthree different levels: the verb as a lexical item, theconstituents of the sentence and the sentence as awhole. The annotation process includes verb sensedisambiguation, syntactic structure analysis1 Databank Sentential Semantics: “Creación de una Base deDatos de Semántica Oracional”. MCyT (BFF2003-06456).2 http://grial.uab.es/projectes/sensem.php(syntagmatic categories, including the annotation ofthe phrasal heads, and syntactic functions),interpretation of semantic roles and analysis of variouskinds of sentential semantics. It is precisely this lastarea of investigation which sets our project apart fromothers currently being carried out with Spanish(Subirats and Petruck, 2003 and García De Miguel andComesaña, 2004).Abstracting from the analysis of a significantnumber of examples, the prototypical behavior of verbsenses will be systematized and encoded in a lexicon.The description of verb senses will focus on theirproperties at the syntactico-semantic interface, andwill include information like the list of syntacticosemanticframes in which a verb can possibly occur. Inaddition, selectional restrictions will be automaticallyinferred from the words marked as heads of theconstituents. Finally, the usage of prepositions will bestudied.The conjunction of all this information will providea very fine-grained description of the syntacticosemanticinterface at sentence level, useful forapplications that require an understanding of sentencesbeyond shallow parsing. In the fields of automaticunderstanding, semantic representation and automaticlearning systems, a resource of this type will beespecially valuable.In the rest of the paper we will describe the corpusannotation process in more detail and examples will beprovided. Section 2 offers a general overview of otherprojects similar to SenSem. In section 3, the levels ofannotation are discussed, and the process of annotationis described in section 4. We then proceed to present

the results obtained to date and the current state ofannotation, and we put forward some tentativeconclusions obtained from the results of the annotationthus far.2 Related WorkAs shown by Levin (1993) and others (Jones et al.,1994; Jones, 1995; Kipper et al., 2000; Saint-Dizier,1999; Vázquez et al., 2000), syntax and semantics arehighly interrelated. By describing the way linguisticlayers inter-relate, we can provide better verbdescriptions since generalizations from the lexiconthat previously belonged to the grammar level oflinguistic description can be established (lexicalistapproach).Within the area of Computational Linguistics, it iscommon to deal with both fields independently(Grishman et al., 1994; Corley et al., 2001). In othercases, the relationship established between syntacticand semantic components is not fully exploited andonly basic correlations are established (Dorr et al.,1998; McCarthy, 2000). We believe this approach isinteresting even though it does not take full advantageof the existing link between syntax and semantics.Furthermore, we think that in order to coherentlycharacterize the syntactico-semantic interface, it isnecessary to start by describing linguistic data fromreal language. Thus, a corpus annotated at syntacticand semantic levels plays a crucial role in acquiringthis information appropriately.In recent years, a number of projects related to thesyntactico-semantic annotation of corpora have beencarried out. The length of the present paper does notallow us to consider them all here, but we willmention a few of the most significant ones.FrameNet (Johnson and Fillmore, 2000) is alexicographic resource that describes approximately2.000 items, including verbs, nouns and adjectives thatbelong to diverse semantic domains (communication,cognition, perception, movement, space, time,transaction, etc.). Each lexical entry has examplesextracted from the British National Corpus that havebeen manually annotated. The annotation reflectsargument structure and, in some cases, also adjuncts.PropBank (Kingsbury and Palmer, 2002;Kingsbury et al., 2002) is a project based on themanual semantic annotation of a subset of the PennTreebank II (a corpus which is syntacticallyannotated). This project aims to identify predicateargumentrelations. In contrast with FrameNet, thesentences to be annotated have not been pre-selectedso examples are more varied.Both FrameNet and Propbank work with the use ofcorpora, although their objectives are a bit different. InFrameNet, a corpus is used to find evidence aboutlinguistic behavior and to associate examples tolexical entries, whereas in Propbank, the objective isto enrich a corpus that has been already annotated at asyntactic level so that it can be exploited in moreambitious NLP applications.For Spanish, only a few initiatives address thesyntactico-semantic analysis of corpus. The DataBase“Base de Datos Sintácticos del Español Actual”(Muñiz et al., 2003) provides the syntactic analysis of160.000 sentences extracted from part of theARTHUS corpus of contemporary texts. Syntacticpositions are currently being labeled with semanticroles (García de Miguel and Comesaña, 2004).FrameNet-Spanish (Subirats and Petruck, 2003) isthe application of the FrameNet methodology forSpanish. Its target is to develop semantic frames andlexical entries for this language. Each verb sense isassociated to its possible combinations of participants,grammatical functions and phrase types, as attested inthe corpus.The SenSem project provides a different approachto the description of verb behavior. In contrast withFrameNet, its aim is not to provide examples for a preexistinglexicon, but to shape the lexicon with thecorpus examples annotated. Another difference fromthe FrameNet approach is that the semantic roles weuse are far more general, they are related to syntacticfunctions, and are less class-dependent.Finally, to the best of our knowledge, no largescalecorpus annotation initiative associates semanticsto sentence such as their aspectual interpretations ortypes of causativity.3 Levels of annotationAs mentioned previously, we are describing verbbehavior so only constituents directly related to theverb will be analyzed. Elements beyond the scope ofthe verb (i.e. extra-sentential elements such as logicallinkers, some adverbs, etc.) are disregarded. Thefollowing is an example of scope of annotation:...El presidente, que ayer inició una visitaoficial a la capital francesa, hizo estasdeclaraciones…...The president, who began an official visitto the French capital yesterday, stated…

Were we annotating the verb iniciar –begin– wewould ignore the participants of the main sentence andonly take into account the elements within the clause.If we were annotating the verb hacer –make– wewould annotate the subject to include the entirerelative clause, with the word president” as the head ofthe whole structure. The relative clause will not befurther analyzed.Sentences are annotated at three levels: sentencesemantics, lexical and constituent level.3.1 Sentential semantics levelAt this level, different aspects of sentential semanticsare accounted for. With regard to aspectualinformation, a distinction is made among three typesof meaning, eventive, procedural or stative, as in thefollowing examples:event: ...El diálogo acabará hoy…...The conversations will finish today...process: …cuando le preguntaron de qué habíavivido hasta aquel momento ...…when he was asked what he had been living onuntil then...state: ...El gasto de personal se acerca a los 2.990millones de euros......Personnel expenses come close to 2,990 millioneuros...Apart from aspectual information, we also annotatesentential level meanings using labels likeanticausative, antiagentive, impersonal, reflexive,reciprocal or habitual. This feature is useful toaccount for the variation in the syntactic realizationsof the argument structures of each verb sense. Forexample, the next sentence with the verb “abrir”(open) presents an antiagentive interpretation:agentive: El alcalde de Calafell […] abrirá unexpediente…… The mayor of Calafell […] will openadministrative proceedings …antiagentive: … el vertedero de Tivissa no seabrirá sin consenso.… Tivissa’s rubbish dump will not be openedwithout a consensus.3.2 Lexical levelAt the lexical level, each example of a verb is assigneda sense. We have developed a verb lexicon in whichthe possible senses for a verb are defined, togetherwith its prototypical event structure and thematic grid,and a list of synonyms and antonyms and its relatedsynsets in WordNet (Fellbaum, 1998).Various lexicographic sources have been taken asreferences to build the inventory of senses for eachverb, mainly the Diccionario de la Real Academia dela Lengua Española and the Diccionario Salamancade la Lengua Española. Less frequent meanings arediscarded, together with archaic and restricted uses.This inventory of senses for each verb is onlypreliminary, and can be modified whenever theexamples found in the corpus indicate the existence ofa distinct sense which has not been considered.Different senses are distinguished by differentthematic grids, different event structures, differentselectional restrictions or different subcategorizations.3.3 Constituent levelFinally, at the constituent level, each participant in theclause is tagged with its constituent type (e.g.: nounphrase, completive, prepositional phrase) andsyntactic function (e.g.: subject, direct object,prepositional object).Arguments and adjuncts are also distinguished.Arguments are defined as those participants that arepart of the verb’s lexical semantics. Arguments areassigned a semantic role describing their relation withthe verb (e.g.: agent, theme, initiator…). In SenSem,each sense is associated with a prototypical thematicgrid describing the possible arguments a verb maytake, but, as in the case of senses, this thematic grid isonly preliminary and is modified when corpusexamples provide enough evidence.The head of the phrase is also signaled in order toacquire selectional restrictions for that verb sense.Sometimes, information that has been consideredrelevant in that it may alter some other informationdeclared at a different level has also been included; forexample, negative polarity or negative adverbs arealso indicated.4 Annotation processThe SenSem corpus will describe the 250 mostfrequently occurring verbs in Spanish. Frequency hasbeen calculated in a journalistic corpus. For each ofthese verbs, 100 examples are extracted randomlyfrom 13 million words of corpora obtained from theelectronic version of the Spanish newspapers, LaVanguardia and El Periódico de Catalunya. Thecorpus has been automatically tagged and a shallowparsing analysis has been carried out to detect thepersonal forms of the verbs under consideration. We

do not take into account uses of the verb as anauxiliary. We also disregard any collocations oridioms in which the verb might participate.The manual annotation of examples is carried outvia a graphical interface, seen in Figure 1, where thethree levels are clearly distinguished.Figure 1. Screenshot of the annotation interface.The interface displays one sentence at a time. First,when a verb sense is selected from the list of possiblesenses, its prototypical event structure and semanticroles are displayed for the annotator to take intoaccount. Then, the clause is assigned its aspectualsemantics, and constituents are identified and analyzedby selecting the words that belong to it. The head ofthe arguments and its possible metaphorical usage arealso signaled in order to facilitate a future automaticextraction of selectional restrictions. Finally,annotators specify any applicable semantics at clauselevel (e.g.: anticausative, reflexive, stative, etc.), andstate any particular fact that they consider might be ofuse in future revision and correction processes.The distribution of the corpus among annotatorshas evolved since the earlier stages of the project. Inan initial stage, when the annotation guidelines werenot yet consolidated, each of the 4 annotators wasgiven 24 different sentences of the same verb, plus 4common sentences that were separately annotated byall of them. Later on, these sentences were comparedin order to identify those aspects of the annotation thatwere unclear or prone to subjectivity, as explained inthe following section. In the current stage, theannotation guidelines have been well established.Annotators work with sets of 100 sentencescorresponding to a single verb. All annotations arerevised and any possible errors are corrected.The final corpus will be available to the linguisticcommunity by means of a soon to be created webbasedinterface.5 Preliminary Results of AnnotationAt this stage of the project, 77 verbs have already beenannotated, which implies that the corpus at thismoment is made up of 7,700 sentences (199,190words). A total of 900 sentences out of these 7,700

have already been validated, which means that acorpus of approximately 25,000 words has alreadyundergone the complete annotation process.5.1 Data analysisIn this section we describe the information about verbbehavior that can be extracted from the corpus in itspresent state. We have found that, out of the 199,190words that have already been annotated, 182,303 arepart of phrases which are an argument of the verb and16,887 are adjuncts.With regard to aspectuality, there is a clearpredominance of events (74.26% of the sentences)over processes (20.67%) and states (8.96%). Thisskewed distribution of clause types, with a clearpredominance of events, may be exclusive to thejournalistic genre. We have yet to investigate itsdistribution in other genres.As concerns syntactic functions, seen in Table 1,the most frequent category is direct object, with asignificant difference in subjects. This is not surprisingif we take into account that Spanish is a pro-droplanguage. However, prepositional objects are lessfrequent than subjects, and indirect objects are alsoscarce. Thus, the clausal core appears to bepredominantly populated by the least markedconstituents.Functionratiodirect object 39.83 %subject 22.57 %circumstantial 23.16 %prepositional object 12.65 %indirect object 1.97 %Table 1. Distribution of syntactical functions in theannotated examples.The distribution of semantic roles can be seen inTable 2. Themes are predominant, as would beexpected given that the most common syntacticfunction is that of direct object, and that there is a highpresence of antiagentive, anticausative and passiveconstructions. Within the different types of thesemantic role theme, unaffected themes (movedobjects) appear most frequently.At the constituent level, the semantic role chosenfor each phrase is often predictive of the other labelsof that phrase, following what was expected fromlinguistic introspection: agents tend to be noun phraseswith subject function, themes tend to be noun phraseswith subject or object function (if they occur in apassive, antiagentive, anticausative or stativesentence), etc. +RoleRationot- affected theme 53.47 %affected theme 14.36%agent and cause 14.02%initiator 2.97%Table 2. Distribution of semantic roles in theannotated examples.Thus, the associations made between labels indifferent levels have been used as a first step to semiautomatethe annotation process: once a role isselected, the category and function most frequentlyassociated with it and its role as a verb argument arepre-selected so that the annotator only has to validatethe information.5.2 Inter-annotator agreementIn order to measure inter-annotator agreement, foursentences of 59 verbs have been annotated by 4different judges so that divergences in criteria could befound. These common sentences were used in thepreliminary phase with the aim of both training theannotators and detecting points of disagreementamong them. This comparison has helped us refineand settle the annotation guidelines and facilitate thesubsequent revision of the corpus.In order to detect these problematic issues, wecalculated inter-annotator agreement for all levels ofannotation. An overview of the most representativevalues for annotator agreement can be seen in Table 3.We determined pair wise proportions of overallagreement, that is, the ratio of cases in which twoannotators agreed with respect to all cases.In addition, we also obtained the kappa coefficient(Cohen, 1960), which gives an indication of stabilityand reproducibility of human judgments in corpusannotation. The main advantage of this measure is thatit factors out the possibility that judges agree bychance. Kappa measures range from k=-1 to k=1, withk=0 when there is no agreement other than whatwould be expected by chance, k=1 when agreement isperfect, and k=-1 when there is systematicdisagreement. Following the interpretation proposedby Krippendorf (1980) and Carletta et al. (1996), forcorpus annotation, kappa>0.8 indicate good stabilityand reproducibility of the results, while k

category agreement kappaeventual semanticsevent 66% .11state 90% .33process 76% .06argumentalityargument 82% .54adjunct 64% .46semantic roleinitiator 70% .37agent 84% .81cause 91% .89experiencer 97% .92theme 68% .43affected theme 74% .55non-affected theme 70% .34goal 79% .70syntactic functionagentive complement 100% 1.00subject 87% .83direct object 80% .63indirect object 77% .79prepositional object 1 67% .65prepositional object 2 66% .28prepositional object 3 78% .24Circumstantial 62% .42Predicative 76% .16syntactic categorynoun phrase 78% .67prepositional phrase 72% .53adjectival phrase 88% .69negative adverbial 100% 1.00adverbial phrase 77% .54adverbial clause 68% .66gerund clause 72% .65relative clause 82% .16completive clause 95% .93direct speech 96% .95infinitive clause 94% .98prep. completive 96% .44clauseprep. infinitive clause 81% .57personal pronoun 97% .81relative pronoun 98% .96other pronouns 94% .82Table 3. Inter-annotator agreement for a selection ofannotated categoriesAs a general remark, agreement is comparable towhat is reported in similar projects. For example,Kingsbury et al. (2002) report agreement between60% and 100% for predicate-argument tagging withinPropbank, noting that agreement tends to increase asannotators are more trained. In SenSem, the level ofannotation that is comparable to predicate-argumentrelations, semantic role annotation, is clearly withinthis 60%-100% range.It is noteworthy that the values obtained for thekappa coefficient are rather low. After a closeinspection, we found that these low values of kappaare mainly due to the fact that the annotationguidelines were still not well-established at this pointof annotation, and that annotators were still undertraining. This led us to further describe and exemplifycases detected as having a low agreement value oncethe preliminary exploration of the corpus hadconcluded. As a result, we expect values for kappa toincrease in evaluations that will be carried out in alater stage of the project.Agreement within aspectual interpretations ofsentences is very close to chance agreement. Thestative interpretation seems to be more clearlyperceived than the rest. Events and processes at timesseem to be confused. In order to reduce this source ofdisagreement, each verbal sense was associated to itsprototypical aspectual semantics, as determined by itslexical meaning. For example, the verb aceptar(accept) is associated to the semantics “event” for itssense “to receive (something offered), especially withgladness or approval” and to the semantics “state” forits sense “to be able to endure, hold or admit (in anordered system)”. We expect that this change in theannotation procedure will dramatically increase interannotatoragreement for this feature.Agreement is also low in the categorization ofconstituents as arguments or adjuncts. To improveconsistency in the annotation of arguments, theprototypical subcategorization frame for each verbsense has been provided, making it easier forannotators to identify arguments associated with averb and to label the rest of constituents as adjuncts.For example, in the case of the verb accept, theeventive sense is associated with a subcategorizationframe of the kind [agent,theme], while the second isassociated with [theme,theme]. The criteria todistinguish constituents dominated by a verb(arguments or adjuncts) and those beyond clausalscope have also been clarified.In contrast, the tagging of semantic roles appears torely on linguistic intuition much more than the abovefeatures. There seems to be perfect agreement for veryinfrequent roles (indirect cause, instrument, location).More frequent roles show a higher level ofdisagreement: initiators are significantly less clearly

perceived than agents or causes (note differences in kagreement). It is also clear that fine-graineddistinctions are more difficult to perceive than coarsegrainedones, as exemplified by low agreement withinthe superclass of theme.Among syntactic functions, the agentivecomplement of passives presents perfect agreement.Agreement is also high for subjects and indirectobjects, but the distinction between different kinds ofprepositional objects and circumstantial complementsis not clearly perceived. Therefore, a clearer decisionmakingprocedure was established in the annotationguidelines to distinguish among these. We expect thatthese changes will improve consistency significantly.Finally, agreement is rather high for some syntacticcategories: pronouns, adverbs of negation, adjectivalcomplements, completive clauses, infinitive clausesand direct speech present k > .7 and ratios ofagreement over 90%. However, major categoriespresent a rather high ratio of disagreement, as well asthose categories that are mostly considered adjuncts.This seems to be a direct effect of the variability in theassignment of argumentality, semantic role andsyntactic function features. We expect that a thoroughinspection of the relations between variability in rolesand functions with relation to variability in categorieswill provide a clearer view of this aspect.After the study of inter-annotator agreement, theguidelines for annotation have been settled (Vázquezet al. 2005). These guidelines serve as a reference forannotators, and we believe they will increase theoverall consistency of the resulting corpus. Later on inthe process of annotation, another study of interannotatoragreement will be carried out to determinethe consistency achieved in corpus annotation.6 Conclusions and Future WorkThe linguistic resource we have presented constitutesan important source of linguistic information useful inseveral natural language processing areas as well as inlinguistic research. The fact that the corpus has beenannotated at several levels increases its value and itsversatility.The project is in its second year of development,with still a year and a half to go. During this time weintend to continue with the annotation process and todevelop a lexical database that will reflect theinformation found in the corpus. We are aware that theguidelines established in the annotation process aregoing to bias, to a certain extent, the resultingresources, but nevertheless we believe that both toolsare of interest for the NLP community.All tools developed in the project and the corpusand lexicon themselves will be available to allresearchers who might have interest in exploitingthem.References(Carletta et al. 1996) J. Carletta, A. Isard, S. Isard, J. C. Kowtko,G. Doherty-Sneddon and A. H. Anderson, HCRC DialogueStructure Coding Manual, HCRC Technical report HCRC/TR-82, 1996.(Cohen 1960) J. Cohen, A coefficient of agreement for nominalscales. in Educational & Psychological Measure, 20, 1960, pp.37-46.(Corley et al. 2001) S. Corley, M. Corley, F. Keller, M. W.Crocker, S. Trewin, Finding Syntactic Structure in UnparsedCorpora in Computer and the Humanities, 35, 2001, pp. 81-94.(Dorr et al. 1998) B. Dorr, M. A. Martí, I. Castellón, SpanishEuroWordNet and LCS- Based Interlingual MT. Proceeding ofthe ELRA Congress. Granada, 1998.(Fellbaum 1998) C. Fellbaum, A Semantic Network of EnglishVerbs, in Christiane Fellbaum (ed.). WordNet: An ElectronicLexical Database, MIT Press, 1998.(Garcia de Miguel & Comesaña 2004) J.M. Garcia de Miguel andS. Comesaña, Verbs of Cognition in Spanish: ConstructionalSchemas and Reference Points, in A. Silva, A. Torres, M.Gonçalves (eds) Linguagem, Cultura e Cogniçao: Estudos deLinguística Cognitiva, Almedina, 2004, pp. 399-420.(Grishman et al. 1994) R. Grishman, C. Macleod and A. Meyers.1994. Comlex Syntax: Building a computational lexicon.Proceedings of COLING.(Johnson &Fillmore 2000) C. Johnson and C. J. Fillmore, TheFrameNet tagset for frame-semantic and syntactic coding ofpredicate-argument structure. Proceedings NAACL 2000,Seattle WA, USA, 2000, pp. 56-62.(Jones 1994) D. Jones (ed), Verb classes and alternations inBangla, German, English and Korean. Memo nº 1517. MIT,Artificial Intelligence Laboratory, 1994.(Jones 1995) D. Jones, Predicting Semantics from Syntactic Cues --- Evaluating Levin's English Verb Classes and Alternations.UMIACS TR-95-121, University of Maryland, 1995.(Kingsbury & Palmer. 2002) P. Kingsbury and M. Palmer, FromTreebank to Propbank. Third International Conference onLanguage Resources and Evaluation, LREC-02, Las Palmas,Spain, 2002.(Kingsbury et al. 2002) P. Kingsbury, M. Palmer and M. Marcus.,Adding Semantic Annotation to the Penn TreeBank.Proceedings of the Human Language Technology Conference.San Diego, California, 2002.(Kipper ET AL. 2000) K. Kipper, H. T. Dang and M. Palmer,Class-Based Construction of a Verb Lexicon. AAAI-2000Seventeenth National Conference on Artificial Intelligence,Austin, TX, USA, 2000.

(Krippendorf 1980) K. Krippendorf, Content analysis: anintroduction, Sage, 1980.(Levin 1993) B. Levin, English Verb Classes and Alternations: APreliminary Investigation. The University of Chicago Press,1993.(McCarthy 2000) D. McCarthy, Using semantic preferences toidentify verbal participation in role switching alternations.Proceedings of NAACL 2000, Seattle, WA, USA, 2000.(Muñiz ET AL. 2003) E. Muñiz, M. Rebolledo, G. Rojo, M.P.Santalla and S. Sotelo, Description and Exploitation of BDS:a Syntactic Database about Verb Government in Spanish, inGalia Angelova, Kalina Bontcheva, Ruslan Mitkov, NicolasNicolov, Nikolai Nikolov (eds.). Proceedings of RANLP 2003.Borovets, Bulgaria, 2003, pp. 297-303.(Saint-Dizier 1999) P. Saint-Dizier, Alternations and verbsemantic classes for French: analysis and class formation .Saint-Dizier, P. (ed.). Predicative Forms in Natural Languagesand Lexical Knowledge Bases. Holanda. Kluwer: 139-170.(Subirats-Rüggeberg & Petruck 2003) Subirats-Rüggeberg, C. yM. R. L. Petruck, Surprise: Spanish FrameNet! Presentation atWorkshop on Frame Semantics, Proceedings of theInternational Congress of Linguists, Praga., 2003.(Vázquez et al. 2000) G. Vázquez, A. Fernández and M. A. Martí,Clasificación verbal. Alternancias de diátesis, Universitat deLleida, 2000.(Vázquez et al. 2005) G. Vázquez, A. Fernández and L. Alonso,Guidelines for the syntactico-semantic annotation of a corpusin Spanish, RANLP, Bulgaria, 2005.