Presuppositions in Spoken Discourse

Between Binding and Accommodation
to develop a classification scheme which is taught to two or more annotators, who
are then to classify data independently into the different categories. The resulting
annotations are compared, the assumption being that a high degree of interannotator
agreement on a classification task guarantees the reliability of our
categories. If, on the other hand, there is a low-degree of inter-annotator agreement
for one or more of the categories, or a consistent confusion between one or more
categories, we can question the validity of the category, or question the correctness
of the definition or delimitation of the category. Improving category definitions, or
changing the categories looked for can often improve inter-annotator agreement.
Inter-annotator agreement can be statistically measured by calculating the Kappavalue.
7
Poesio & Vieira (1998) did two different large-scale annotation experiments
of definite NPs in the Wall Street Journal in an attempt to find a classification that
is reliable and where the results could be used as a key for the evaluation of a
program for definite description classification and resolution, implemented in
Vieira & Poesio (2000).
Each of the two experiments used different definitions of bridging. The first
experiment (2 annotators, 1,040 NPs) used a classification with five categories that
referred explicitly to surface characteristics of the definite NPs. In this experiment,
they included co-reference relationships that demanded some type of inference as
bridging, but treated co-reference relationships between noun phrases with the
same head noun as anaphoric. They obtained a Kappa score of 0.68 for the first
experiment. 204 NPs were identified as “associative”, i.e. bridging anaphors (cf.
Poesio et al. 1997). With the definition used in this experiment, bridging examples
made up about 20% of all the NPs studied. This gives an idea of how frequent
potential bridging examples actually are in written text. A more restrictive definition
would probably result in a smaller percentage of the examples being considered
bridging.
The second experiment (3 annotators, 464 definite NPs) used a different
annotation scheme that asked subjects to classify definite noun phrases based on a
semantic definition of the relationship to anchors and antecedents rather than the
surface form. Co-reference relationships were not considered bridging. In this
experiment, linguistically naive annotators were used and this could, along with the
different annotation categories, account for the somewhat poorer agreement score,
which was K = 0.58. It is interesting to note that 164 out of 464 NPs in this
experiment were classified as co-referential by all three coders (for which there was
95% agreement) but only 7 were classified as bridging by all three, though the
number of bridging descriptions identified by each annotator were 40, 29 and 49,
7 The Kappa statistic takes the agreement as well as the number of categories in a classification
task in to the equations. A Kappa value between 0.6 - 0.8 is supposed to signify some degree of
agreement and over 0.8. should allow conclusions to be drawn. See also Carletta et al. (1997) and
Poesio & Vieira (1998) for an explanation of how the Kappa value can be calculated for an
annotation task.
163