Seeing clearly: Frame Semantic, Psycholinguistic, and Cross ...

CHAPTER 4. PSYCHOLINGUISTIC EXPERIMENTS 143
logic of the statistic, see the excellent introduction in Siegel & Castellan (1988:284-91).)
The omega statistic (Morey & Agresti 1984) is based on whether or not two raters
classify each pairofstimuli in the same category or not, without regard to the classi cation
of other pairs. Like kappa, it is corrected for chance agreement, so that it varies from 0
for chance agreement to 1.0 for perfect agreement. Omega is inherently less powerful than
kappa, since it considers each pair of stimuli in isolation, however, it has the great advantage
that it can be used in cases in which the number of categories di ers from rater to rater.
We will therefore use omega to measure agreement among subjects on the Sorting task,
where di erent subjects created di erent numbers of categories.
Both kappa and omega are insensitive to the number of categories involved, or the
type of distribution of instances into categories. The variance of the sampling distribution
is known for both, so that the probability of a particular outcome can be calculated.
Results and Analysis
Task 1: Sorting
The number of categories per subject ranged from 6 to 21, with a mean of 11.
The proportion of examples in the categories varied greatly, from 33% for eye and 15% for
recognize to 0 for some categories. The omega coe cient of agreement ranged from 0.09
to 0.49, with a median of 0.245.
Task 2: Classi cation
All subjects nished 99 sentences of the rst set. Some subjects continued on to
other sets, but the order of the sets was randomized across subjects, so that there was little
overlap beyond the rst set. The overall agreement among raters, measured by the kappa
statistic, was .38 5 . This value is low, but understandable; not only were a large number of
senses listed, but also many of the sentences were ambiguous when presented out of context.
Discussion
In evaluating the results of Experiment 1, the strengths and weaknesses of using
corpus examples became apparent. On the one hand we had learned something about the
5 This and all of the values of kappa reported in this study are statistically signi cant at p