Seeing clearly: Frame Semantic, Psycholinguistic, and Cross ...

CHAPTER 4. PSYCHOLINGUISTIC EXPERIMENTS 147
12
10
8
6
4
2
0
Freq.
H
HH H
HH
HH
HH
HH
H
HH H
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Figure 4.1: Experiment 2. Number of Categories used by each subject
Results and Analysis
Task 1: Sorting
Figure 4.1 shows the number of categories created by each subject in the Sorting
task. The median is 6, with 5 and 7 being the next most common numbers of categories;
this is a fair approximation to the 7 categories intended by the experimenters. (One subject
produced 19 piles of cards; after reading the de nitions he wrote and looking at the cards
contained in each pile, we can nd no regular basis for the distinctions made.)
A calculation of the omega statistic between each pair of subjects showed that there
was substantial agreement among subjects even before any instructions as to categorization
were given; the mean = 0:57. There was considerable variation among subjects, but there
was no cluster of subjects who agreed with each other and disagreed with the experimenters'
initial categorization. This suggests that, although there may be agreement among subjects
and disagreement with the experimenters on individual pairs of senses, there is no other
well-de ned \dialect" for the senses of see among our subjects.
As a test of the e ectiveness of our manipulation of the experimental variables,
the omega statistic was used to compare the subjects' initial sortings with the values of
the manipulated variables, including the intended sense. Table 4.3 on the next page shows
the results for a representative group of nine subjects; the agreement for the irrelevant
manipulated factors is essentially zero (because of the correction for chance agreement, the
value of omega can sometimes be less than zero). The agreement with the intended sense
ranges from a low of .36 for subject number 30 to a high of .82 for subject number 33; this
variation in agreement seems to be due to individual di erences. These results suggest that
the subjects were able to follow the instructions to pay attention only to the sense of see
occurring in each sentence and to ignore the other syntactic and semantic factors.
6 Fellbaum et al. (1998) found such e ects in a similar classi cation task.