22.03.2013 Views

Seeing clearly: Frame Semantic, Psycholinguistic, and Cross ...

Seeing clearly: Frame Semantic, Psycholinguistic, and Cross ...

Seeing clearly: Frame Semantic, Psycholinguistic, and Cross ...

SHOW MORE
SHOW LESS

You also want an ePaper? Increase the reach of your titles

YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.

CHAPTER 4. PSYCHOLINGUISTIC EXPERIMENTS 141<br />

Task 1: Sorting<br />

In this task, subjects were given cards with one sentence containing see <strong>and</strong> asked<br />

to read each sentence one at a time <strong>and</strong> place it in a pile with sentences containing the<br />

same sense of see. They were instructed to ignore grammatical factors such as tense <strong>and</strong><br />

voice, <strong>and</strong> to use as many piles as they wanted. Subjects were also asked to write a brief<br />

de nition or characterization of each group.<br />

Each subject sorted 100 corpus examples, followed by the 44 constructed examples,<br />

followed (if time permitted) by the remaining 100 corpus examples. To control for the<br />

in uence of the particular examples in the rst part of the sorting, the rst 100 sentences<br />

given to each subject were either all from Set 1, all from Set 2, or half from each set.<br />

Task 2: Classi cation<br />

Each subject sat at a computer on which the materials were displayed using HTML<br />

<strong>and</strong> a web browser. First the subjects read the directions <strong>and</strong> saw the list of de nitions <strong>and</strong><br />

examples. Then the sentences were presented one at a time at the top of the screen, with<br />

the list of senses displayed below. Subjects chose the sense which best matched the use in<br />

the sentence by clicking next to it <strong>and</strong> then clicked again to go on to the next sentence.<br />

Both the response <strong>and</strong> the response latency were recorded, but the task was not paced, <strong>and</strong><br />

subjects could refer back to the de nitions <strong>and</strong> examples whenever they wanted, either on<br />

screen or on a printed h<strong>and</strong>out. All subjects completed the same 99 sentences (1 block of<br />

trials) within one hour.<br />

Statistical Measures of Agreement<br />

Three di erent measures of agreement were used in analyzing the results of the<br />

experiments reported here, simple agreement, kappa, <strong>and</strong> omega.<br />

Simple proportion of agreement with a \gold st<strong>and</strong>ard" is obviously the easiest to<br />

interpret, but it has serious limitations. It does not re ect agreements among raters that<br />

do not match the st<strong>and</strong>ard.<br />

For example, suppose in an experiment on perception of dialects, recordings of<br />

three speakers, A, B, <strong>and</strong> C, are played to a total of 12 subjects (called \raters"), who must<br />

decide which speakers are from New York, which from Chicago <strong>and</strong> which from Philadelphia.<br />

Several possible outcomes of such an experiment are shown in the four parts of Table 4.1.

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!