Seeing clearly: Frame Semantic, Psycholinguistic, and Cross ...

More documents

Recommendations

Info

CHAPTER 4. PSYCHOLINGUISTIC EXPERIMENTS 142 The rows represent the speakers, and the columns represent the frequencies of the subjects' judgements (called \ratings"). Suppose further that Speaker A is really from New York, B from Chicago, and C from Philadelphia; then the numbers on the diagonal represent the number of agreements between the ratings and the actual dialects of the speakers. In Table 4.1 (b), the simple agreement between the actual dialect and the ratings is 2/3, or 0.67. But (c) will have exactly the same simple agreement, even though it is clear that the raters are agreeing among themselves in ways that do not match the actual dialects. In Table 4.1 (d), the simple agreement is 1/3 (0.33), even though the data do not show any relation at all between the actual dialects and the ratings. (a) (c) NY Chi Phil A 10 0 2 B 2 10 0 C 0 2 10 simple= 0:83 K= 0:55 NY Chi Phil A 8 0 4 B 0 8 4 C 4 0 8 simple= 0:67 K= 0:24 (b) (d) NY Chi Phil A 8 2 2 B 2 8 2 C 2 2 8 simple= 0:67 K= 0:18 NY Chi Phil A 4 4 4 B 4 4 4 C 4 4 4 simple= 0:33 K= ,0:09 Table 4.1: Examples of Measures of Agreement Because of the limitations of simple agreement, it is often preferable to use the kappa statistic (Scott 1955, Cohen 1960), which is corrected for chance agreement. This is the standard statistic for inter-rater reliability when the number of categories is xed for all raters. The basic formula is K = P (A) , P (E) 1 , P (E) where P(A) is the proportion of actual agreement, and P(E) is the proportion of expected agreement. Since the numerator is the di erence between the actual and the expected, results no better than chance will give kappas around 0. Returning to Table 4.1, (a) shows that kappa drops o quite sharply when even a little disagreement occurs, and (c) gives a higher kappa (0.24) than (b) (0.18) because of the greater agreement of the raters among themselves, even if they are incorrect. As expected, kappa is near 0 (actually slightly negative) when there is no agreement among raters, as in (d). (For further discussion of the
CHAPTER 4. PSYCHOLINGUISTIC EXPERIMENTS 143 logic of the statistic, see the excellent introduction in Siegel & Castellan (1988:284-91).) The omega statistic (Morey & Agresti 1984) is based on whether or not two raters classify each pairofstimuli in the same category or not, without regard to the classi cation of other pairs. Like kappa, it is corrected for chance agreement, so that it varies from 0 for chance agreement to 1.0 for perfect agreement. Omega is inherently less powerful than kappa, since it considers each pair of stimuli in isolation, however, it has the great advantage that it can be used in cases in which the number of categories di ers from rater to rater. We will therefore use omega to measure agreement among subjects on the Sorting task, where di erent subjects created di erent numbers of categories. Both kappa and omega are insensitive to the number of categories involved, or the type of distribution of instances into categories. The variance of the sampling distribution is known for both, so that the probability of a particular outcome can be calculated. Results and Analysis Task 1: Sorting The number of categories per subject ranged from 6 to 21, with a mean of 11. The proportion of examples in the categories varied greatly, from 33% for eye and 15% for recognize to 0 for some categories. The omega coe cient of agreement ranged from 0.09 to 0.49, with a median of 0.245. Task 2: Classi cation All subjects nished 99 sentences of the rst set. Some subjects continued on to other sets, but the order of the sets was randomized across subjects, so that there was little overlap beyond the rst set. The overall agreement among raters, measured by the kappa statistic, was .38 5 . This value is low, but understandable; not only were a large number of senses listed, but also many of the sentences were ambiguous when presented out of context. Discussion In evaluating the results of Experiment 1, the strengths and weaknesses of using corpus examples became apparent. On the one hand we had learned something about the 5 This and all of the values of kappa reported in this study are statistically signi cant at p
Page 1 and 2:
Seeing clearly: Frame Semantic, Psy
Page 3 and 4:
Seeing clearly: Frame Semantic, Psy
Page 5 and 6:
Next three psycholinguistic experim
Page 7 and 8:
BasicSenses........................
Page 9 and 10:
Acknowledgements Careful citations
Page 11 and 12:
acquisition. Aside from needing to
Page 13 and 14:
Chapter 1 Introduction Xue er bu s,
Page 15 and 16:
CHAPTER 1. INTRODUCTION 3 In lingui
Page 17 and 18:
CHAPTER 1. INTRODUCTION 5 Although
Page 19 and 20:
CHAPTER 1. INTRODUCTION 7 part on t
Page 21 and 22:
CHAPTER 1. INTRODUCTION 9 is not cl
Page 23 and 24:
CHAPTER 1. INTRODUCTION 11 1980:10-
Page 25 and 26:
CHAPTER 1. INTRODUCTION 13 L.U. Lex
Page 27 and 28:
CHAPTER 1. INTRODUCTION 15 spondenc
Page 29 and 30:
CHAPTER 1. INTRODUCTION 17 happy an
Page 31 and 32:
CHAPTER 1. INTRODUCTION 19 shows th
Page 33 and 34:
CHAPTER 1. INTRODUCTION 21 Regular
Page 35 and 36:
CHAPTER 1. INTRODUCTION 23 base for
Page 37 and 38:
CHAPTER 1. INTRODUCTION 25 d. The i
Page 39 and 40:
CHAPTER 1. INTRODUCTION 27 (12) a.
Page 41 and 42:
CHAPTER 1. INTRODUCTION 29 linguist
Page 43 and 44:
CHAPTER 1. INTRODUCTION 31 Introspe
Page 45 and 46:
CHAPTER 1. INTRODUCTION 33 on the o
Page 47 and 48:
CHAPTER 1. INTRODUCTION 35 they con
Page 49 and 50:
CHAPTER 1. INTRODUCTION 37 meaning
Page 51 and 52:
Chapter 2 A Frame Semantic Analysis
Page 53 and 54:
CHAPTER 2. A FRAME SEMANTIC ANALYSI
Page 55 and 56:
Page 57 and 58:
Page 59 and 60:
Page 61 and 62:
Page 63 and 64:
Page 65 and 66:
Page 67 and 68:
Page 69 and 70:
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104: CHAPTER 2. A FRAME SEMANTIC ANALYSI
Page 123 and 124: Chapter 3 Other Cognitive Approache
Page 125 and 126: CHAPTER 3. OTHER COGNITIVE APPROACH
Page 139 and 140: Semantics of Seer entity setting ph
Page 145 and 146: CHAPTER 4. PSYCHOLINGUISTIC EXPERIM
Page 153: CHAPTER 4. PSYCHOLINGUISTIC EXPERIM
Page 177 and 178: CHAPTER 5. WHAT THE DICTIONARIES SA
Page 205 and 206:
CHAPTER 5. WHAT THE DICTIONARIES SA
Page 207 and 208:
Page 209 and 210:
Page 211 and 212:
Page 213 and 214:
Page 215 and 216:
Page 217 and 218:
Page 219 and 220:
Page 221 and 222:
Page 223 and 224:
Page 225 and 226:
Page 227 and 228:
Page 229 and 230:
Page 231 and 232:
Page 233 and 234:
Page 235 and 236:
Page 237 and 238:
Page 239 and 240:
Page 241 and 242:
CHAPTER 6. FUTURE RESEARCH DIRECTIO
Page 243 and 244:
Page 245 and 246:
Page 247 and 248:
Page 249 and 250:
Page 251 and 252:
BIBLIOGRAPHY 239 Bierwisch, Manfred
Page 253 and 254:
BIBLIOGRAPHY 241 ||, & B.T.S. Atkin
Page 255 and 256:
BIBLIOGRAPHY 243 ||. 1990. The inva
Page 257 and 258:
BIBLIOGRAPHY 245 Osgood, C.E. 1970.
Page 259 and 260:
BIBLIOGRAPHY 247 Wasow, Thomas. 198
Page 261 and 262:
Appendix A Additional Corpus Exampl
Page 263 and 264:
APPENDIX A. ADDITIONAL CORPUS EXAMP
Page 265 and 266:
APPENDIX A. ADDITIONAL CORPUS EXAMP
Page 267 and 268:
APPENDIX B. SUMMARY OF MORPHOLOGY A
Page 269 and 270:
APPENDIX C. EXPERIMENT 1 257 Recogn
Page 271 and 272:
APPENDIX C. EXPERIMENT 1 259 to lea
Page 273 and 274:
APPENDIX C. EXPERIMENT 1 261 At the
Page 275 and 276:
Appendix D Stimuli for Experiments
Page 277 and 278:
APPENDIX D. STIMULI FOR EXPERIMENTS
Page 279 and 280:
Page 281 and 282:
Page 283 and 284:
Page 285 and 286:
Page 287 and 288:
Index activation, 13, 136, 137, 231
Page 289 and 290:
INDEX 277 description, 82 see proce
Page 291:
INDEX 279 173{178, 186, 190{192, 20
show all

Seeing clearly: Frame Semantic, Psycholinguistic, and Cross ...

Create successful ePaper yourself

Delete template?

Save as template?