A Wordnet from the Ground Up

More documents

Recommendations

Info

3.3. Evaluation 533.3.1 Wordnet-based synonymy test for PolishThe application of LSA to TOEFL data became unattractive as a method of comparingMSRs once the result of 97.5% hits has been achieved (Turney et al., 2003). Freitaget al. (2005) proposed a new test, WBST. It was seen as more difficult because itcontained many more questions. An instance of the test consists of many — hundreds oreven thousands — question-answer pairs [QA pairs]: 〈q, A〉, where A = a 1 , a 2 , a 3 , a 4and q, a i are LUs included in the wordnet that underlies the test ((Freitag et al., 2005)used PWN 2.0). In each QA pair there is a i , henceforth called the correct answer,such that there is a synset S in the wordnet and q, a i belong to S. None of the otherthree a j belongs to the same synset as q or as a i . We will call such a j detractorsfor the given QA pair. During evaluation, MSR generates values for the pairs 〈q, a i 〉,a i ∈ A, expected to favour the correct answer against the detractors.The WBST has been, amongst other applications, used to evaluate MSRs for PolishLUs (nominal, verbal and adjectival). The underlying resource was plWordNet, usedin different development versions for different tests. Further in this section we discusshow the wordnet used influences the difficulty of the test.The test had to be slightly modified. In plWordNet, many synsets have only1–2 LUs, in accordance with the definition of the synset and usage of LUs as basicplWordNet entries, see Section 2.1. In order to get a better coverage of LUs by WBSTquestions, and not to leave LUs in singleton synsets untested, the direct hypernymsof LUs from singleton synsets were taken to form QA pairs 1 (Piasecki et al., 2007a).We named this modification the WBST with Hypernyms [WBST+H]. The inclusion ofhypernyms in QA pairs did not make the test easier, as was shown in (Piasecki et al.,2007a).plWordNet has been evolving from the early versions including fewer LUs, broadersynsets with more vague understanding of near-synonymy (larger percentage of synsetswith more than two LUs) and shallower hypernymy structure, to the present versionof plWordNet expanded semi-automatically (Section 4.5.3), in which most synsets arenarrow (1–2 LUs on average) and the hypernymy structure is significantly deeper.Having broader synsets puts in the same broad synset the LUs hard to distinguishusing an MSR (they are very close in meaning). There is, therefore, no need todistinguish between their meaning during the test. In a version of plWordNet withnarrower synsets, the same LUs may have already been separated into two differentsynsets usually not related by direct hypernymy. One can expect that narrower synsetsobtained by dividing a broader one would be co-hyponyms. The hypernymy hierarchydeepened with the subsequent versions of plWordNet. This tendency was due to the1 In the case of adjectival LUs this technique has a limited application, because the number ofhypernymy instances is very small in the case of adjectival synsets – only 142 instances (plWordNet fromOctober 2008).
54 Chapter 3. Discovering Semantic Relatednesspartitioning of broad synsets into (usually) hyponyms of the original one, and to theintroduction of new lemmas in synsets which are hyponyms of the existing ones. Thisis illustrated in Table 3.1. We can observe the continuous decrease of the averagesynset size and the increase of the number of single-LU synsets.plWordNet 12.2006 9.2007 6.2008 11.2008 plWordNet 1.0Lexical units 11690 13164 16549 19620 26984Synsets 5314 8045 9085 11880 17695Singelton synsets 874 3745 5055 7660 12609LUs in synset (average) 2.20 1.64 1.82 1.65 1.52Table 3.1: Changes in synset structure during the development of plWordNetIn order to visualise the evolution of the WBST+H instances we used the best MSRsextracted for: nominal, verbal and adjectival LUs on the basis of the MSR GRW F (Lin)algorithm, which will be discussed in Section 3.4. The same three MSRs weretested with different versions of WBST+H produced from different archival versionsof plWordNet. The results appear in Table 3.2.WBST+HEWBSTplWordNet PoS Acc. [%] Lemmas QA Acc. [%] Lemmas QA12.2006 N 86.90 3661 10402 64.81 1780 402912.2006 V 81.34 2567 3905 — — —12.2006 A 82.63 1547 3484 — — —9.2007 N 85.99 3921 7522 66.15 3492 65129.2007 V 79.16 2567 4179 — — —9.2007 A 84.48 1580 3530 — — —6.2008 N 86.30 3816 6729 68.10 3391 57466.2008 V 75.29 2688 4734 — — —6.2008 A 83.61 1567 2690 — — —11.2008 N 88.14 5413 9486 69.75 5061 868911.2008 V 71.85 2677 5484 — — —11.2008 A 83.26 1574 2814 — — —plWN 1.0 N 87.60 9250 16826 73.28 8828 15832plWN 1.0 V 71.06 2910 6340 — — —plWN 1.0 A 81.53 1595 2875 — — —Table 3.2: The accuracy of the MSRs based on the Rank Weight Function algorithm (Section 3.4) inrelation to different tests and plWordNet versions (Lemmas – the number of lemmas in QApairs, QA – the number of QA pairs for the given test); plWN 1.0 refers to plWordNet,version 1.0Examples of QA pairs taken from different WBST+H versions are presented inFigures 3.1 and 3.2.
Page 1 and 2:
A Wordnetfrom the Ground Up
Page 3 and 4: Work financed by the Polish Ministr
Page 7 and 8: 6 Prefaceheartfelt thanks go to all
Page 9: 8 Chapter 1. Motivation, Goals, Ear
Page 12 and 13: 1.1. Motivation 11[a] special form
Page 14 and 15: 1.1. Motivation 13Affect (Strappara
Page 16 and 17: 1.2. The Goals of the plWordNet Pro
Page 18 and 19: 1.2. The Goals of the plWordNet Pro
Page 20 and 21: 1.3. Early Decisions 19Merge Model:
Page 22: 1.3. Early Decisions 214. On the ot
Page 25 and 26: 24 Chapter 2. Building a Wordnet Co
Page 49 and 50: 48 Chapter 3. Discovering Semantic
Page 53: 52 Chapter 3. Discovering Semantic
Page 103 and 104: 102 Chapter 4. Extracting Relation
Page 105 and 106:
104 Chapter 4. Extracting Relation
Page 107 and 108:
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121 and 122:
Page 123 and 124:
Page 125 and 126:
Page 127 and 128:
Page 129 and 130:
Page 131 and 132:
Page 133 and 134:
Page 135 and 136:
Page 137 and 138:
Page 139 and 140:
Page 141 and 142:
Page 143 and 144:
Page 145 and 146:
Page 147 and 148:
Page 149 and 150:
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
166 Chapter 5. Polish WordNet Today
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 186 and 187:
Appendix ATests for Lexico-semantic
Page 188 and 189:
187Test for adjectives (T. IX)1. p1
Page 190 and 191:
189RelatednessTest for nouns (T. XV
Page 192 and 193:
BibliographyAgarwal, Abhaya and Alo
Page 194 and 195:
Bibliography 193on Deep Lexical Acq
Page 196 and 197:
Bibliography 195Derwojedowa, Magdal
Page 198 and 199:
Bibliography 197Grefenstette, Grego
Page 200 and 201:
Bibliography 199Kurc, Roman. (2008)
Page 202 and 203:
Bibliography 201Mohammad, Saif and
Page 204 and 205:
Bibliography 203. (2006) “The pot
Page 206 and 207:
Bibliography 205and Technology 7(1-
Page 208 and 209:
List of Tables2.1 The size of the c
Page 210 and 211:
List of Figures2.1 The LU perspecti
Page 212 and 213:
List of Figures 2114.16 Completely
Page 214 and 215:
Index 213CBC, see Clustering by Com
Page 216 and 217:
Index 215169, 177, 178, 180, 182hyp
Page 218 and 219:
Index 217mutual hypernymy, 24Mutual
Page 220 and 221:
Index 219SUMO, 14Supported Vector M
Page 222:
A language without a wordnet is at
show all

A Wordnet from the Ground Up

Create successful ePaper yourself

Delete template?

Save as template?