A Wordnet from the Ground Up

56 Chapter 3. Discovering Semantic RelatednessSeveral sets of tests compared the results of MSR with human performance. Eachtime a subset of QA pairs was randomly selected from a complete WBST+H test anda group of native speakers of Polish were asked to solve the test. They were instructedto select for each question word only one answer, the closest in meaning to the question.There was no time limit in the task. Most participants were Computer Science students,but the LUs selected were mostly frequent units without technical senses, so the raters’background need not have influenced the results.The first two tests for nominal LUs were generated from early versions of the coreplWordNet:• plWordNet from June 2006, 24 native speakers of Polish tested on 2 randomsubsets of WBST+H; a set included 79 QA pairs; the average score was 89.29%,and interjudge agreement within one set, measured by Cohen’s kappa (Cohen,1960), ranged between 0.19 and 0.47 (Piasecki et al., 2007a);• plWordNet from March 2007, several native speakers of Polish, a random subsetof WBST+H; the average result close to 100%.The results of the second test showed the limits of WBST+H. Lacking a fullerversion of plWordNet, we decided to define a more difficult test, WBST-style test tofacilitate further work on MSRs for Polish nouns. This Enhanced WBST is presentedin detail in the next section.We also ran tests for verbal and adjectival LUs, both generated from the March2007 version of plWordNet. Twenty raters solved each test of a hundred QA pairs.The participants’ average scores appear in Table 3.3. The inter-judge agreement wasmeasured by Fleiss’s kappa, which accounts for agreement among many participants(Fleiss, 1971). The high value of kappa, supported by the manual evaluation of thetest results, shows that the agreement was high, and the raters made similar errors.Examples of QA pairs appear in Figure 3.2. A comparison of the results of humanraters on the verbal and adjectival QA pairs – 88.21% and 88.9%, respectively, withalmost 100% for the nominal pairs – shows that the verbal and adjectival parts ofWBST+H are more difficult for humans 2 and that one should expect lower results fromthe automatically extracted MSRs (Section 3.4.5).In 2008, another WBST+H was generated for nouns, verbs and adjectives fromthe final version of the core plWordNet (June 2008). 80 LUs were selected randomlyin 4 groups of 20 LUs for each range of LU frequency in the IPI PAN corpus(Przepiórkowski, 2004). We asked invited native speakers of Polish, mainly studentsof Computer Science, to solve the tests via dedicated Web pages. The results and thenumber of raters appear in Table 3.4.2 All three tests were generated from the same version of plWordNet.