A Wordnet from the Ground Up

3.3. Evaluation 57PoS Min [%] Avg [%] Max [%] KappaVerb 84 88.21 95 0.84Adjective 82 88.9 95 0.85Table 3.3: Results of a manual WBST for Polish verbs and adjectives – the evaluation performed for(Broda et al., 2008) (May 2007)R Min [%] Max [%] Avg [%]Noun 29 73.84 96.24 86.64Verb 50 57.54 90.04 81.84Adjective 43 76.24 96.24 89.94Table 3.4: Results of human raters in WBST+H tests generated from the final version of the core plWord-Net (R — a number of raters for the given test)It is misleading to compare the results in Table 3.4 with the almost 100% inWBST+H generated from the May 2007 plWordNet. The increase from 89.29% forJune 2006 plWordNet to nearly 100% for May 2007 plWordNet was caused by theremoval of many obvious errors in broad synsets of the early version of plWordNet. Inmany QA pairs of the former test, raters were misled by strange QA pairs occurring inthe test. So, we can assume the level of almost 100% as the starting point. Consideringthis, when people solve the tests, we can observe a relation between the wordnet usedand the difficulty of the WBST+H test opposite to what happens when MSR is applied:the results are slightly higher for new versions of WBST+H, see Table 3.2. The testresults (produced for the same MSR) stayed approximately at the same level for thesubsequent versions of the core plWordNet, and increased with the present version ofplWordNet expanded semi-automatically with several thousand LUs (Section 4.5.4).3.3.2 Enhanced WBSTIn the WBST defined by Freitag et al. (2005) the elements of the answer set A notsynonymous with Q are chosen at random from the whole wordnet. Thus, the differencein meaning between Q and the detractors is usually obvious to test-takers 3 . It also tendsto be relatively easy for a good MSR, e.g. (Piasecki et al., 2007b). Our overall goal,however, was to construct an MSR that expresses clear preference for the wordnetrelations (focused on semantic similarity in the sense of Mohammad and Hirst (2006)— Section 3.4.2). Such MSR could be used to automatically extract synsets, i.e. to3 The latest versions of the expanded plWordNet introduced more fine-grained distinctions betweenlemma senses. This made WBST+H more difficult for humans, as shown in Table 3.4 in relation to theprevious test results discussed.