A Wordnet from the Ground Up

3.3. Evaluation 533.3.1 Wordnet-based synonymy test for PolishThe application of LSA to TOEFL data became unattractive as a method of comparingMSRs once the result of 97.5% hits has been achieved (Turney et al., 2003). Freitaget al. (2005) proposed a new test, WBST. It was seen as more difficult because itcontained many more questions. An instance of the test consists of many — hundreds oreven thousands — question-answer pairs [QA pairs]: 〈q, A〉, where A = a 1 , a 2 , a 3 , a 4and q, a i are LUs included in the wordnet that underlies the test ((Freitag et al., 2005)used PWN 2.0). In each QA pair there is a i , henceforth called the correct answer,such that there is a synset S in the wordnet and q, a i belong to S. None of the otherthree a j belongs to the same synset as q or as a i . We will call such a j detractorsfor the given QA pair. During evaluation, MSR generates values for the pairs 〈q, a i 〉,a i ∈ A, expected to favour the correct answer against the detractors.The WBST has been, amongst other applications, used to evaluate MSRs for PolishLUs (nominal, verbal and adjectival). The underlying resource was plWordNet, usedin different development versions for different tests. Further in this section we discusshow the wordnet used influences the difficulty of the test.The test had to be slightly modified. In plWordNet, many synsets have only1–2 LUs, in accordance with the definition of the synset and usage of LUs as basicplWordNet entries, see Section 2.1. In order to get a better coverage of LUs by WBSTquestions, and not to leave LUs in singleton synsets untested, the direct hypernymsof LUs from singleton synsets were taken to form QA pairs 1 (Piasecki et al., 2007a).We named this modification the WBST with Hypernyms [WBST+H]. The inclusion ofhypernyms in QA pairs did not make the test easier, as was shown in (Piasecki et al.,2007a).plWordNet has been evolving from the early versions including fewer LUs, broadersynsets with more vague understanding of near-synonymy (larger percentage of synsetswith more than two LUs) and shallower hypernymy structure, to the present versionof plWordNet expanded semi-automatically (Section 4.5.3), in which most synsets arenarrow (1–2 LUs on average) and the hypernymy structure is significantly deeper.Having broader synsets puts in the same broad synset the LUs hard to distinguishusing an MSR (they are very close in meaning). There is, therefore, no need todistinguish between their meaning during the test. In a version of plWordNet withnarrower synsets, the same LUs may have already been separated into two differentsynsets usually not related by direct hypernymy. One can expect that narrower synsetsobtained by dividing a broader one would be co-hyponyms. The hypernymy hierarchydeepened with the subsequent versions of plWordNet. This tendency was due to the1 In the case of adjectival LUs this technique has a limited application, because the number ofhypernymy instances is very small in the case of adjectival synsets – only 142 instances (plWordNet fromOctober 2008).