A Wordnet from the Ground Up

3.3. Evaluation 51any pair of LUs, but also people are notoriously bad at working with real numbers.A linear ordering of dozens of LUs is nearly impossible, and even comparing twoterms requires a significantly complicated setup (Rubenstein and Goodenough, 1965).Given a small sample of the lists of the most semantically related LUs to the givenone, e.g., Table 3.11 and 3.12, people can easily distinguish a bad MSR from a goodone; we must distinguish good MSRs from those that are merely passable from theperspective of support for linguists working on wordnet development.We note three forms of MSR evaluation (Budanitsky and Hirst, 2006, Zeschand Gurevych, 2006):• mathematical analysis of formal properties (for example, the property of a metricdistance (Lin, 1998)),• application-specific evaluation,• and comparison with human judgement.Mathematical analysis gives few clues with respect to the results of future applicationsof an MSR. Evaluation via an application may make it difficult to separate the effect ofan MSR and other elements of the application (Zesch and Gurevych, 2006). A directcomparison to a manually created resource seems the least trouble-free. The constructionof such resources, however, is labour-intensive even if it only labels LU pairs assimilar (maybe just related (Zesch and Gurevych, 2006)) or not similar; this does notallow a fair assessment of the ordering of LUs on a continuous scale, as an MSR does.Indirect comparison with the existing resources (Grefenstette, 1993) is anotherpossibility. For example, one could compare an MSR constructed automatically andanother based on the semantic similarity across the hypernymy structure of PWN. Thisis how the main approaches work – see (Lin, 1998, Weeds and Weir, 2005, Geffetand Dagan, 2004). Two list of the k LUs most similar to the given one – for example,one constructed from an MSR and one from a wordnet – are transformed to ranknumbers of the subsequent LUs on the lists, and compared by the cosine measure. Thedrawback of such an evaluation is that we know how close the two similarity functionsare, but not how people perceive an MSR. The evaluation also strongly depends onthe wordnet similarity function applied. There are a number of such functions – see(Budanitsky and Hirst, 2006) – but many of them perform indifferently for a smallwordnet without full-fledged hypernymy structure (like the core plWordNet that wehad at our disposal during most experiments) or require synset probabilities. Moreover,wordnet similarity functions based on the hypernymy structure do not always work forverbs and adjectives, whose hierarchies tend to be quite limited. The similarity measureproposed by Mihalcea and Moldovan (1999) also does not apply in our case becauseplWordNet, like many other new wordnets, does not yet include glosses.