A Wordnet from the Ground Up

52 Chapter 3. Discovering Semantic RelatednessAutomatic differentiation between words synonymous and not synonymous witha given LU is a natural application for an MSR, especially in the context of generation ofsuggestions for a linguist. In Latent Semantic Analysis [LSA] (Landauer and Dumais,1997) the MSR constructed using a statistical analysis of a corpus (cf Section 3.4.2)was used to make decisions in a synonymy test, a component of the Test of English asa Foreign Language [TOEFL]. This gave 64.4% of hits. Turney (2001) reported 73.75%hits, and Turney et al. (2003) 97.5% hits; the latter practically solved the TOEFLsynonymy problem. TOEFL is focused on humans, a big advantage for applicationsin MSR evaluation. On the other hand, it is manually constructed, hence its maindrawbacks: limited size and fixed orientation on synonymy.Freitag et al. (2005) proposed a WordNet-Based Synonymy Test [WBST], whichseems to offer an interesting response to the limitations of TOEFL. WBST has beenbased on the use of PWN to generate “a large set of questions identical in formatto those in the TOEFL”. WBST is discussed in details in Section 3.3.1, but its twoproperties are worth emphasising now. First, it is larger and broader than TOEFLbecause it is automatically generated from a very large manually constructed resource.Second, with a change in the way of selecting question-answer pairs, a WBST-like testcan evolve from a synonymy test to a test oriented toward wordnet relations or in thesense of (Mohammad and Hirst, 2006).The best reported result for English nouns is 75.8% (Freitag et al., 2005). A slightlymodified WBST was used to evaluate an MSR for Polish nouns (Piasecki et al., 2007a)with the result of 86.09%.The evaluation of an MSR via a synonymy test shows the ability of the MSR todistinguish synonyms from non-synonyms. Since the MSR is the centrepiece of theapplication, the achieved results can be directly attributed to it. There was, however,a problem: WBST appeared to be too easy, as we show in Section 3.3.1. It is orientedtoward testing the main distinction — closely semantically related versus unrelated —because the incorrect answers are selected randomly and on average they are semanticallyunrelated to the question and the answer. The usefulness of WBST is thereforelimited with respect to its use in the development of more sophisticated MSRs focusedon semantic similarity and wordnet relations.In view of these findings, we have explored the possibility of generating more demandingautomatic methods of MSR assessment, following the general idea of WBST.We proposed an Enhanced WBST [EWBST] which is precisely a template of WBSTlikeevaluation methods parameterised by the way in which detractors, i.e. false answers,are selected. We wanted its results to be easily interpreted by people and itsfeasibility tested on people. We also expected that it would pick the MSR that isa better tool for the recognition of lexico-semantic relations between LUs.