A Wordnet from the Ground Up

164 Chapter 4. Extracting Relation Instancesintroduced mainly for comparison with other approaches (but it is unnatural fromthe point of view of the linguists’ work);• Best P ≥1 – one closest attachment site is evaluated (similarly to the P ≥1 inSec. 4.5.4).Table 4.6 presents the result with a distinction between strong (marked S) andweak fitness (W). As expected, the accuracy of suggestions based on strong fitness issignificantly higher then for weak fitness. Because of the intended use, we assumedthat not only direct hits are useful – if the proposal is close enough to the correct placein plWordNet structure, then it is also a valuable suggestion. The same applies if thereis meronymy or holonymy between the suggested and correct synset.The results are encouraging. Almost half of the suggestions based on strong fitnessare in the close proximity of the correct place in wordnet structure. If making onlyone suggestion was required, the accuracy was boosted to 73.58%. For our goal, thisis an artificial constraint, but it shows how well the AAA algorithm would behave ina fully unsupervised way. Our ultimate goal, though, is to create a tool for supportinga linguist’s work, so the result for Best P ≥1 strategy shows more meaningful data:for how many words there is at least one useful suggestion. The AAA algorithmssuggested at least one strictly correct attachment site for 42.81% words, or for 81.96%words if we consider that close proposals are also useful.Comparison to other ways of automatic expanding a wordnet can be misleading.That is because our primary goal was to construct a tool that facilitates and streamlinesthe linguists’ work. Still, even if we compare our automatic evaluation with the resultsin (Widdows, 2003) during comparable tests on the PWN, our results seem to be better.For example, we had 34.96% for the highest-scored proposal (One S+W in Table 4.6),while Widdows reports a 15% best accuracy for a “correct classifications in the top 4places” (among the top 4 highest proposals). Our similar result for the top 5 proposalsis even higher, 42.81%. The best results reported by Alfonseca and Manandhar (2002)and Witschel (2005) are also at the level of about 15%, but were achieved in tests ona much smaller scale. Witschel also performed tests only in two selected domains. Thealgorithm of Snow et al. (2006), contrary to ours, can be applied only to probabilisticevidence.We made two assumptions: attachment based on the activation area and the simultaneoususe of multiple knowledge sources. The assumption appears to have beensuccessful in boosting the accuracy above the level of the MSR-only decisions (whichis roughly represented in our approach by weak fitness).WNW seems to improve the linguist’s efficiency a lot, but longer observations arenecessary for a reliable justification.The AAA algorithm is overburdened with parameters. Further research is requiredto find either a simplified form or an effective method of parameters optimization.