A Wordnet from the Ground Up

3.5. Sense Discovery by Clustering 97in language data. The indirect evaluation defined in (Pantel, 2003, Lin and Pantel,2002) will measure the level of resemblance between the division into senses made bylinguists constructing the wordnet and that extracted via clustering.We wanted to evaluate the algorithm’s ability to reconstruct plWordNet synsets.That would confirm the applicability of the algorithm in the semi-automatic constructionof wordnets. We put nouns from plWordNet on the input list of nouns (E in thealgorithm). Because plWordNet is constructed bottom-up, the list consisted of 13298most frequent nouns in IPIC plus some most general nouns, see Section 3.4.5. Theconstraints were parameterised by 96142 features (41599 adjectives and participles,and 54543 nouns).Several thresholds used in the CBC algorithm (plus a few more in the evaluation)are the major difficulty in its exact re-implementation. No method of optimising CBCin relation to thresholds was proposed in (Pantel, 2003, Lin and Pantel, 2002) 27 and thevalues of all thresholds in (Pantel, 2003) were established experimentally. There alsowas no discussion of their dependence on the applied tools, corpus and characteristicsof the given language.Broda et al. (2008) performed such analysis in relation to Polish. Here we willoutline only most important conclusions. Experiments with using RWF instead ofPMI showed that RWF gives higher precision (38.81% versus 22.37%), but leads tofewer resulting word assigned to groups (744 versus 2980). The value of σ, whichcontrols when to stop assigning words to a committee (step 2b in Phase III of thealgorithm), must be carefully selected for each type of MSR separately. As the valueof σ increases, the precision increases too, but the number of words clustered dropssignificantly. When we make σ small and θ ElCom (meaning that word “is not similar”to any committee), we get relatively good precision but more words clustered. Wefound that contrary to the statement and chart in (Pantel, 2003), tuning both thresholdswas important in our case.The experiments confirmed our intuition that removing overlapping features inPhase III of CBC is too radical. The application of both proposed heuristics was testedexperimentally and resulted in the increased precision. The minimal-value heuristicincreased the precision from 38.8% to 41.0% on 695 words clustered. The usage ofthe ratio heuristic improves the result even further: the precision rises to 42.5% on 701words clustered. A manual inspection of the results showed that the algorithm tendsto produce too many overlapping senses when it uses the ratio heuristic.Because of indirect nature of evaluation proposed in (Pantel, 2003) we wanted toevaluate CBC in more direct and intuitive way. We assumed that proper clustering27 Automatising this process is very difficult, because the whole process is computationally very expensive.A full iteration takes 5–7 hours on a 2.13GHz PC with 6GB of RAM, which makes, say, anapplication of Genetic Algorithms barely possible.