A Wordnet from the Ground Up

4.5. Hybrid Combinations 141only (F-score = 0.3268) and measured in relation to manually annotate examples. Ourproblem setting is more difficult (we expect the classifier to distinguish e.g. betweenP2 and P3, while Snow et al. included all indirect hypernyms in KH) and we hadmuch fewer learning examples. Also, Snow et al. worked with a hybrid system thatcombined the hypernymy classifier with a MSR. It is more related to our WordNetWeaver system presented in the next section. Snow et al. had the best F-score of0.2714 for the classifier-only version.Leaving aside automatic evaluation, one can notice that the percentage of falsepositives is still significantly below 50%, which is a ratio that seems acceptable fora tool to support linguists. On the other hand, the number of LU pairs presented tolinguists dropped dramatically in comparison to MSR RW F alone, from 2300 to 733 –31.87% of the initial list. The classifier cannot be used alone as a support tool, butits ability ‘concentrate’ KH pairs in the positively classified group will be leveraged inthe next section for the construction of a tool combining different types of evidence inexpanding plWordNet.The results achieved on M for all classifiers were much poorer than the results onsets selected from plWordNet. We tried SVM as well, hoping for its usually goodperformance on numerical features without discretisation, but in contrast with thefindings of Kennedy (2006) we have not achieved any valuable result.In Figure 4.7 we present examples of classifier decisions made for elements of setM (classifier C4.5, ratio KH to NK 1:10, E included in NK).A manual inspection of false positives in the classification results on set M showsthat many pairs are co-hyponyms. They can be treated as positive answer from a linguists’spoint of view, but we tried to train the classifier not to select co-hyponyms asrelevant pairs.The results achieved on the data extracted from plWordNet are very promising,especially when we compare them to the results of similar experiments in (Snow et al.,2005), where the highest value of F-score was 0.348. A direct comparison, however,is not possible, because we used examples of KH and NH generated directly fromplWordNet, not from sentences in the corpora. Randomly generated pairs can includea larger percentage of obviously negative cases. On the other hand, plWordNet is muchsmaller than PWN applied in (Snow et al., 2005), so some NH pairs are in fact relevantpairs not yet added to plWordNet. This introduces substantial noise during training.The results on the manually annotated set M, and manually inspected, show thatthe performance of the classifiers on real data is lower. They have problems withdistinguishing co-hyponym pairs from relevant pairs, and there are more errors for lessobvious cases. Still, if we consider a task of delivering valuable suggestions to thelinguists, we have achieved an enormous improvement in comparison with the lists ofk most semantically related LUs. That is to say, a majority of the list elements are