A Wordnet from the Ground Up

4.5. Hybrid Combinations 139ratio was randomly reduced to 1:1 (the uniform distribution of probability was appliedin drawing a new subset NH).P R F 1 P R F 1ratio 1:1 1:10Naïve BayesH+P2,P3+R 89.80 47.10 61.79 46.30 45.80 46.05H+P2,P3+R+E 84.70 59.10 69.62 34.60 53.50 42.02C4.5H+P2,P3+R 82.10 77.50 79.73 66.90 43.10 52.43H+P2,P3+R+E 81.70 78.40 80.02 60.70 39.90 48.15LMTH+P2,P3+R 81.80 80.60 81.20 72.80 39.40 51.13H+P2,P3+R+E 81.00 78.20 79.58 65.40 34.50 45.17Table 4.4: Evaluation for both sets using tenfold cross-validationPrecision and recall are calculated in Table 4.4 according to the description ofexamples extracted from plWordNet (H, P2, P3, R) or defined manually (E). Theresults achieved by both decision trees are very similar, and high by all three measures.However, the inclusion of the set E decreases the result significantly in comparisonto the high ratio |R| : |E|, that is to say, a small number of more difficult examplesnegatively influence the result. The R set includes more obvious and on average lessclosely semantically related pairs of LUs; it is generated randomly from plWordNet,but E includes only tricky examples. That is why we ran additional tests on a separateset of LU pairs selected randomly from MSRlists(a, 20) generated using MSR RW F .The set was annotated manually, and will be referred to as the manual test set (M).The best classifiers shown in Table 4.4 appeared to be biased towards positive decision,contrary to the classifiers trained on the 1:10 version of the learning data.In Figure 4.6 we present sample results of the classification selected from oneof the folds of the tenfold cross-validation (classifier C4.5, ratio KH to NH 1:10, Eincluded in NH) 13 .We prepared the set M in order to go outside plWordNet with the tests and to lookinto the work of the classifiers from the point of view of their potential application inlinguistic practice. As we wrote earlier, the set M was selected randomly from pairs ofLUs with the highest value of semantic relatedness according to MSR RW F . M consistsof 1984 negative and 316 positive examples.The C4.5 classifier trained on the KH=H+P2 and NH=P3+R+E with the ratio 1:10achieved a 21.69% precision, a 50.32% recall and a 30.31% F-score. This is a littlelower than the best result achieved by (Snow et al., 2005) using corpus-based attributes13 Many words in these pairs are polysemous in both languages. The English translations “select” theintended meaning.