A Wordnet from the Ground Up

138 Chapter 4. Extracting Relation Instancesfor Machine Learning. We manually annotated randomly selected pairs of LUs whichoccurred on MSRlists (a,20) for the LUs described by the MSR.From this selection, 1159 pairs classified as not relevant were collected into a setE. In some experiments, we added E to NH, see below.We experimented with two training sets produced by combining our regular datasets. Test sets were excluded randomly from training sets during tenfold cross-validation.Training sets are named in Table 4.4 according to the following description scheme:KH 1 + . . . + KH n ,NH 1 + . . . + NH mi.e. first the sets comprising KH are listed, next the sets from NH. The training setH+P2,P3+R includes only pairs extracted from plWordNet. It consists of 5027 KHpairs (H+P2) and 56531 NH pairs (P3+R). Tests on this set were done only on dataalready present in plWordNet. It is also more difficult than the sets used in (Snowet al., 2005), because the classifier is expected to distinguish between close hypernymsand more indirect hypernymic ancestors (P3 included in NH).Because plWordNet (the version June 2008) was still small, the second training setwas extended with the set E of manually classified pairs. We added only negative pairs,assuming that positive examples are well represented by pairs from plWordNet, whilemore difficult negative examples are hidden in the huge number of negative examplesautomatically extracted from plWordNet. The second training set consists of 5027 KH(H+P2) and 57690 NH (P3+R+E).In the experiments, we used Naïve Bayes (Mitchell, 1997) and two types of decisiontrees, C4.5 (Quinlan, 1986) and Logistic Model Tree [LMT] (Landwehr et al., 2003),all in the versions implemented in the Weka system (Witten and Frank, 2005). NaïveBayes classifiers are probabilistic, C4.5 is rule-based, and LMT combines rule-basedstructure of a decision tree with logistic regression in leaves. In order to facilitate acomparison of classifiers, we performed all experiments on the same training-test dataset. Because we selected C4.5 as our primary classifier, and we generated examplesfrom the same corpus (so the frequencies occurring as values of some attributes couldbe compared directly), we did not introduce any data normalisation or discretisation.The range of data variety was also limited by the corpus used. The application of thesame data to the training of a Naïve Bayes classifier resulted in a bias towards its morememory-based-like behaviour. According to the clear distinctions in the main groupof the applied data sets, however, the achieved result was positive, see Table 4.4.All experiments were run in the Weka environment (Witten and Frank, 2005). Ineach case, we applied tenfold cross-validation; the average results appear in Table 4.4.Because some classifiers, for example C4.5, are known to be sensitive to the biasedproportion of training examples for different classes (here, only two), we also testedthe application of random subsampling of the negative examples (NH) in the trainingdata. The ratio KH:NH in the original sets is around 1:10. In some experiments the