A Wordnet from the Ground Up

124 Chapter 4. Extracting Relation Instances4. the dependency on the filtering infrequent and very frequent patterns and instances.5. the way in which various statistical similarity measures used in reliability calculationchange the precision of the results.Human Eval. [%] Relative Recall InstancesEST+nm:th1.0 12 0.79 24552EST+nm:th2.6 37 1.00 10000EST+nm:th5.2 48 0.54 4170EST+nm:5seeds 22 0.71 11882EST+nm:10seeds 25 0.84 12476EST+nm:15seeds 24 0.85 13189EST+nm:5insts/1patt 24 0.83 12773EST+nm:10insts/1patt 29 1.03 13188EST+nm:40insts/1patt 37 1.00 10000EST+nm:k4 37 1.00 10000EST+nm:k8 41 2.80 25361EST+nm:k12 38 2.70 26501Table 4.3: The dependence of the algorithms on the parameter values (Kurc, 2008)In case 1 it seems that the highest threshold gives the best results – see Table 4.3and the first three rows, but a too high threshold decreases the total number of theextracted proper instances, as the relative recall is significantly decreased. There must,however, be a balanced ratio between instances selected for the next iteration andnew patterns induced. With few instances, there is no statistical evidence to induceproper patterns, and EST/ESP crawls picking almost random patterns. That leads tothe decrease in precision.Initial seeds, case 2 (marked ‘nnseeds’ in Table 4.3, where nn is the number) aremeant to generate a skeleton of a model of the lexico-semantic relation. If the numberof seeds is not high enough, the best extracted patterns can be random. Of course, onecould collect a small number of seeds that would indicate only expected patterns, butthat would require a precise analysis of the corpus used for instance extraction. Thatis pointless, because by using more seeds one can acquire the same patterns with lesseffort.The influence of the number of instances preserved between two subsequent iterationsis similar to the influence of the number of seeds, see the rows marked‘nninsts/1patt’ in Table 4.3 – nn preserved instances for one pattern. More instanceskept, and next used for the evaluation of the patterns, give better description of thewhole model. According to the experiments, at least 15 seeds and 10 instances for