A Wordnet from the Ground Up

120 Chapter 4. Extracting Relation InstancesThe former introduces a bias – plWordNet still is relatively small – enables testingthe whole set of instances, while manual evaluation is always laborious and can beperformed only on a sample. Yet, the samples have been chosen as for the manualpatterns (Israel, 1992), so the results can apply to the whole sets with a 95% confidence.In both types of comparison we applied the standard measures of precision andrecall (Manning and Schütze, 2001) 9 .Precision and recall are defined in the standard way: tp is the number of truepositives (extracted pairs of LUs which are instances of the target relation), fp is thenumber of false positives (incorrect instances marked by algorithms as correct), fn –false negatives (correct instances in text but not extracted by the algorithm).P =R =tptp + fptptp + fn(4.7)(4.8)Note that the denominator in R accounts for correct patterns or instances thatwere either marked as incorrect or not extracted at all. We cannot treat the limitedcore plWordNet as the exhaustive description of relations. That is why recall in ourapproach only measures the ratio of rediscovery of the plWordNet structure. It is not arecall in terms of all correct instances in the corpus or patterns that the corpus supports.Thus, following Pantel and Pennacchiotti (2006), we also use the relative recallmeasured in relation to the results of some other algorithm (Kurc, 2008, pp. 72):R A|B = R AR B=tp ACtp BC= tp Atp B= P A × (tp A + fp A )P B × (tp B + fp B )(4.9)where R A and R B denote the recall of the algorithms A and B, and C is the unknownnumber of instances occurring in the corpus.We extracted a ranked list of possible instances which can be sorted in descendingorder by reliability. The values are real numbers and there is no characteristic pointbelow which we can cut off the rest of pairs according to some analytical properties.Thus, instead of pure precision and recall, we prefer to use cut-off precision and cut-offrecall calculated only in relation to some n first positions on the sorted list of results(instances or patterns).In the end, then, we used three evaluation measures.1. Cut-off precision based on plWordNet marks as correct only those instances andpatterns that were found both in plWordNet and on an additional list provided a9 The F-measure could not be applied because of the limitations of recall based on plWordNet, to bediscussed later.