A Wordnet from the Ground Up

122 Chapter 4. Extracting Relation InstancesESPfree- – extends ESPmorf- by the representation of the free order of the instanceelements,EST- — Estratto without generic patterns, exploiting specific features of Polish, e.g.the agreement on values of selected categories is represented,EST-nm — Estratto without generic patterns, exploiting specific features of Polishlanguage and the extended reliability measures (4.6),EST+nm – the same as EST-nm but using generic patterns.If not stated otherwise, the threshold for confidence is 1.0 for all ESP systems and2.6 for EST systems. The number k of top patterns was set to k = I + 2, where I isthe iteration count. There were four iterations. In those experiments whose results wepresent, the focus was only on the hyponymy/hypernymy relation. IPIC was selected asthe main corpus, on which we ran all experiments with results presented in the tables.We ran three groups of experiments on Espresso and Estratto (Kurc, 2008). Webegan with experiments designed to analyse the influence of the proposed extended reliabilitymeasure (4.6) and six pattern schemes: ESP-, ESP-mn, ESPmorf-, ESPfree-,EST-nm and EST+nm. The question was whether they improve the results, since theymay better cope with certain characteristics of Polish. In Table 4.2, the precisionbased on human judgement is presented in the column labelled “Hum. eval.”. Thelevels of precision defined in the column group labelled “Ranking” are achieved forthe top subsets of instances described in the column group named “Inst.”. They showon how large a portion of the extracted instances we can rely, and how strongly. Take,for example, the first row: ESP- extracted the top 8% of instances with the precisionabove 70% and the top 22% with the precision 60%. The higher the numbers, thehigher concentration of positive instances in the upper part of the extracted list. Theprecision measured in relation to plWordNet is presented in the column “Prec. plWN”(the number of the extracted plWordNet instances is also given). The column labelled“Rel. R” refers to the recall calculated in relation to the result of ESP-.The results of the first group of experiments, presented in Table 4.2, allow us toconclude that the modified reliability measure (4.6) performs better either in the caseof the original Espresso scheme patterns (ESP-nm is the winner) or Estratto patternswhich take into account some properties of Polish (EST-nm had the best overall result).The situation is less clear in the case of the precision based on plWordNet – thedifferences are smaller – but still ESP-nm and EST-nm produce better results thanthe other versions; plWordNet is relatively small, however, and this could bias thecalculation. The manual evaluation showed that in fact plWordNet might be used onlyfor a very rough estimation of precision. The plWordNet-based precision of ESPnmversus EST-nm is almost identical, but EST-nm is much better in relation to the