Onto.PT: Towards the Automatic Construction of a Lexical Ontology ...

74 Chapter 4. Acquisition of Semantic Relations
of error for the correct relations, with a confidence level of 95% 14 (ME(2)). The
same table presents the agreement between the judges, quantified as the number of
classification matches (IAA) and as the Kappa value (κ, see Cohen (1960); Carletta
(1996)), where the amount of agreement expected by chance is removed, because
the probability of each judge giving a certain classification is considered.
Relation Judge
n synonym-of n
v synonym-of v
n hypernym-of n
n member-of n
v causation-of n
v purpose-of n
adj property-of v
Resource
DLP DA Wikt.PT
0 1 2 0 1 2 0 1 2
J1 0% 0% 100% 1% 0% 99% 1% 0% 99%
J2 0% 1% 99% 1% 0% 99% 2% 0% 98%
J1 0% 0% 100% 1% 0% 99% 5% 0% 95%
J2 1% 1% 98% 01% 0% 99% 5% 2% 93%
J1 2% 4% 94% 5% 3% 92% 4% 12% 84%
J2 3% 6% 91% 9% 7% 84% 4% 8% 88%
J1 4% 0% 93%+3% 7% 5% 85%+3% 12% 5% 79%+4%
J2 4% 3% 76%+17% 12% 3% 42%+43% 18% 14% 54%+14%
J1 4% 1% 95% 1% 3% 96% 7% 10% 83%
J2 4% 6% 90% 4% 5% 91% 7% 7% 86%
J1 31% 0% 69% 25% 1% 74% 24% 1% 75%
J2 29% 2% 69% 20% 1% 79% 25% 0% 75%
J1 23% 5% 72% 18% 11% 71% 26% 5% 69%
J2 12% 1% 78% 12% 10% 78% 15% 1% 75%
Table 4.11: Results of the manual evaluation of tb-triples according to resource.
Relation Judge
n synonym-of n
v synonym-of v
n hypernym-of n
n member-of n
v causation-of n
v purpose-of n
adj property-of v
Total
0 1 2
J1 2 (1%) 0 298 (99%) 1.1%
J2 3 (1%) 1 (≈0) 296 (99%) 1.1%
J1 6 (2%) 0 294 (98%) 1.6%
J2 7 (2%) 3 (1%) 290 (97%) 1.9%
J1 11 (4%) 19 (6%) 270 (90%) 3.4%
J2 16 (5%) 21 (7%) 263 (88%) 3.7%
J1 23 (8%) 10 (3%) 257+10 (86%+3%) 3.5%
J2 34 (11%) 20 (7%) 172+74(57%+25%) 4.3%
J1 12 (4%) 14 (5%) 274 (91%) 3.2%
J2 15 (5%) 18 (6%) 267 (89%) 3.5%
J1 80 (27%) 2 (1%) 218 (73%) 4.9%
J2 74 (25%) 3 (1%) 223 (74%) 4.9%
J1 67 (22%) 21 (7%) 212 (71%) 5.1%
J2 39 (13%) 30 (10%) 231 (77%) 4.7%
ME (2) IAA κ
Table 4.12: Results of the manual evaluation of tb-triples.
0.99 0.66
0.98 0.68
0.93 0.64
0.67/0.88 0.32/0.55
0.93 0.60
0.79 0.48
0.81 0.56
Manual validation showed that synonymy relations are the most reliable in
CARTÃO, as their accuracy is always close to 100%. This was somehow expected,
because these relations are also the easiest to extract (see section 4.2.3). The lowest
accuracy of synonymy was that of the triples extracted from Wiktionary.PT (93%-
95%). This happens mainly due to parsing errors on the wikitext indicating the kinds
of verb (e.g. transitive, intransitive). This problem has however been corrected for
further versions of CARTÃO. About 90% of the hypernymy triples are correct. Of the incorrect ones, most
connect related words. This happens especially in Wiktionary, where definitions, as
the following, result in the extraction of hypernyms and not members or synonymys:
14 The calculation of the margins of error assumed that the evaluated samples were selected
randomly, which is not exactly what happened. Although the triples in the samples were selected
randomly, there was a constraint to make each sample contain exactly 100 tb-triples from each
resource, and the resources have different sizes.