Onto.PT: Towards the Automatic Construction of a Lexical Ontology ...

8.3. Evaluation 141
the most reliable. Following this kind of evaluation, we concluded that the accuracy
of tb-triples extracted from dictionaries depends on the relation type. Accuracies
are between 99%, for synonymy, and slightly more than 70%, for purpose-of and
property-of. Hypernymy, which is the relation with more extracted instances, is
about 90% accurate.
The enrichment of synsets (chapter 6) was evaluated after the comparison with
a reference gold standard, especially created for this task. The agreement on the
creation of the reference is moderate, and the accuracy of assigning a synonymy
pair to a synset with our procedure and the selected parameters is between 76% and
81%, depending on the judge. We recall that we are using TeP as a starting point for
the synset-base. Since the previous resource is created manually, the final quality of
synsets should be higher than the aforementioned values. As for the establishment
of new clusters from the remaining synonymy pairs, manual evaluation was once
again followed, and the accuracy of this step has shown to be around 90%. This
is an improvement towards the discovery of clusters from the complete synonymy
network, where accuracy for nouns was about 75% (chapter 5).
Finally, for evaluating the attachment of the term arguments of tb-triples to
synsets (chapter 7) we used two gold standards – one created manually, for the
hypernymy, part-of and purpose-of relations; and TeP, for antonymy relations. Using
the best performing algorithms, the precision of this step was measured to be
between 99% for antonymy, and 60-64%, for the other relations. This number,
however, considers only attachments to synsets with more than one lexical item.
For lexical items that originate a single-item synset, attachment is straightforward.
Given that more than two thirds of the Onto.PT synsets are single-item, ontologising
performance will be higher in the actual creation of Onto.PT.
All combined, we understand that there will be some reliability issues with the
contents of Onto.PT, common in an automatic approach. However, it is not possible
to speculate on a value or an interval for this reliability, because the ECO approach
is not linear. Having this in mind, we decided to complement the previous evaluation
hints by classifying a small part of the Onto.PT sb-triples manually, with results
provided in the next section. Although manual evaluation has almost the same
problems as creating large resources manually – it is tedious, time-consuming, and
hard to repeat – it is also the more reliable kind of evaluation.
Besides the manual evaluation, in section 8.4 we present how Onto.PT can be
useful in the achievement of some NLP tasks. This can be seen as a task-based
evaluation, which is the fourth strategy referred by Brank et al. (2005).
8.3.2 Manual evaluation
As referred earlier, manual evaluation suffers from similar issues as creating large
resources manually. It is a tedious, time-consuming job, and hard to repeat. However,
one of the few alternatives we found in the literature for evaluating a wordnet
automatically is based on dictionaries (Nadig et al., 2008). Since all the available
Portuguese dictionaries were exploited in the creation of Onto.PT, a similar evaluation
would be biased.
The manual evaluation of Onto.PT considered, first, the synsets alone and, second,
the proper sb-triples. More precisely, this evaluation had the following steps:
• Two random samples of 300 sb-triples were collected from Onto.PT: one with