Onto.PT: Towards the Automatic Construction of a Lexical Ontology ...

98 Chapter 6. Thesaurus Enrichment
6.2 Evaluation of the assignment procedure
The evaluation of the assignment procedure has two main goals. First, it quantifies
the performance of the assignment algorithm. Second, it enables the selection of the
most adequate settings, including the similarity measure and the best threshold σ
to use in the integration of the synpairs of PAPEL/CARTÃO in TeP.
6.2.1 The gold resource
To compare the performance of the assignment algorithm using different settings, we
randomly selected 355 noun synpairs of PAPEL 2.0 (Gonçalo Oliveira et al., 2010b)
and had them assigned, by two human annotators, to the synsets of TeP 2.0 (Maziero
et al., 2008). Before their selection, we made sure that all 355 synpairs had at least
one candidate synset in TeP. The manually assigned synpairs constitute a small gold
collection, used to evaluate the procedure with different settings. Even though the
creation of this resource was a time-consuming task, we now have a reference that
helps us understand the behavior of the algorithm. Furthermore, it is now possible
to repeat this kind of evaluation as many times as needed.
Lexical-semantic knowledge is typically subjective and thus hard to evaluate.
Besides depending heavily on the vocabulary range and intuition of the human
annotator, when it comes to the division of words into senses, even for expert lexicographers,
there is not a consensus because word senses are most of the time fuzzy
and also because language evolves everyday (see section 4.3).
In order to minimise this problem, both annotators manually selected the assigments
for the same 355 synpairs. On average, there were 4.31 candidate synsets for
each synpair with a standard deviation of 3.27. Also on average, the first annotator
assigned each synpair to 2.03±1.37 synsets, while, for the second, this number was
2.64±2.30. Their matching assignments were 70% and their kappa agreement 0.43,
which means fair/moderate agreement (Landis and Koch, 1977; Green, 1997) and
shows, once again, how subjective it is to evaluate this kind of knowledge.
6.2.2 Scoring the assignments
In order to select the best assignment settings, we performed an extensive comparison
of the assignment performance, using different similarity measures (introduced
in section 6.1.2) and different thresholds σ. In all the experimentation runs,
we used all the noun synpairs in CARTÃO, which includes PAPEL 3.0 and the synpairs
extracted from Wiktionary.PT and DA, to establish the synonymy network
for computing similarities. More about the size of this network and on its coverage
by TeP can be found in section 6.4.1.
The evaluation score of each setting was obtained using typical information retrieval
measures, namely precision, recall and F -score. For a synpair in the set of
assigned synpairs, pi ∈ P , these measures are computed as follows:
P recisioni = |Selectedi ∩ Correcti|
|Selectedi|
P recision = 1
|P |
|P |
P recisioni
i=1