Onto.PT: Towards the Automatic Construction of a Lexical Ontology ...

7.2. Ontologising performance 123
Table 7.2 presents the scores obtained for each measure, according to algorithm
and relation type. In RP and RP+AC, the threshold θ was empirically set to 0 and
0.55, respectively. In NT+AC, θ was set to 3.
Algorithm
Hypernym-of (210 tb-triples)
Precision (%) Recall (%) F1(%) F0.5(%) RF1(%)
RP 53.8 12.4 20.2 32.3 50.3
AC 60.1 15.7 24.9 38.4 59.8
RP+AC 58.5 15.6 24.6 37.7 58.5
NT 57.7 17.3 26.6 39.4 57.7
NT+AC 58.7 15.3 24.3 37.4 58.6
PR 46.2 11.5 18.5 28.9 45.7
MD 58.6 15.8 24.9 38.0 58.6
Part-of (175 tb-triples)
Precision (%) Recall (%) F1(%) F0.5(%) RF1(%)
RP 56.9 10.6 17.9 30.4 47.0
AC 58.7 14.9 23.8 37.0 58.7
RP+AC 64.1 16.6 26.3 40.7 64.1
NT 50.7 15.8 24.1 35.2 50.7
NT+AC 59.2 15.2 24.2 37.5 59.2
PR 50.6 12.6 20.2 31.6 49.9
MD 59.1 15.3 24.4 37.6 59.1
Purpose-of (67 tb-triples)
Precision (%) Recall (%) F1(%) F0.5(%) RF1(%)
RP 51.5 5.1 9.3 18.3 32.6
AC 63.2 13.0 21.5 35.6 63.2
RP+AC 63.4 13.6 22.3 36.5 63.4
NT 48.1 15.4 23.3 33.7 48.1
NT+AC 62.2 13.9 22.7 36.6 62.2
PR 56.3 10.8 18.2 30.6 56.3
MD 60.9 12.7 20.9 34.5 60.9
Table 7.2: Ontologising algorithms performance results.
The comparison shows that the best performing algorithms for hypernymy are
AC and NT, which have close F1 and RF1. NT is more likely to originate ties
for the best attachments than AC, and thus to have higher recall. However, its
precision is lower than AC’s. For part-of, RP+AC is clearly the best performing
algorithm. For purpose-of, RP+AC has also the best precision and RF1, but its
scores are very close to AC. Moreover, it is outperformed by NT and NT+AC in
the other measures. Once again, NT has higher recall and thus higher F1. NT+AC
combines good precision and recall in an interesting way and has therefore the best
F0.5. However, as that the set of purpose-of tb-triples contains only 67 instances, the
results for this relation might not be significant enough to take strong conclusions.
These results show as well that PR has the worst performance for hypernymy
and part-of tb-triples, which suggests that PageRank is not adequate for this task.
For purpose-of, RP is the worst algorithm, especially due to the low recall.
7.2.3 Performance against an existing gold standard
In the second performance evaluation, we used the proposed algorithms to ontologise
antonymy relations. For this purpose, the antonymy sb-triples of TeP were converted
to tb-triples. This resulted in 46,339 unique antonymy pairs – 7,633 between nouns,