GfKl 2008 - Legos

Evaluation Strategies for Learning Algorithms
of Hierarchical Structures
Korinna Bade 1 and Dominik Benz 2
1 Faculty of Computer Science, Otto-von-Guericke-University Magdeburg,
D-39106 Magdeburg, Germany, Email: korinna.bade@ovgu.de
2 Department of Electrical Engineering/Computer Science, University of Kassel,
D-34121 Kassel, Germany, Email: benz@cs.uni-kassel.de
Abstract. The idea to automatically induce a hierarchical structure among a set
of objects or integrate a given hierarchy into the learning process is common to
a number of disciplines like hierarchical clustering (Bade and Nürnberger, 2008)
and classification or ontology learning. A crucial aspect hereby is how to assess the
quality of the learned hierarchical scheme. Existing evaluation approaches can be
broadly classified in methods defining quality metrics on the resulting scheme alone
and methods which invoke an external “gold-standard” for comparison. We focus
on the latter case, for which various similarity metrics have been proposed, mostly
depending on the characteristics of the applied learning procedure.
This work aims at bringing together the different disciplines by presenting and
comparing existing gold-standard based evaluation methods for learning algorithms
that generate hierarchical structures. We present an interdisciplinary framework in
order to enable comparison across the different contexts, from which the metrics
originate. Our goal is to emphasize the strong similarities of evaluation tasks in different
disciplines and to create a general pool of evaluation methods. Based on prior
work (Dellschaft and Staab, 2006), we analyze properties of (good) evaluation measures.
Different types of structural errors in the learned hierarchies are identified and
their effects on existing measures are shown. Observing strengths and weaknesses of
existing methods, we also suggest some new methods.
Key words: evaluation metrics, hierarchical clustering, ontology learning, goldstandard
References
Bade, K. and Nürnberger, A. (2008): Creating a Cluster Hierarchy under Constraints
of a Partially Known Hierarchy. In: Proceedings of the 2008 SIAM International
Conference on Data Mining. (to appear)
Dellschaft, K. and Staab, S. (2006): On How to Perform a Gold Standard Based
Evaluation of Ontology Learning. In: Proc. of 5 th Int. Semantic Web Conference.
228–241.
− 8 −