pdf 820Kb - INSEAD CALT

4 Gathering the Data
Evaluation report of the use of Onto-Logging
platform in the user site
Deliverable ID: D8b
Page : 36 of 110
Version: 1.0
Date: 27 january 2004
Status: Final
Confid.: Public
In this section, we are going to present the orientation chosen for this evaluation (qualitative
rather than quantitative), the selection and elaboration of the instruments for capturing the
data, as well as a description of the process of gathering of the data that was conducted.
4.1 Previous work
We do not pretend that the use of Ontology in the context of knowledge management is
totally new, and that nothing has been done in the field of evaluating the performance of
these systems. However, we believe that the previous work does not address (or at least only
partially cover) some of the critical dimension of the contribution of ontology to the design of
the next generation knowledge management systems, in particular related to the new usages.
In the next chapters, we are going to examine different approaches that have been used to
evaluate ontology-based knowledge management systems, and that we could consider using
for our evaluation.
4.1.1 The IR (Information Retrieval) approach
Many of the previous researches work on evaluation originate from a database conceptual
background, and focus in evaluating the performance of information retrieval (IR) from a
search perspective. In this context, one of the principal methods used for the evaluation is the
“precision and recall” method (Ragha van, Jung, and Bollmann, 1989), which consists in
measuring the noise associated to the result of a search (percentage of relevant documents
among the retrieved ones) and the coverage of this search (percentage of retrieved relevant
documents among all existing relevant documents).
This method has been applied in several cases to the evaluation of Ontology-based system
(Ehrig, 2002), sometime to demonstrate how Ontology can contribute to the improvement of
the search process (Aitken and Reid, 2000; Clark, et al, 2000).
Investigating how an ontology-based knowledge management system can compare to a
traditional document-centred (IR focussed) present however little interest. Indeed documentcentred
knowledge management systems have mainly failed to get widely adopted by the
companies (with the exception of some niches such as competitive intelligence), probably
because the functions they propose and usages they support do not really fulfil enough the
companies’ and people’s needs. Besides, trying to make a innovation only to improve the
support of an existing practice is doomed to fail. Innovations succeed because of their
capacity to invent and to support new practices. The example of the Object oriented databases
technologies that have tried in the past to mimic the approach of the older relational database