pdf 820Kb - INSEAD CALT

More documents

Recommendations

Info

Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 38 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public For instance (Missikoff, Navigli and Velardi, 2002) reports some experiences in the design of domain ontology building, and for instance the use of some tools (OntoLearn) for speedingup the elaboration of such ontology and its content population. These tools propose for instance to provide support to the automatic extraction of concepts, to the validation of the extracted concepts and to ontology content population. The assessment of these tools has used as criteria some proxy for the quality of the ontology (coverage, consensus and accessibility) and for the quality of the content (measured via precision and recall). In a similar way (Guarino and Welty, 2002) proposes, with OntoClean, a methodology for designing and evaluating “good” ontology, in a more systematic manner than what is the current practice. Whilst these approaches are useful, and important to really asset the quality of a technical system, they do not guaranty the value of this system according to an organizational perspective. In particular, they fail to recognize (or do not justify) the profound factors that are important to the real adoption and success of a system in an operational environment. Besides they can, in particular in an early stage of a research (pursuing radical innovation), distract the attention from the factors that are really matters. We refer here to the metaphor of “the person looking for his lost keys under the street lamp” (although the keys were most likely lost far from the street light, the person laments that he has little chance of finding the keys in the dark so why waste time looking there). Indeed, the availability of instruments introduces some cognitive bias, and in particular tend to reinforce research in areas that have already been explored in disfavour of more peripheral areas that can potentially lead more to radical innovation, and for which the instruments has not been invented yet. For instance, in the case of an Ontology-based system, such an evaluation may acknowledge the efficiency of a search mechanism but fail to recognise that value of this mechanism is limited (people use mainly navigation). In a similar way, this evaluation may acknowledge the capability of this system to manage an important quantity of documents (given a sophisticated model to capitalize documents) but fail to recognise that in a real environment, it may be undesirable to formalize extensively documents (because of problems of obsolescence, and because of problems of diminishing (or even negative) return related of the generation of noise). For instance, (Buckingham Shum 1997) advocates to avoid premature formalization and that only stable and sanctioned knowledge should be formalised. This confirms the personal experience of the author of this Ontologging evaluation report that documents represent generally chunk of knowledge that outdate very rapidly, and therefore should not be systematically formalized. This is to compare to “knowledge items” such as projects, people and topics on the contrary have a much longer lifespan (experts usually do not easily radically change their domain of expertise, projects have generally stronger identity than documents and can be recalled a long time after their completion, topics are not really changin g but are mainly getting richer) and that are better candidate for extensive capitalization. To conclude, we believe that this category of evaluation is particularly meaningful in a second stage once the ideas underlying an approach have been clarified and validated. Used too early, these methods bring the risk of distracting the attention to the investigation of the radically new knowledge management practiced that Ontology-based systems support. To be
Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 39 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public also frank, these methods also usually require a cons iderable amount of effort and resources that may better be allocated in an initial stage, to more qualitative mode of evaluation. 4.1.2.2 Evaluating according to the usage, perception and business value perspective The second approach is more oriented towards usage, cognitive perception and business value of the systems and the tools, and addresses the effectiveness of the approach (covering the level 3, 4 and 5 of our adapted Kirkpatrick model). The objective of the evaluation consists in this case in assessing to which level the new approach support new practices, that are more effective at the individual level (for instance by supporting higher level cognitive processes) and at an organizational level (for instance, by enabling the organisation to be more flexible, more continuously learning, more able to exploit previous experiences, and more creative). Practically, the nature of the evaluation is more qualitative, and relies on instruments such as questionnaire or experiments via scenarios (Giboin et. al., 2002) trying to elicitate needs, perception, and more effective usage patterns that could be better supported by a technical infrastructure, and offered to companies. 4.2 The approach used for evaluating Ontologging The Ontologging project possesses the following characteristics: • A domain relatively new, for which little previous work (theories, experience, etc.) is available. • The relatively immaturity of the technologies (the system is still far from an industrial product). • The limited size of the “population” conducting the test (it is difficult indeed to get a large sample of users for a system that is only in a prototypical stage). • The desire to evaluate the high-level usage value of the knowledge that is generated (and in particular, radically new usage patterns). Besides, it was decided to orient the focus of the evaluation towards the effectiveness of the approach (what is the substantive value delivered to the users?), rather than the efficiency of the technical infrastructure (are the tools “functioning” well?) that was being developed. As a consequence, it was decided to use a more qualitative form of evaluation. Questionnaires were therefore elaborated and their result analysed, focussed groups and experiments (cognitive walkthroughs) were conducted. Note:
Page 1 and 2: The ONTO-LOGGING Consortium consist
Page 3 and 4: Evaluation report of the use of Ont
Page 5 and 6: Executive Summary Evaluation report
Page 37: Evaluation report of the use of Ont
Page 53 and 54: • List of connected users to the
Page 67 and 68: 5.5.1 Description of the evaluation
Page 69 and 70: 6 Discussion and Conclusions Evalua
Page 75 and 76: 8 Annex Evaluation report of the us
Page 89 and 90:
Evaluation report of the use of Ont
Page 91 and 92:
Page 93 and 94:
esolver consultas. C42 Existen axio
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
1 The functionalities provided by D
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
show all

pdf 820Kb - INSEAD CALT

Create successful ePaper yourself

Delete template?

Save as template?