pdf 820Kb - INSEAD CALT

More documents

Recommendations

Info

Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 46 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public in the case of the agent mechanism for Collaborative Ontology Authoring or the Ontology Knowledge Evaluator (OKE) that still appear more than ever as potentially valuable, but for which easier way of operationalising them would be desirable; (3) the usage patterns and expectation of an ontology system: some usage feedback was available to which kind of knowledge should be present in the system (users were expecting a rich variety of high quality content but that would be limited in quantity). 4.3.5 Experiments, via Scenarios Finally, a set of usage scenarios was identified (from current practices), further elaborated and tested to assess the capability of the Ontologging system to effectively support and improve some of the knowledge processes, and also to understand the usage patterns of a next generation of an Ontology-based knowledge management system. 4.4 Some directions towards a more quantitative (and rigorous?) evaluation 4.4.1 The reason of the qualitative evaluation As indicated previously, the Ontologging project has favoured the choice of a more qualitative evaluation versus a more quantitative evolution. This decision was actually not taken initially (a quantitative evaluation was even considered), but appeared later after observing the usage patterns of the end users. Indeed, contrary to the what was foreseen at the beginning of the project, the users did not see the Ontologging management system as a more powerful document management system that would allow to search more effectively an important amount of documents, but rather as a system able to capture the complex relationship between a relatively small numbers of a variety of objects (project, people, topics, and documents) intervening in knowledge management processes, and giving the possibility to the user to navigate this network of semantic relations. In this perspective, quantitative evaluation aiming at assessing the capability (quality) and the performance of the system at managing (manipulating, searching, retrieving, etc.) huge amount of data presented little interest since it was only covering usages (Information retrieval, and document orientation) that were not central for Ontologging. Does it mean that qualitative evaluation is meaningless? The answer is definitively no: as indicated previously, more quantitative evaluation should be useful at a later stage, once the different knowledge processes have been better understood, and a more precise evaluation is needed to assess how much the system is able to support these processes. 4.4.2 Some direction for a quantitative evaluation
Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 47 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public Concerning the quantitative evaluation of aspects that would be more semantic web oriented (semantic domain definition, semantic knowledge capture, and semantic navigation), much less previous experience is available (than evaluation accordin g to an Information Retrieval perspective). We can first indicate some work in this direction related to semantic disambiguation that was done by (Missikoff, Navigli and Velardi, 2002), and which measured the performance of different disambiguating heuristics, and more generally all the work on Ontology building, and ontology learning and population that has developed over recent years. However, the perspective of these approaches remains very technical, and is quite far from a more semantic web and cognitive vision that were adopted by the ontologging users and that we would like to promote in this project. A probably better perspective to explore is the domain Organizational Memory research (Abecker et al. 2003), which is more in line with higher-level knowledge management concepts, but for which evaluation still would need more elaboration (see for instance (Weinberger, Teeni and Frank 2003) for some work in this direction, for instance in evaluating the completeness of a manual knowledge population). Fina lly, an even more promising approach (but also far fetched) is the approach of knowledge management emphasizing cognitive and social factors (Thomas, Kellogg, and Erickson, 2001). Along this line is all the work related to social translucence, which rely on tools monitoring quantitatively the knowledge activity of a whole community. Obviously, the monitoring of the activity via the usage log represents a source of quantitative data that could easily be exploited (actually it is exploited by the Ontologging OKE) by quantitative evaluation, and in particular related to the (not only usage retrieval) usage of the system. 4.4.3 Some prospective operationalization of “SMART” quantitative evaluation Let’s now indicate some direction of how we could build some quantitative evaluation for the Ontologging project. In the illustration that will be given later will try to follow the SMART principle and be Specific, Measurable, Attainable, Relevant and Time-bound. 4.4.3.1 Ontology building The first quantitative evaluation could relate to Ontology building, and could assess the capacity of the Ontology approach and tools to support the elaboration of the domain Ontology. Different quantitative indicator could be used here: • The time to design or redesign the Ontology. How many hours, days, or weeks would be necessary to design the main Ontology, or some sub-ontology? • The complexity of the designed ontology. What is the level of complexity of the Ontology being elaborated (number of concepts, number of properties associated to this concept, level of nestledness / how need is the inheritance hierarchy). • Quality of the resulting Ontology (redundancy, ambiguity, etc.).
Page 1 and 2: The ONTO-LOGGING Consortium consist
Page 3 and 4: Evaluation report of the use of Ont
Page 5 and 6: Executive Summary Evaluation report
Page 45: Evaluation report of the use of Ont
Page 53 and 54: • List of connected users to the
Page 67 and 68: 5.5.1 Description of the evaluation
Page 69 and 70: 6 Discussion and Conclusions Evalua
Page 75 and 76: 8 Annex Evaluation report of the us
Page 93 and 94: esolver consultas. C42 Existen axio
Page 97 and 98:
Evaluation report of the use of Ont
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
1 The functionalities provided by D
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
show all

pdf 820Kb - INSEAD CALT

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?