pdf 820Kb - INSEAD CALT

More documents

Recommendations

Info

Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Table of Contents Page : 2 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public 1 Introduction ....................................................................................................................................6 2 The Principles of the Evaluation.....................................................................................................7 2.1 Evaluation: Objective & Challenges .........................................................................................7 2.2 Evaluating What? ....................................................................................................................8 2.2.1 Levels of evaluation ............................................................................................................8 2.2.2 The phases of the evaluation: formative, summative and “substantive value”.........................10 2.3 The Evaluation Methods & Tools ...........................................................................................12 2.3.1 Evaluation: an overview .....................................................................................................12 2.3.2 Observational methods ......................................................................................................13 2.3.3 User’s Opinions methods ...................................................................................................13 2.3.4 Interpretative methods .......................................................................................................14 2.3.5 Predictive methods ............................................................................................................14 2.3.6 Test, Experiment and Usability Engineering methods ..........................................................15 2.4 Defining the evaluation process ..............................................................................................16 2.4.1 The definition of the research questions ..............................................................................16 2.4.2 Selecting the evaluation methods (defining the instrument) ..................................................18 2.4.3 Executing the instrument and gathering the data ..................................................................19 2.4.4 Analysing the data, theory confirmation & building .............................................................19 3 Evaluating the Ontologging System ..............................................................................................21 3.1 Overview ..............................................................................................................................21 3.2 Goals and scope of the evaluation of Ontologging ...................................................................22 3.2.1 Knowledge Management & Ontologies, some clarification ..................................................22 3.2.2 The goals of Ontologging: the design of a Knowledge Management System that better support some knowledge processes ..................................................................................................24 3.3 Selecting the elements to be evaluated (evaluation criteria) ......................................................25 3.3.1 Identifying the elements associated to each of the goals .......................................................25 3.3.2 Levels of evaluation for Ontologging. What could be evaluated............................................27 3.3.3 The different phases of the evaluation .................................................................................30 3.3.4 A pre-selection of the elements to be evaluated ...................................................................31 4 Gathering the Data.......................................................................................................................36 4.1 Previous work.......................................................................................................................36 4.1.1 The IR (Information Retrieval) approach ............................................................................36 4.1.2 The Ontology-based approaches .........................................................................................37 4.2 The approach used for evaluating Ontologging........................................................................39 4.3 The definition of the instruments and the gathering of the data .................................................40 4.3.1 The context of the evaluation .............................................................................................40 4.3.2 The Questionnaires............................................................................................................41 4.3.3 Focus groups.....................................................................................................................44 4.3.4 Interviews.........................................................................................................................45 4.3.5 Experiments, via Scenarios ................................................................................................46 4.4 Some directions towards a more quantitative (and rigorous?) evaluation...................................46 4.4.1 The reason of the qualitative evaluation ..............................................................................46
Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 3 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public 4.4.2 Some direction for a quantitative evaluation........................................................................46 4.4.3 Some prospective operationalization of “SMART” quantitative evaluation............................47 4.4.4 Concluding remarks on the quantitative evaluation ..............................................................49 5 Results & Analysis........................................................................................................................50 5.1 Phase 0: State of the situation (formative evaluation) ...............................................................50 5.1.1 Results..............................................................................................................................50 5.1.2 Analysis............................................................................................................................53 5.2 Phase 1: Ontology Building (and some content population) ......................................................54 5.2.1 Ontology building (at INDRA) ...........................................................................................54 5.2.2 The reengineering of the Ontology .....................................................................................57 5.2.3 Some lessons learned. ........................................................................................................60 5.3 Phases 3: Content population .................................................................................................60 5.4 Phases 4: Evaluating Ontologging “knowledge retrieval” .........................................................61 5.4.1 Evaluating of the basic knowledge retrieval.........................................................................62 5.4.2 Evaluating of the user-centred usages .................................................................................65 5.5 A Comparison with a more traditional knowledge management system (KnowNet) ...................66 5.5.1 Description of the evaluation..............................................................................................67 5.5.2 Comparing the two systems................................................................................................67 5.5.3 Lessons learned from this comparison.................................................................................67 5.6 Final words ...........................................................................................................................68 6 Discussion and Conclusions ..........................................................................................................69 7 References.....................................................................................................................................72 8 Annex...........................................................................................................................................75 8.1 Annex 1: Ontologging Goals & focus. The consortium perspective. ..........................................75 8.2 Annex 2: Description of INDRA (the main user group)............................................................76 8.2.1 An overview .....................................................................................................................76 8.2.2 Activities of Indra’s Competence Centres ...........................................................................77 8.2.3 The tendering process at INDRA........................................................................................80 8.2.4 References ........................................................................................................................81 8.3 Annex 3: Ten Usability Heuristics (Nielsen J. and Molich P.)...................................................81 8.4 Annex 4: Glossary of terms....................................................................................................82 8.5 Annex 5: Pre-questionnaire for the participants in the Ontologging usability test .......................83 8.6 Annex 6: Questionnaire Ontology-based approach for the structuring of knowledge ..................86 8.7 Annex 7 Spanish questionnaires .............................................................................................89 8.8 Annex 8 User modelling tools and knowledge distribution agents questionnaires .......................95 8.9 Annex 9 Ontologging project questionnaire........................................................................... 101
Page 1: The ONTO-LOGGING Consortium consist
Page 5 and 6: Executive Summary Evaluation report
Page 7 and 8: Evaluation report of the use of Ont
Page 53 and 54:
• List of connected users to the
Page 55 and 56:
Evaluation report of the use of Ont
Page 57 and 58:
Page 59 and 60:
Page 61 and 62:
Page 63 and 64:
Page 65 and 66:
Page 67 and 68:
5.5.1 Description of the evaluation
Page 69 and 70:
6 Discussion and Conclusions Evalua
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
8 Annex Evaluation report of the us
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
esolver consultas. C42 Existen axio
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
1 The functionalities provided by D
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
show all

pdf 820Kb - INSEAD CALT

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?