pdf 820Kb - INSEAD CALT

Evaluation report of the use of Onto-Logging
platform in the user site
Deliverable ID: D8b
Table of Contents
Page : 2 of 110
Version: 1.0
Date: 27 january 2004
Status: Final
Confid.: Public
1 Introduction ....................................................................................................................................6
2 The Principles of the Evaluation.....................................................................................................7
2.1 Evaluation: Objective & Challenges .........................................................................................7
2.2 Evaluating What? ....................................................................................................................8
2.2.1 Levels of evaluation ............................................................................................................8
2.2.2 The phases of the evaluation: formative, summative and “substantive value”.........................10
2.3 The Evaluation Methods & Tools ...........................................................................................12
2.3.1 Evaluation: an overview .....................................................................................................12
2.3.2 Observational methods ......................................................................................................13
2.3.3 User’s Opinions methods ...................................................................................................13
2.3.4 Interpretative methods .......................................................................................................14
2.3.5 Predictive methods ............................................................................................................14
2.3.6 Test, Experiment and Usability Engineering methods ..........................................................15
2.4 Defining the evaluation process ..............................................................................................16
2.4.1 The definition of the research questions ..............................................................................16
2.4.2 Selecting the evaluation methods (defining the instrument) ..................................................18
2.4.3 Executing the instrument and gathering the data ..................................................................19
2.4.4 Analysing the data, theory confirmation & building .............................................................19
3 Evaluating the Ontologging System ..............................................................................................21
3.1 Overview ..............................................................................................................................21
3.2 Goals and scope of the evaluation of Ontologging ...................................................................22
3.2.1 Knowledge Management & Ontologies, some clarification ..................................................22
3.2.2 The goals of Ontologging: the design of a Knowledge Management System that better
support some knowledge processes ..................................................................................................24
3.3 Selecting the elements to be evaluated (evaluation criteria) ......................................................25
3.3.1 Identifying the elements associated to each of the goals .......................................................25
3.3.2 Levels of evaluation for Ontologging. What could be evaluated............................................27
3.3.3 The different phases of the evaluation .................................................................................30
3.3.4 A pre-selection of the elements to be evaluated ...................................................................31
4 Gathering the Data.......................................................................................................................36
4.1 Previous work.......................................................................................................................36
4.1.1 The IR (Information Retrieval) approach ............................................................................36
4.1.2 The Ontology-based approaches .........................................................................................37
4.2 The approach used for evaluating Ontologging........................................................................39
4.3 The definition of the instruments and the gathering of the data .................................................40
4.3.1 The context of the evaluation .............................................................................................40
4.3.2 The Questionnaires............................................................................................................41
4.3.3 Focus groups.....................................................................................................................44
4.3.4 Interviews.........................................................................................................................45
4.3.5 Experiments, via Scenarios ................................................................................................46
4.4 Some directions towards a more quantitative (and rigorous?) evaluation...................................46
4.4.1 The reason of the qualitative evaluation ..............................................................................46