pdf 820Kb - INSEAD CALT

More documents

Recommendations

Info

Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 8 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public effect of the evaluation on the dynamic of exec ution of a project (with the bureaucratic syndrome). The approach we have decided to adopt in this document is to concentrate as much as possible on the dimension of the evaluation that is related to the substantive value (effectiveness) of the knowledge that is generated (in order work the effectiveness of the approach), and to support the other dimensions when this first dimension can not be adequately covered. 2.2 Evaluating What? 2.2.1 Levels of evaluation Donald Kirkpatrick, who started his work on learning evaluation in 1959, proposes four levels of evaluation of training programmes (Donald Kirkpatrick, 1996). Level 1, measures if the Learner liked the training (Did Learners like it?); Level 2 measures if the learning objectives have been met (Did Learners learn?); Level 3 measures the transfer of skills back to the job (Are they using it?); Level 4 measures the impact on the business (Did it matter?). We believe that this model of evaluation, which addresses both the efficiency and the effectiveness aspects of training programmes, can easily be transposed to the evaluation of systems in general, and in particular in systems that include a strong technical component (such as it is the case for the Ontologging system). We will therefore rely for the evaluation, to the four Kirkpatrick levels, plus two additional levels (technicalities & economics) that appear to be useful for evaluating technical systems: • Level 0: Technicalities. Do the approach/system perform technically well? • Level 1: Users’ acceptance: do they like the approach/system? • Level 2: Assessment: Do the management of knowledge approach/system function? • Level 3: Transfer: Is the approach/system used? • Level 4: Impact: Measures the impact in supporting the organizational processes (Does it deliver substantive value to the Organization?) • Level 5: Economics. Do the approach/system performs economically well? Level 0: Do the system performs technically well? This level reflects the performance of the system according to a technical perspective. It covers elements such as speed, scalability, reliability, (technical) flexibility, simplicity (capability to evolve) and openness (ability to interoperate with other systems). Level 1: Users’ acceptance (do they like it)? This level reflects the perception of the system by the user and is sometimes called the “smile sheet”. It is important to note that a good perception by the user do not guaranty that the system is useful (for instance, very nice looking graphics or gizmos will please the user, but will not contribute to his performance).
Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 9 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public However, the importance of user’s perception should not be underestimated, since, it contributes to the motivation of the user, which can be considered very important for the adoption any radically new system. The elements that can be evaluated at this level include the user -friendliness of the system, and the perceived value and limitation that the user gets from the system, or level of interactivity. The table 1 summarize the different levels of evaluation. Level 2: Assessment: Do the approach/system function? This level reflects the richness & the completeness of the functionalities of the system. In particular, it covers elements such as the ability to conduct a full session of the system, the number of functions available, and how deep the system support the user’s activities. Level 3: Transfer: Is the approach/system used? This level reflects the adoption of the system in an operational context (beyond the testing). In particular, it covers elements such as the adoption of the approach/system by an organization, and also clarifies the context of use. Indicators may include the number of installation effectively used in the organization, the number of activities supported for the organization, the features adopted by the use, and the rate of growth of new installation. Level 4: Impact: Measures the impact on the organization. This level reflects the substantive value of the approach/system for the organization, and for the people that use it. Indicators of impact include: improved work productivity, better work quality, faster result, and cognitive / behavioural transformation of the users leading to an organization better adapted to its environment. Level 5: Do the approach/system performs economically well? This level is concerned with the evaluation of the system according to a market perspective, and in particular its capability to be exploited in a competitive environment. Indicators include the cost of acquisition and the cost of ownership of the solution. It is important to remind that users organizations are ultimately driven by results & therefore should mainly be interested by level 4 (Impact on the Organization), and in particular to the answer to the following questions: (1) Is the proposed approach & system delivers substantive value to the organization that is translated into improved flexibility and better adequation to the company needs? (2) What are the key elements of this approach & system that contribute the most to the impact, and what is the nature of their contribution? However, Level 4 cannot easily be measured, and the answer to these questions can mainly be observed in the long run. The evaluation of Level 2 (assessment) provides a useful perspective: It reflects the level of functionalities delivered by the approach & system. However, this level can definitively not be considered sufficient since it only provide a partial picture that do not take into account the meaningfulness & usefulness of the approach and of the system. For instance, we can very well imagine systems (and actually, this applies to most approaches & systems which design
Page 1 and 2: The ONTO-LOGGING Consortium consist
Page 3 and 4: Evaluation report of the use of Ont
Page 5 and 6: Executive Summary Evaluation report
Page 7: Evaluation report of the use of Ont
Page 53 and 54: • List of connected users to the
Page 59 and 60:
Evaluation report of the use of Ont
Page 61 and 62:
Page 63 and 64:
Page 65 and 66:
Page 67 and 68:
5.5.1 Description of the evaluation
Page 69 and 70:
6 Discussion and Conclusions Evalua
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
8 Annex Evaluation report of the us
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
esolver consultas. C42 Existen axio
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
1 The functionalities provided by D
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
show all

pdf 820Kb - INSEAD CALT

Create successful ePaper yourself

Delete template?

Save as template?