pdf 820Kb - INSEAD CALT

Evaluation report of the use of Onto-Logging
platform in the user site
2 The Principles of the Evaluation
Deliverable ID: D8b
Page : 7 of 110
Version: 1.0
Date: 27 january 2004
Status: Final
Confid.: Public
2.1 Evaluation: Objective & Challenges
The evaluation of a research project always raises many questions and challenges.
A first set of questions is related to the objective of the evaluation. Is the evaluation
conducted to guaranty that the resources have been properly utilized for what they were
intended for (in particular when it concerns the spending of public money), or is the objective
of the evaluation to provide the participants an assessment and some feedback that will help
them to better pilot the project and in particular maximize the generation of value by this
project?
A second set of questions is related to the scope of the evaluation. Are we interested in
assessing the process of advancement of the project or in evaluating the quality of the results
that are generated by this project? Are we interested by evaluating the technical system (the
demonstrator) that is being designed, or by the approach that this system is expected to
validate?
Another set of questions concerns the dynamics of the execution of the project. What are the
consequences of the evaluation on the dynamics of advancement of a project? Indeed the
evaluation is rarely neutral and brings several secondary effects such as: an increase
transparency and a magnification of the importance of the elements that are being monitored;
a reduction of the flexibility and an increase risk-avoidance by the people whose activity is
being monitored.
An additional set of questions has to do we the operationalization of the evaluation: What is
the amount of resources that should to be dedicated to the evaluation of the project? How can
we evaluate the effort, and in particular decide how the evaluation resources have to be
allocated? How should we direct the effort (prioritization)? How do we deal with all the risks
associated with the evaluation, and in particular the resistance of people and organizations to
participate in an activity that consume their time, and may threaten their position?
Finally, a last set of question is related to the analysis of the result of the evaluation and the
use of the evaluation. How do we proceed to extract the maximum of this evaluation, identify
the most significant results and learn from them?
The answer to all these questions is difficult, and is well is beyond the scope of this
document. Indeed, if the main focus of a research project should be the maximization of the
effectiveness of the evaluation effort in the perspective of the value of the generated
knowledge (value for the end user; innovativeness of the solution; capability to exploit this
knowledge), a project very rarely provides the time to evaluate all the potential impact on the
society of the knowledge that has been created.
Besides, and as indicated, many factors can make the operationalization of the evaluation
delicate and difficult to achieve such as: cost and time factors (the evaluation do not come for
free, and usually takes time); or some less tangible factors such as the willingness of people
and organization to participate in the evaluation that they can consider as a thread, and the