pdf 820Kb - INSEAD CALT

Evaluation report of the use of Onto-Logging
platform in the user site
Deliverable ID: D8b
Page : 9 of 110
Version: 1.0
Date: 27 january 2004
Status: Final
Confid.: Public
However, the importance of user’s perception should not be underestimated, since, it
contributes to the motivation of the user, which can be considered very important for the
adoption any radically new system.
The elements that can be evaluated at this level include the user -friendliness of the system,
and the perceived value and limitation that the user gets from the system, or level of
interactivity. The table 1 summarize the different levels of evaluation.
Level 2: Assessment: Do the approach/system function? This level reflects the richness &
the completeness of the functionalities of the system. In particular, it covers elements such as
the ability to conduct a full session of the system, the number of functions available, and how
deep the system support the user’s activities.
Level 3: Transfer: Is the approach/system used? This level reflects the adoption of the
system in an operational context (beyond the testing). In particular, it covers elements such as
the adoption of the approach/system by an organization, and also clarifies the context of use.
Indicators may include the number of installation effectively used in the organization, the
number of activities supported for the organization, the features adopted by the use, and the
rate of growth of new installation.
Level 4: Impact: Measures the impact on the organization. This level reflects the
substantive value of the approach/system for the organization, and for the people that use it.
Indicators of impact include: improved work productivity, better work quality, faster result,
and cognitive / behavioural transformation of the users leading to an organization better
adapted to its environment.
Level 5: Do the approach/system performs economically well? This level is concerned
with the evaluation of the system according to a market perspective, and in particular its
capability to be exploited in a competitive environment. Indicators include the cost of
acquisition and the cost of ownership of the solution.
It is important to remind that users organizations are ultimately driven by results & therefore
should mainly be interested by level 4 (Impact on the Organization), and in particular to the
answer to the following questions: (1) Is the proposed approach & system delivers
substantive value to the organization that is translated into improved flexibility and better
adequation to the company needs? (2) What are the key elements of this approach & system
that contribute the most to the impact, and what is the nature of their contribution? However,
Level 4 cannot easily be measured, and the answer to these questions can mainly be observed
in the long run.
The evaluation of Level 2 (assessment) provides a useful perspective: It reflects the level of
functionalities delivered by the approach & system. However, this level can definitively not
be considered sufficient since it only provide a partial picture that do not take into account the
meaningfulness & usefulness of the approach and of the system. For instance, we can very
well imagine systems (and actually, this applies to most approaches & systems which design