pdf 820Kb - INSEAD CALT

More documents

Recommendations

Info

Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 18 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public First, these elements should include the most important mechanisms that the system is providing to support in the domain addressed. For instance “the search function” will be considered as a critical function to be evaluated related to the Goal “providing support to knowledge identification”. Second (and perhaps even more importantly), these elements should include the features that are considered as unique for this system. For instance, if automatic clustering represents a unique feature of this system, it will have to be evaluated in dept, since it will represent the competitive advantage of this particular system when addressing the goal of “supporting knowledge identification”. Another element not to underestimate is that the choice of the priority also has consequences on the dynamics of the project. The evaluation if not neutral, and we can believe that the elements that be evaluated will receive more attention in the execution of the project. Besides, the evaluation may trigger some resistance both from people participating to this evaluation (they may see it as an additional effort, and consider the associated monitoring threatening their position) and from organizations (who may be reluctant to put some light on some of their current practices and systems they are currently using). 2.4.2 Selecting the evaluation methods (defining the instrument) The concept of population is crucial (what kind of data we are interested to collect). It defines the set of entities from which the research sample is to be drawn, controls variation and helps define the limits of generalizing the findings. Theory-building research relies on theoretical sampling Crafting Instruments and Protocols. Theory-building researchers typically combine multiple data collection methods. The choice of the evaluation methods to use and the elaboration of the action plan will depend of different factors such as: • The elements and indicators that have been previously identified. • The stage of system development • The resources available • The time available • The availability of potential users • Acceptability of intrusion on user • The type of output required • The precision / reliability desired • Etc.
Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b The action plan will consist in different phases: Page : 19 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public • The formative phase. Conducted mainly using interviews, mock-ups and early prototypes, the objective of the formative evaluation is to pr ovide an assessment that will be used to provide guidance for the design of the system. • The summative phase. This phase will focus more on assessing the functionalities of the system being designed (this phase will focus on the testing of the prototype, and will collect data via interviews, and observational methods). • The “substantive value”. This phase will focus on evaluating the value of the system. It will involve the users testing a consolidated version of the prototype, as well as experts and business development manager assessing the real value of the system (what are the unique features, how it compare with other systems) and its “economical” potential (ROI, business plan, etc.). 2.4.3 Executing the instrument and gathering the data For each phase will have to be defined the following steps: • Identification of the source of data for the evaluation (and in particular identifying the users’ groups participating in the evaluation and the experts). • Any preparation that would be necessary before the evaluation (for instance some evangelisation work towards the users, and their recruitment). • The setting of dates for the tests to be conducted negotiated with the users’ groups (date of sessions, starting date and duration in the case of continuous evaluation, etc.). • The setting of date for the analysis of the data (milestones). • The definition of how the results of the analysis are going to be fed-back to the system. Note: The evaluation is not necessary as linear as it may appear, but consist in a series of iterations: test, analysis, and integration of the findings. 2.4.4 Analysing the data, theory confirmation & building Analysing the data. This is the least codified part of the process. The overall idea is to become intimately familiar with the data collected. This process allows the unique patterns of each case to emerge before investigators push to generalize patterns across cases. Searching for Patterns. There are various tactics for doing that. The idea behind these tactics is to force investigators to go beyond initial impressions, especially through the use of structured and diverse lenses on the data. Shaping Hypotheses. Researchers constantly compare theory and data - iterating toward a theory, which closely fits the data. This stage includes sharpening the constructs by refining
Page 1 and 2: The ONTO-LOGGING Consortium consist
Page 3 and 4: Evaluation report of the use of Ont
Page 5 and 6: Executive Summary Evaluation report
Page 17: Evaluation report of the use of Ont
Page 53 and 54: • List of connected users to the
Page 67 and 68: 5.5.1 Description of the evaluation
Page 69 and 70:
6 Discussion and Conclusions Evalua
Page 71 and 72:
Evaluation report of the use of Ont
Page 73 and 74:
Page 75 and 76:
8 Annex Evaluation report of the us
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
esolver consultas. C42 Existen axio
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
1 The functionalities provided by D
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
show all

pdf 820Kb - INSEAD CALT

Create successful ePaper yourself

Delete template?

Save as template?