pdf 820Kb - INSEAD CALT

More documents

Recommendations

Info

2.4 Defining the evaluation process Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 16 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public For the different reasons we have already mentioned (cost, time, availability to find participants, etc.), evaluating everything is not possible, and to some certain extend it is not really desirable (lack of focus, introduction of rigidity, etc.). An evaluation process aiming at delimiting the research and conducting it realistically can be used. The different stages of this research process in social sciences are the following: • Stage 1: The definition of a research question. This involves contributing to building a body of knowledge and developing theory. • Stage 2: The development of an instrument. Having defined the research question, the research investigator needs to develop measurement instruments to capture the data for future analysis and to select the context (the site, the users, etc.) in which the measurement will take place. • Stage 3: The data gathering. This stage is related to the execution of the chosen instruments in the selected context in order to collect the data. This stage may include the execution of some preliminary actions, such as for instance trust building in the case that the data has to be collected from people. • Stage 4: The analysis of the data. This stage consists in the analysis of the data in the perspective of the research questions that have been previously identified, and the determination of the learning of the research. • Stage 5: The dissemination. This stage consists in the selection and the dissemination of the most important (originality, impact) research finding generated by this research. These different stages can be slightly relaxed in the Case Study Research method (Eisenhardt 1998; Yin 1994; Meredith 1998), a method favoured for conducting research in a field of research that is still largely unexplored, and which characterised by a paucity of theories, and the complexity and lack of well supported definitions and metrics (Stuart et al., 2002). For example, Eisenhardt (1989), while acknowledging the role of good research questions and theoretical constructs, argues that propositions can be developed (and tested) during data collection, rather than prior to it. Because the aim is to obtain a rich understanding of the cases in all their complexity, insights gained during data collection can be used to inform the theory. 2.4.1 The definition of the research questions An initial definition of the research question should be made in order not to become overwhelmed by the volume of the data. A priori specification of constructs can also help to shape the initial design of theory-building research, although this type of specification is not
Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 17 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public common in such studies. It is important to recognize that the early definitions of the research question and the possible constructs are only tentative. 2.4.1.1 Identifying the goals and the scope The identification of the most important points and objectives is very specific to the syste m & approach that is subject to the evaluation. Some of the critical points to identify are the main processes and dimensions that the system is expected to support. For instance, in context of learning systems, the processes and dimension may include “individual learning”, “experiential learning” or “collaborative learning” (via learning network). In the context of knowledge management, the selected processes and dimension are to be taken from the different knowledge processes used in organization such as “knowledge creation” (creativity), “knowledge acquisition”, “knowledge structuring”, “knowledge capitalization”, “knowledge searching”, knowledge sharing (and in particular the support of the social aspects), etc. These objectives and points should include both short-term objectives & long-term objectives, distinguish the points that are very general or very narrow and also be used to provide insights both for the design process itself (summative evaluation), for the functioning of the result (formative evaluation), and its effectiveness (substantive value of the solution). More horizontal & global characteristics may also be evaluated such as: • “Usability”: A system is usable if desired task quality can be achieved with acceptable costs to users. • “Flexibility”: A system is flexible if it is able to adapt with a reasonable level of effort, to changes that were not initially planned. • “Social acceptability”: A system is socially acceptable, if its introduction does not generate a too important level of resistance from the people due to a too radical change in their role. • Etc. Again, the choice of these characteristics is very dependant of the target characteristic of the system to be evaluated. For instance, “flexibility” may be considered of very little relevance in the application to the design of a satellite system for which the possibilities of intervention are very limited once the satellite has been launched. 2.4.1.2 Defining elements (criteria and indicators) for evaluating the goals. The evaluation of each goa l will rely on the definition a set of elements (criteria and indicator) that reflect the best how the different goals are fulfilled by the approach / system that is to be evaluated. The selection of the elements to be evaluated will take into account the relevance of these elements related to the importance of the goals and points that have been identified, as well as the level of detail desired or possible (affordable).
Page 1 and 2: The ONTO-LOGGING Consortium consist
Page 3 and 4: Evaluation report of the use of Ont
Page 5 and 6: Executive Summary Evaluation report
Page 15: Evaluation report of the use of Ont
Page 53 and 54: • List of connected users to the
Page 67 and 68:
5.5.1 Description of the evaluation
Page 69 and 70:
6 Discussion and Conclusions Evalua
Page 71 and 72:
Evaluation report of the use of Ont
Page 73 and 74:
Page 75 and 76:
8 Annex Evaluation report of the us
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
esolver consultas. C42 Existen axio
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
1 The functionalities provided by D
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
show all

pdf 820Kb - INSEAD CALT

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?