pdf 820Kb - INSEAD CALT

More documents

Recommendations

Info

2.3 The Evaluation Methods & Tools 2.3.1 Evaluation: an overview Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 12 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public Evaluation is based on some observations, and of a model of interpretation. In quantitative research, these observation usually correspond to quantitative data that can be captured, and the model of interpretation correspond to a set of measures that consist in a me ans of associating numerical values to these data that provide some scale of comparison. In qualitative research (McBride and Schostak, 1995), these observations may take the form of the capture of events (such as “the user has done a specific action”) or the result of an interview with a user (.i.e. some more or less structured text or a video that record the content of the interview). The model of interpretation corresponds to method of analysis (the next chapter will present the main methods available). The quality of the sample used (characteristics and the size of the data) is also an element to take into account in the evaluation, since it has some important implications in the validity of the evaluation (for instance if the size of the observation is too small, or if the sample use for the observation is biased). Many works on this subject exist in quantitative research (and actually constitutes a major dimension of statistics) and to a less extent in qualitative research methodology. In the case of quantitative research, probability theories will be used to calculate a size of the sample that correspond to a given error margin. In qualitative evaluation, the size of the sample may be chosen more empirically in a way that is considered reasonable and significant enough to provide useful insight. Note: The size and on the characteristic of the sample used for the observation is a variable that should be manipulated with precaution when defining an evaluation since it is directly correlated to the ef fort needed to run the evaluation. Besides, the choice of the size and of the composition of the sample may be limited by real world constraints (for instance finding a large sample of motivated test users can represent an important difficulty). Many evaluation methods can be used to capture and analyse the data. Some methods (considered to be more objective) try to observe and evaluate the system in real operations, while other methods use more indirect proxy such as the collection of users’ perception and opinions. Each method has its advantages and disadvantages (related to the reliability, the level of effort required, the time, the complexity, etc.), and different methods should be used conjointly for the evaluation of a system. The next paragraph will present different categories of methods available.
Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 13 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public 2.3.2 Observational methods Observational methods involve real people using working systems. They are based on observing users interacting with the actual system and can range from being almost entirely informal to highly structured. Observational methods should try to be as invisible as possible to the user. Direct observation. Users are observed directly doing special or ordinary task in their work, while the observer makes notes about the their behavio ur and performance that she consider important to record. The evaluators do not have direct contact with the user. Indirect observation and software logging. Videoing a user interaction with a system allows recording several aspects of the user activity. Combined with some forms of automatic keystroke or interaction logging enable to collect vast amount of data. In some cases, software-logging tools can be used to record time-stamped key presses and real time recording of the interaction between users and technology. Besides, in many cases, information systems are usually able to track and monitor the activities of the users and to log them (for instance, that a user has posted a document, or has launched a search). Think aloud protocol. The data collected by think aloud protocol contains users´-spoken observation by addressing the cognitive activity and provide a subjective feeling about the activity they are performing. Users are asked to think aloud to describe what they are doing at each stage, and why. The evaluators record the users actions by using tape recordings, video, or pencil and paper. Note: Of course, the think aloud protocol may not be considered as totally neutral on the activity of the user since it introduce a cognitive load that can dist ract this user. 2.3.3 User’s Opinions methods The objective of the users’ opinion method is to elicitate the (subjective) perception of the system by the different categories of users. Structure interview. Interviews are particularly useful in eliciting information about user attitudes, impressions, and preferences. Interviews can be used in conjunction with thinkingaloud protocols for retrospective analysis of user responses and events. Questionnaires. Questionnaires have been used as instruments to assess different aspects of usability of the human-computer interface. Collecting users' subjective opinions about a system can remove unpopular and unusable parts early in the design or after delivery. Focus groups. Focus Groups are effective Research Tool and are very commonly used in software development. They are similar to interviews, but because we have a group of participants the group dynamics generate more data then one to one interviews. Focus groups involve gathering 8 - 12 members of a target audience for an open-ended discussion of
Page 1 and 2: The ONTO-LOGGING Consortium consist
Page 3 and 4: Evaluation report of the use of Ont
Page 5 and 6: Executive Summary Evaluation report
Page 11: Evaluation report of the use of Ont
Page 53 and 54: • List of connected users to the
Page 63 and 64:
Evaluation report of the use of Ont
Page 65 and 66:
Page 67 and 68:
5.5.1 Description of the evaluation
Page 69 and 70:
6 Discussion and Conclusions Evalua
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
8 Annex Evaluation report of the us
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
esolver consultas. C42 Existen axio
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
1 The functionalities provided by D
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
show all

pdf 820Kb - INSEAD CALT

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?