Deliverable 4.4 - INSEAD CALT

AtGentive IST-4-027529-STP - Attentive Agents for Collaborative Learnerssatisfaction (freedom from discomfort – physical and psychological – and with apositive attitude).3.4.2 Effectiveness, Efficiency, Satisfaction“Effectiveness” refers to the results the user is able to achieve by using the software.This is a measure of what may be achieved and the quality of the finished result, inrelation to what was expected. “Efficiency” refers to the user effort required to achievethe finished result. This is a measure of the process of achieving a result, the amount oftime and effort required of the user, the problems encountered along the way and thesoftware’s success in assisting the user. Together, “Effectiveness” and “Efficiency” aretaken here as indicators of “Performance”.“Satisfaction” is a subjective concept and as such it is difficult to use empirical measures.It is therefore usually measured by asking the user to estimate their satisfaction on aLikert-style rating scale, as well as asking satisfaction-related questions, for example, “Ifeel in control when using AtGentSchool”. Other, less subjective but none-the-lessindirect, measures may be used, such as “number of complaints”. There is also thepossibility of measuring some elements of satisfaction empirically. For example, howmuch software is used when use is voluntary?Finally, there is evidence to suggest that satisfaction and dissatisfaction are separatefactors and may not correlate (Jokela, Aikio, & Jounila, 2005). This is because“satisfaction” is built from a number of elements – the individual perceptions andexperiences of an artefact. For example, one may be satisfied with a cup of coffee due tobenefits of its caffeine content, yet dissatisfied with the same coffee because of itsflavour. Therefore questionnaires ask about both satisfaction and dissatisfaction.The following sections detail the forms of measures for each of the Key Indicators.“Satisfaction” and “Performance” incorporate elements from ISO 9241 as these are seenas Key Indicators that could be particularly affected by subtle usability issues. However,in accordance with the CIAO! Framework (see Section 3.3) the full range of system useis considered in a holistic manner, from usability and user activity, to user attitudes andoutcomes.3.5 HeuristicsThe purpose of using heuristics as a software evaluation tool, prior to testing with users,is to trap and remove a large proportion of usability problems before user testing begins,and in so doing to validate the software as being to appropriate usability standards forthe user pilot.The most-used heuristics for usability evaluation were developed primarily by JacobNielsen during the 1990s, with particular emphasis on the World-Wide Web (Nielsen &Mack, 1994). They were developed originally from an examination of usability problemsfound in one piece of software by a large number of “usability specialists” (Rolf & Jakob,1990). The list was later refined using a formal factor analysis of a larger number ofusability problems (Nielsen, 1994). For a full list of Nielson’s heuristics see Appendix 1.However the heuristics most appropriate to this study are:• Visibility of system status• Match between system and the real world• User control and freedom• Consistency and standards• Error preventionDel 4.4: AtGentive Final Evaluation Report page 23