Deliverable 4.4 - INSEAD CALT

More documents

Recommendations

Info

AtGentive IST-4-027529-STP - Attentive Agents for Collaborative Learners• Control – shortcuts to general frequently-used pages (e.g. “News”)• Personal – shortcuts to frequently-used pages relating to the user (e.g. “Mymessages”)• News, highlight – features a recent news item (defined by the systemadministrator)• News, latest – lists the most recent news items• Personal – lists recent unread entries• Last visitors – lists visitors over the last six hours• Knowledge – lists the recent most popular postings• *Watch – Shows an overview of items (postings / people) that the user haspreviously selected to keep track of• *Agent On / Off – allows the user to request a number of general “interventions”,such as help in using the platform. These are delivered by the embodied agent• *Agent, Pending interventions – the main initial communication point betweenagent (not embodied agent) and user, detailing “interventions” (i.e. suggestions,such as specific postings the user may benefit from reading)Figure 4 - Screenshot of AtGentNet, showing the home page portlets1.5 Introduction to Summative EvaluationThe objective of the summative evaluation is to evaluate the success of the attentionrelatedmodifications to the collaborative e-learning platforms, as implemented in the twopilot systems. The summative evaluation described in this document evaluates theAtGentive interventions, the overarching aim of which is to assist collaborative learningthrough the direction of attention. This phase of AtGentive began at month 10(September 2006) with the actual pilots taking place between months 17 and 23 (April-October 2007).1.5.1 ApproachFor the summative evaluation, two concepts are considered, usability and usefulness.Usability measures the ease with which the user is able to pursue their activities usingthe system, while usefulness is the beneficial effect the system has on what the userDel <strong>4.4</strong>: AtGentive Final Evaluation Report page 6
AtGentive IST-4-027529-STP - Attentive Agents for Collaborative Learnersachieves. So for example, being able to obtain today’s date is useful, while being able tosee today’s date just by looking at the bottom-right of the screen is usable.In practice, the two concepts are usually linked. The more usable a system is, the moreuseful it is likely to be. For example, if the system proffers advice to the user, but thatadvice requires a difficult combination of button presses to access, the advice itself maybe less acceptable (and therefore less helpful) because of the context of frustration itsaccessing generated. Less acceptable advice may reduce the likelihood of its beingacted upon, and thence the usefulness of the system as a whole. Therefore, whileusability and usefulness are considered separately, the same measures may contributeto the evaluation of both concepts. (See Section 3.3 for further discussion.)The overall approach taken has been a combination of heuristic evaluation and fieldobservation and measurement. Heuristic evaluation generates rapid results which maybe used immediately to improve the product, with 3 evaluators being expected to findover 60% of usability problems (Nielsen & Mack, 1994). Laboratory and fieldexperiments, on the other hand, take much longer to set up and produce results thatoften require interpretation, but can access the more complex and hidden usabilityproblems. The two approaches work well together:‘By identifying obvious or clearcut usability problems, heuristic evaluation “harveststhe low-hanging fruit and provides a focus for laboratory testing” ’ (Kantner &Rosenbaum, 1997, pg. 2).Heuristic evaluation took place as soon as the software became available. Wherenecessary, feedback from this evaluation was used to improve the software prior to thenext stage. Field tests then examined the software in a naturalistic environment. Thisuser pilot provided a good insight into the main benefits, advantages, problems anddifficulties offered by the software. As a first pilot, these insights will be used to improvethe software further and ensure that any future comprehensive large-scale testing of thesoftware focuses on the main benefits and problems the software exhibits.1.5.2 MethodologyThe methodology employed for the user evaluation pilot has been to develop a set of keyindicators against which the performance of the software may be measured. These keyindicators are based upon the six scenarios used as a basis for the systems’ design(three per system). The key indicators were then expanded to define the means by whichthey may be measured for each platform.1.5.3 HeuristicsThe use of heuristics is a well established methodology in the area of usability evaluationof computer software. The procedure (Nielsen, 1993) consists of several evaluators whoseparately examine the software for problems, using predefined criteria (heuristics). Itoffers the ability to identify a large proportion of usability problems quickly and at lowcost. Further, the use of "Extreme Programming" as a development methodology withinAtGentive requires an evaluation methodology capable of offering rapid feedback as tothe quality, usability and relevance of each released software iteration. Heuristicevaluation offers this rapid feedback, especially at the early stages of development.By the use of appropriate heuristics, described later, this methodology has been adaptedfor use with the summative evaluation of the AtGentive system. Heuristic evaluation hasbeen used as a first-line test of the AtGentNet and AtGentSchool systems. By identifyingproblems at this stage, the effectiveness of the user evaluations was maximised. Mostimportantly, the use of heuristic evaluation allowed the separation of primary usabilityDel <strong>4.4</strong>: AtGentive Final Evaluation Report page 7
Page 1 and 2: ATGENTIVEIST-4-027529-STPDELIVERABL
Page 3 and 4: AtGentive IST-4-027529-STP - Attent
Page 11: AtGentive IST-4-027529-STP - Attent
Page 22 and 23: May 2007AtGentive IST-4-027529-STP
Page 60 and 61: CollaborationAtGentive IST-4-027529
Page 62 and 63:
AtGentive IST-4-027529-STP - Attent
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
Page 74 and 75:
Page 76 and 77:
Page 78 and 79:
Page 80 and 81:
Page 82 and 83:
Page 84 and 85:
Page 86 and 87:
Page 88 and 89:
DescriptionAtGentive IST-4-027529-S
Page 90 and 91:
Integratability:AtGentive IST-4-027
Page 92 and 93:
Page 94 and 95:
Page 96 and 97:
Page 98 and 99:
Page 100 and 101:
Page 102 and 103:
Page 104 and 105:
Page 106 and 107:
Page 108 and 109:
Page 110 and 111:
Page 112 and 113:
Page 114 and 115:
Page 116 and 117:
Page 118 and 119:
Page 120 and 121:
Page 122 and 123:
Page 124 and 125:
Page 126 and 127:
Page 128 and 129:
Page 130 and 131:
Page 132 and 133:
Page 134 and 135:
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Page 150 and 151:
Page 152 and 153:
Page 154 and 155:
Page 156 and 157:
Page 158 and 159:
Page 160 and 161:
Page 162 and 163:
Page 164 and 165:
Page 166 and 167:
Page 168 and 169:
Page 170 and 171:
Page 172 and 173:
The effects of a computer agent’s
Page 174 and 175:
Page 176 and 177:
Page 178 and 179:
Page 180 and 181:
4. ResultsAtGentive IST-4-027529-ST
Page 182 and 183:
Page 184 and 185:
Additional pedagogical investigatio
Page 186 and 187:
Page 188 and 189:
Page 190 and 191:
Page 192 and 193:
Page 194 and 195:
Page 196 and 197:
Page 198 and 199:
Page 200 and 201:
Page 202 and 203:
Page 204 and 205:
Page 206 and 207:
Page 208:
show all

Deliverable 4.4 - INSEAD CALT

Create successful ePaper yourself

Delete template?

Save as template?