Deliverable 4.4 - INSEAD CALT

AtGentive IST-4-027529-STP - Attentive Agents for Collaborative Learners(Animation 5) includes the animation labelled as Happy-01, which had been alreadyidentified as problematic in phase 1. It was tested in the survey due to the limitednumber of alternatives available to choose from.One-Sample Test Test Value = 495% Confidence IntervalMean of the Differencet df Sig. (2-tailed) Difference Lower UpperAnimation1 * 3.236 99 .002 .490 .19 .79Animation2 19.221 99 .000 1.820 1.63 2.01Animation3 * 2.002 99 .048 .240 .00 .48Animation4 -4.700 99 .000 -.690 -.98 -.40Animation5 * 1.625 99 .107 .200 -.04 .44Animation6 -18.193 99 .000 -1.770 -1.96 -1.58Animation7 * -2.690 99 .008 -.370 -.64 -.10Animation8 -12.797 99 .000 -1.410 -1.63 -1.19Animation9 21.184 99 .000 2.230 2.02 2.44Animation10 5.989 99 .000 .640 .43 .85Animation11 8.867 99 .000 .790 .61 .97Animation12 18.211 99 .000 1.630 1.45 1.813.2.3 ConclusionTable 4. One-sample t-test results (test value =4)This survey study provided quantitative support to many of the findings evinced in thefirst phase of the evaluation. In particular, it indicated that the communication clarityof Colette’s non verbal language is far from perfect. However, the study allowedidentifying four stimuli (2 positive and 2 negative emotions) to be tested in the stroopexperiment. These stimuli are highlighted in Table 3 by a grey background. It has tobe noted that although 3 animations classified as suitable in the positive dimensions(high average), Animation 12 was not included in the experiment, as we could not finda third negative emotion which clearly conveyed its intended meaning. Furthermore,Animation 12 is quite similar to animation 9, and hence it was discarded.3.3 DiscussionThis in depth evaluation of the non verbal behaviour of Colette confirms our previousremarks on the difficulty of designing effective non verbal communication for virtualagents. It clearly emerged that often the designer stereotypes did not match the userones. Although we believe that Colette’s gestures could be strongly improved by therigorous application of theories and models of emotions, as well as by the strictapplication of a user-centred design approach, we must admit that Colette is aprototypic exemplar of the technological level of the embodied agents currently on themarket. The many limitations of current technology urge the systematic analysis ofpossible semantic conflicts between verbal and non verbal messages in human-agentinteraction. This state of the art supports the value of our research approach basedon the stroop task paradigm over a simple eye-tracking experiment: it is not onlyimportant to understand if a user look at the agent, but also if the information iselaborated.Del 4.4: AtGentive Final Evaluation Report – Appendix A page 13