Deliverable 4.4 - INSEAD CALT

AtGentive IST-4-027529-STP - Attentive Agents for Collaborative LearnersExecutive SummaryThis document describes the summative and strategic evaluations for the AtGentiveproject. The project consists of the development of attention-aware versions of theOntdeknet eLearning platform for children aged 8-15 (to produce AtGentSchool) andINSEAD’s collaborative learning platform for adults (to produce AtGentNet).Summative evaluation began with usability testing of the software, using expertconductedheuristic evaluations, to ensure a suitable level of usability. The purpose ofusing heuristics as a software evaluation tool is to validate the software as being toappropriate usability standards separate to its effects on the users. The primary phaseof summative evaluation comprises an extended period of direct user testing.AtGentSchool was tested over a six week period by classes in five schools in theCzech Republic (one or two classes per school). The AtGentSchool platform createsvirtual learning relationships between subject experts and students, and providesguidance to support individuals to learn together based upon common interests.AtGentNet was tested over a five month period by business people, across Europeand beyond, enrolled in a business-related distance learning course organised by theSwedish Trade Council (STC). Learners attend seminars and meet with local tutorswho provide expert advice and assistance to help learners set and achieve practicalgoals. The process is supported by the AtGentNet platform which provides bothaccess to information and a social forum.The criteria for success were based upon three scenarios, implemented for eachsystem (AtGentSchool and AtGentNet). These scenarios relate to the guidance oflearning, the (re)direction of attention and user-notification. Research questions weregenerated from these scenarios, leading to five key indicators: Attention, Performance,Satisfaction, Learning and Collaboration. These indicators are used as part of thesummative evaluation to assess the benefits gained by incorporating attentive agents.In terms of overall success, the data indicate that use of an animated agent for childrencan successfully promote Performance, Satisfaction and Collaboration, with noapparent detriment to other factors. The use of scenarios as design elements provedvery effective in designing for children. For adults, careful and subtle perceptualenhancements appear to be a better approach than animated agents. The data hereindicate that performance and collaboration may be enhanced, but only where themotivation pre-exists. Perceptual-based attention support does not engendermotivation, compared to the motivation provided to children by the animated agent inAtGentSchoolStrategic evaluation examines the real contribution of AtGentive in the outside world.The primary project outputs are identified, along with their strengths and weaknesses.Overall, the project approach of attacking the problem from several perspectivescorresponding to the strengths and expertise of individual participants has proved veryfruitful. AtGentive has met its principle objectives to successfully design and run twopilot studies which advance the support and understanding of attention with regard toeducational software. While the support is situated in particular applications,knowledge gained may have implications in a wider context. This work has contributedacross several disciplines in the research community – teaching and learning,collaborative systems, human- computer interface, and intelligent agents to mentionbut a few – and offers a basis for continuing research in this new and expanding area.Del 4.4: AtGentive Final Evaluation Report