Deliverable 4.4 - INSEAD CALT

AtGentive IST-4-027529-STP - Attentive Agents for Collaborative Learners4. Summative Evaluation – Results4.1 Heuristic Evaluations4.1.1 AtGentSchoolThe full results of the heuristic evaluation are included in Appendix 4. In total, 39suggestions were made for improvements. A summary is given below. It was notconsidered that any of the issues found would prevent the successful conduct of the pilotstudy.• Possible problems with parts of the system available prior to login• Slow response when clicking on the “smileys”• Difficulty with navigation outside of the home screen• Improvements suggested for the agent’s behaviour4.1.2 AtGentNetThe full results of the heuristic evaluation are included in Appendix 5. In total, 77suggestions were made for improvements. A summary is given below. Some changeswere deemed necessary and carried out to the software during the early stages of thepilot.• A number of important usability improvements needed for the interface• General lack of transparency between screen options and system function• A number of words and concepts of a technical nature – may cause problems forthe general user• The agent is not enough proactive, and takes no action without a direct userrequest4.2 Pilot study – AtGentSchoolData collected as part of the AtGentSchool pilot, and available for analysis, is as follows:• assessment of students’ work• log files• questionnaire responseso students pre-trial testo students feedbacko students post-trial testo teachers feedback• teachers’ diaries• post-trial workshop4.2.1 Teachers’ experience4.2.1.1. Teacher QuestionnairesTeachers were asked to complete an attitude survey as soon after the studentscompleted theirs as practicable. This provides an interesting insight into the practicalitiesof using AtGentSchool in a classroom setting. Two questionnaires were completed; thefirst soon after the pilot began, the second towards the end. (The exact dates ofDel 4.4: AtGentive Final Evaluation Report page 28