Deliverable 4.4 - INSEAD CALT

AtGentive IST-4-027529-STP - Attentive Agents for Collaborative LearnersFinally, perhaps the clearest statement “I really enjoyed the lesson” improved over timefor the Experimental group only. The first week or two was a difficult time for both groups,as they were not used to this form of software (where one takes instructions from thescreen, not the teacher). This finding suggests that provision of an active agent helpedstudents recover from these difficulties, making later lessons more enjoyable.Overall, then, we accept the Experimental hypothesis, that the Experimental group weremore satisfied than the Control group.LearningWhen evaluating educational technologies, “learning” is very difficult to measure, asmuch of the intended learning is not specifically factual, but rather at a general or a metacognitivelevel. For example, in comparing two countries, students may learn about themeaning of “compare” and of “contrast”, about how to assemble information, about howto communicate with an expert or with peers, and so forth. This learning is useful,indeed, potentially far more useful than learning facts about New Zealand. This moregeneral learning may, however, not be demonstrated by the students during the pilot. Forexample, a student may decide on a goal at the start of the pilot that is not suitable (andtherefore marked as a “bad” goal), but during the pilot may reflect on the goal and realisetheir mistake. Although they will have learned how to choose a better goal next time, thislearning is not visible during the relatively short pilot study.In order to assess learning within the confines of the pilot, a number of factual questionswere posed to the students. The results were not clear.The Experimental group scored BETTER on one pre-test question AND on one post-testquestion. They also scored LESS WELL on one post-test question. This does not provideevidence for rejecting H0 – the null hypothesis – and thus suggests that no evidence foran improvement in learning was found.Conversely, we must take into account that the hypotheses are two-tailed; that is, thatthey include the possibility that the Control group learned more than the Experimentalgroup. If the post-test had found only the significant result of the Control group scoringmore than the Experimental on one question then this conclusion may be indicated.However, the post-test in fact has one result for each possible direction, that is, in onequestion the Control group learned more than the Experimental group and in onequestion the Experimental group learned more than the Control group. The most likelyexplanation here is that the post-test is not a sufficiently accurate instrument to draw aconclusion from this result, which should be attributed to “noise” (extraneous factors). Forexample, students in each group sat together. This may have resulted in individualknowledge items that one student may say out-loud being overheard by others in thatgroup, leading to greater “knowledge” for that question without learning that item throughuse of the software.Overall, then, we retain the Null hypothesis, that there is no significant difference inlearning between the Experimental and Control groups.Finally, as described above, more subtle learning would be expected to have takenplace. In particular, since the Experimental group wrote papers that were, overall, ofsignificantly better quality than those of the Control group, it would be safe to speculatethat the Experimental group would be able to write better papers in future assignments,and thus that they have learned, overall, how to write better papers.Del 4.4: AtGentive Final Evaluation Report page 53