January 2012 Volume 15 Number 1 - Educational Technology ...

All the three groups completed the main study on economics which was divided into four phases which lasted about
two hours: pre-test phase, learning phase, teaching phase and post-test phase. In the pre-test phase, all participants
fulfilled the domain knowledge pre-test in supply and demand. Participants received more specific instructions from
researchers after the pre-test. In the learning phase, participants were allowed to study the text and diagrams in the
materials. They were encouraged to take notes, and were told that their notes and the text would be available during
the teaching phase. During the teaching phases, participants were working with the agent tutee system to teach what
they have learnt in the learning phase. Both the SP and GP groups were required to write down their reflection notes
in the dialog window to respond to the agent prompts to proceed with their teaching activities. Subsequently, all
participants completed the post-test domain knowledge assessment which was the same as the pre-test knowledge
assessment.
One week later, a transfer test was conducted on the same three groups of students on river ecosystems in ecology
(Leelawong, 2005). Transfer is the ability to generalize from the familiar to the less familiar, for example, it is the
ability to use problem-solving knowledge learnt in the context or domain of one task in another context or different
task (Bransford, Brown, and Cocking 2000). This transfer test was intended to investigate whether students had
experienced a transformative change to apply what they have learnt in the domain of economy to understand and
solve problems in the domain of ecology. All of them worked with the basic agent tutee without prompts. They
studied the materials in river ecosystems first and then began to teach their agent tutee. The river ecosystems include
common concepts such as fish, plants, bacteria, dissolved oxygen, carbon dioxide and algae. Each group was asked
to teach Betty about river ecosystems to maintain the balance of the natural system.
Measures
There were two categories of measures for the two hypotheses: 1) Learning Outcomes, and 2) Response Statements
to Agent Prompts.
Learning Outcomes
Three measures were used to study students’ learning outcomes as a result of the interventions: Improvements in
students’ scores from pre-test to the post-test; quality of the students’ concept maps as compared to expert maps;
participant’s ability to transfer.
Pre- and Post- Test Scores: The pre- and post- test administered to the students contains 14 questions that include
both multiple-choice and true-or-false questions. Question 1, 2, 3, 10 are multi-choice questions which are supposed
to check students’ factual knowledge about the domain. Question 4, 5, 6, 7, 8, 9 and 11 are multi-choice questions to
check students’ reasoning capacity in determining causal relationships among economic concepts about demand and
supply (when something goes up, something else goes down, etc. ). Question 12, 13 and 14 are true/false questions to
let students make judgments using economic knowledge in a market context of the real world. As to the scoring of
these 14 questions, one point is assigned to one question when students marked the correct choice and zero points,
otherwise.
Quality of Student Maps: We developed a systematic procedure which is adapted from a coding scheme developed
by Leelawong (2005) to evaluate the concepts and propositions in student maps which were grouped into two
categories: Expert and Relevant. Expert concepts or propositions are those also shown in the expert concept map
(Note that students never saw the expert map before). Relevant concepts or propositions are those found in the
domain materials or online resources but not a part of the expert map.
Ability to Transfer: In the transfer test, students learned the domain of river ecosystems (Leelawong, 2005) and
taught the basic version of Betty without the support of agent prompts. Students read the resources and extracted the
important concepts and their relations from the text to teach Betty. The measuring procedure of this transfer test is
mainly on the assessment of the quality of student maps.
344