January 2012 Volume 15 Number 1 - Educational Technology ...

Analysis of Response Statements
An analysis was conducted to analyze participants’ response statements to the agent prompts both qualitatively and
quantitatively. The qualitative content of the response statements was examined and quantified to find out at which
levels of reflection have occurred in participants’ response statements. In the following subsections, we describe the
coding scheme of measuring respoinse statements, the coding process and inter-rater-reliability.
Coding Scheme of Measuring Response Statements
A tripartite coding scheme, adapted by Ortiz (2006) from the categorization scheme proposed by Surbeck, Han, &
Moyer (1991), was adopted to analyze the participant’ response statements to the agent prompts. This scheme, which
includes categories of reaction, elaboration, and contemplation, adopts the perspective that the nature of the stimulus
to reflect will impact the quality of reflection (Moore & Whitfield, 2008), which conforms to what this study seeks to
investigate.
In Table 2, the categories of response statements used in this study, tailored from the categorization scheme
mentioned above, are provided for comparison and to illustrate and judge participants’ elicited levels of reflection
revealed in the learning-by-teaching process.
Categories Description
Table 2. Categories of Response Statements
Sample Response statements
Reaction/ Reactive The entry reports facts, feelings, concern of [Respond to how to evaluate the performance
Statements an opinion. Course issues or concepts may of the agent tutee] “Good that you got the
be referred with no development or answer right, but your explanation isn’t very
elaboration.
good. ”
– Reaction: Report feelings
Elaboration/ The entry expands upon descriptive [Respond to the quest to make explanations to
Elaborative information by explaining feelings, or the agent tutee]
Statements giving illustrate example of course
“Economics fundamentally reflects on supply
concepts. The work uses interpretation of and demand. Supply is market can offer. It
course concepts
refers to producer supplying of goods.”
– Elaboration: Interpret course concepts
Contemplation/ The entry reference assumption about one’s [Respond to why to teach the agent tutee]
Contemplative self or one’s behaviour, indicating a shift in “To emphasize the importance of prior
Statements thought or attitude about one’s self. knowledge to succeed in future business”
– Contemplation: Indicate a shift in attitude
about oneself
Other Statements which are irrelevant or off-topic [Respond to what is the most thing trying to
teach the agent tutee] “Music Player. Old and
new modern”
Coding Process and Inter-Rater Reliability
Two raters were involved in the coding process. To begin, the first rater analyzed a sample of ten response
statements to become confident with the coding scheme. Then, the first rater explained the coding scheme, as shown
in Table 2, to the second rater. The second rater then analyzed the first ten response statements as well. The
consensus was achieved between the two raters to differentiate among the three categories: reaction, elaboration and
contemplation.
After that, the two raters analyzed the remaining response statements independently. Cohen’s kappa was used to
estimate the level of agreement between the two raters, taking the agreement occurring by chance into account.
Calculated with Kohen’s Kappa, agreement between the two raters appeared to be good, κ = 0.653 (A Cohen’s kappa
value between 0-0.4 is considered poor, 0.4-0.6 fair, 0.6-0.75 good, and 0.75 outstanding.).
345