January 2012 Volume 15 Number 1 - Educational Technology ...

In the nickname group, the student’s created identity was shown at the top of the field containing his or her generated
questions and comments (See Fig. 2). Students are free to change their nicknames to reflect their current state of
mind each time they construct or assess a new question. In the anonymity group, information on the question-author
or assessor was not shown, and only the word “anonymous” appeared at the top of the question and comment field.
(See Fig. 3).
Experimental procedures
Three intact classes were randomly assigned to different treatment conditions. Considering that true/false, fill-in-theblank
and multiple-choice questions are among the most frequently encountered question types in middle schools,
they were adopted for the study.
To ensure that participants possessed the fundamental skills needed for the generation and assessment of these types
of questions, a training session with hands-on activities was held at the beginning of the study, in addition to a lesson
on the operational procedures of QuARKS. One pamphlet containing (a) learning objectives, (b) QuARKS key
features and functions, (c) question generation criteria and sample questions, and (d) peer assessment criteria and
sample feedback/comments, was distributed for individual reference.
During each weekly online learning activity, students were first directed by the instructor to individually compose at
least one question of each type in accordance with the covered instructional contents. Each student then individually
assessed at least one question from the pool of peer-generated questions for each question type.
To establish a baseline regarding students’ perceptions of the different aspects of the activity, a real-identity mode
was used for all conditions during the first two sessions. Afterwards, students in different conditions used their
respective identity revelation modes. Students’ performance on the first biology exam was collected. A questionnaire
about the examined variables was disseminated for individual completion before different treatment conditions were
implemented in different groups, which started at the third week. After exposure to the activity for six weeks,
students completed the same questionnaire. Students’ performance on the second biology exam was then also
collected.
Measurements
The effects of different identity revelation modes on students’ academic performance were assessed by the first and
second biology exams of the participating school. Generally, the three exams spaced evenly throughout a 5-month
long semester (i.e., approximately six weeks separation between two exams) are arranged by schools and
administered at the same time for all students and all major subjects at high school levels in Taiwan. Item analyses
were conducted which ascertained that the test items correctly discriminated between high and low scores (average
0.61 and 0.44 for the first and second exam, respectively) and that the items as a whole were of moderate difficulty
(average 0.66 and 0.63 for the first and second exam, respectively).
The effects of different treatment conditions on students’ perceptions of various aspects of the engaged activity were
assessed using the same pre- and post-questionnaire, which consists of a set of four 5-point Likert scales (5=strongly
agree, 4=agree, 3=no opinion, 2=disagree, 1=strongly disagree). Existing instruments on related areas (i.e., peerassessment,
perceptions toward interacting parties, perceptions toward communication process and perceived
learning environment) were referred to, and items were adapted to fit the targeted experimental context. Specifically,
for “Attitudes toward Peer-Assessment Scale,” “Peer Assessment Questionnaire” (Brindley & Scoffield, 1998), “Peer
Assessment Questionnaire” (Wen & Tsai, 2006) and “Fairness of Grading Scale” (Ghaith, 2003) were referred to.
When constructing “Perception Toward Assessors Scale,” “Peer Assessment Rating Form” (Lurie, Nofziger,
Meldrum, Mooney & Epstein, 2006), ”The Questionnaire on Teacher Interaction” (Wubbels & Brekelmans, 2005),
“Student Perceptions of their Own Dyad” (Yu, 2001), “Student Perceptions of Other Dyads” (Yu, 2001) and
“Cooperative Learning Scale” (Ghaith, 2003) were used as a reference whereas “Student Evaluations of their
Experience of Assessment” (Stanier, 1997), “Student Perceptions of the Communication Process within the Dyad”
(Yu, 2001) “Student Perceptions of the Communication Process among the Dyads” (Yu, 2001), “Cooperative
Learning Scale” (Ghaith, 2003) and “Peer Assessment Rating Form” (Lurie, et al., 2006) were referred to when it
comes to the construction of “Perception toward the Interaction Process with Assessors Scale.” Finally, “Learning
68