January 2012 Volume 15 Number 1 - Educational Technology ...

each group. The experimental group received the seven topics of the physical science Recurrent On-Line
Synchronous Scientific Argumentation learning for one semester, with 25 class periods, each class lasting about 45
minutes. This was about one third of the physical science class periods over a semester. The teacher introduced the
five components of argumentation to the students and used it in the classroom for one class before the students
received the Recurrent On-Line Synchronous Scientific Argumentation learning and assessment. The conventional
group students went through the same content of physical science in traditional instruction and traditional laboratory
work.
All students were administered the two-tier Physical Science Conception test (PSCT) and the Physical Science
Dependent Argumentation Test (PSDAT) before and one week after learning. In addition, the experimental group
students’ on-line scientific argumentation process also was collected to determine the quality and quantity of
students’ argumentation and scientific conceptions they held before and after learning from the OLSA.
Instruments
Physical Science Conception test (PSCT)
The PSCT is a two-tier multiple choice diagnostic instrument that was developed to measure the degree of students’
conceptual change in physical science conceptions. The content validity was established by the same panel of six
evaluators, ensuring that the items were properly constructed and relevant to the seven topics of physical science
Web-learning materials that we developed. There are five items for each topic, and each item contains two tiers. In
the first tier, students are required to choose the correct scientific concepts, while in the second tier they choose the
correct reason for choosing these specific concepts. There are 35 items and each item has two tiers. Students need to
answer both tiers of each question correctly in order to receive one point, so the highest possible score is 35. The
Cronbach α of ADRT was 0.86 for the pre-test and 0.92 for the post-test.
Physical Science Dependent Argumentation Test (PSDAT)
The PSDAT is a two-tier multiple choice diagnostic instrument that was developed to measure the degree of
students’ argumentation ability involving physical science conceptions. There are five scenarios, covering five units
of seven topics. Each scenario includes the contextual background and argumentation discourses. There are five
questions under each scenario, for a total of 25 questions. Each question contains two tiers. The first tier of each
question requires the student to identify a specific statement from the argumentation discourses at scenario as a
correct data, claim, warrant, backing, or rebuttal, respectively, and justify why they chose that specific statement as a
correct data, claim, warrant, backing or rebuttal. The content validity was established by the same panel of six
evaluators, ensuring that the items were properly constructed and relevant to the five units of the OLSA physical
science learning program. There are 25 items covering five units. Students need to answer both tiers correctly in
order to receive one point, so the highest possible score is 25. The Cronbach α of PSDAT was 0.91 for the pre-test
and 0.92 for the post-test.
Qualitative Analysis of On-line scientific argumentation
The qualitative data collected from students’ on-line scientific argumentation was analyzed from two perspectives.
Each statement generated by an individual was classified into two different levels of claim, warrant, backing and
rebuttal, respectively. Data is considered to be non-argumentative statements. A level 1 claim is an argument
consisting of a claim without any data or fact. A level 2 claim is an argument consisting of a claim with data or fact.
A level 1 warrant is an argument consisting of a theory or principle without connection to the claim, or one which
does not clearly describe the theory. A level 2 warrant is an argument consisting of a claim with a clearly described
theory or principle. A level 1 backing is an argument only consisting of a backing without any connection to
claim/warrant, or one which does not clearly describe the connection among them. A level 2 backing is an argument
consisting of a claim with backing, and or with data or warrant. A level 1 rebuttal is an argument consisting of a
weak rebuttal without clear explanation. A level 2 rebuttal is an argument consisting of a claim with a clearly
identifiable rebuttal (Table 1). The cross-coder reliability is 0.91.
202