Domain Testing: Divide and Conquer - Testing Education

Chapter 5: Experiment Design
5.01 Overview of the Experiment
After designing and developing training material for domain testing, I conducted
two rounds of pilot studies. These were basically formative evaluations that lead to
immense revision and refinement of the training material content, my instruction
and the way I was organizing the training material. Once I felt the training material
was polished enough to be used in the real experiment, I conducted actual training
sessions using the revised training material with 23 learners.
I started with 18 learners in my actual experiment. Each training period
lasted for five days, starting on a Monday and ending on a Friday. The entire
training period totaled 18 hours. The learners took a paper-based pretest on the first
day following a brief introductory lecture on domain testing. The pretest was open
book and the learners were given reference materials to consult during the test.
On the second, third and fourth training days, the learners were subject to
lectures that involved demonstration of examples that addressed one or more
instructional objectives. They were then required to solve exercises based on what
they learned in the lecture. The strategy was to alternate between examples and
exercises.
The learners could refer to the reference materials and lecture slides to solve
exercises. However, they were not allowed to take the reference materials home.
This helped control the amount of time subjects actually spent on the material
during the week. We didn't want the variability of some students working for
several hours per night on the material while others worked only in class. We also
wanted to reduce the chances of learners scheduled to attend future training
sessions being exposed to the training material, which could influence those future
learners’ performance.
On the fifth and final training day, the learners were required to take two
posttests. One was a paper-based test which was equivalent to the pretest, and the
69