Contents

316 PART IV: Applied ResearchKey ConceptKey ConceptKey Conceptcourse to the second test. During the first test the students gain familiarity withthe testing procedure and with the instructor’s expectations. This familiaritythen affects their performance on the second test. Likewise, in the context of apsychology experiment in which more than one test is given (e.g., in a pretestposttestdesign), testing is a threat to internal validity if the effect of a treatmentcannot be separated from the effect of testing.Instrumentation Changes over time can take place not only in the participants ofan experiment (e.g., maturation or increased familiarity with testing), but also inthe instruments used to measure participants’ performance. This is most clearlya possibility when human observers are used to assess behavior. For instance,observer bias can result from fatigue, expectations, and other characteristics ofobservers. Unless controlled for, these changes in the observers represent aninstrumentation threat to internal validity by providing alternative explanationsfor differences in behavior between one observation period and another.Mechanical instruments also may change with repeated use. A researcher knownto the authors once found that a machine used to present material in a learningexperiment was not working the same at the end of the experiment as it wasat the beginning. Measures made near the end of the experiment differed fromthose made at the beginning of the experiment. Thus, what looked like a learningeffect was really just a change in the instrument used to measure learning.Regression Statistical regression is always a problem when individuals havebeen selected to participate in an experiment because of their “extreme” scores.Extreme scores on one test are not likely to be as extreme on a second test. Inother words, a very, very bad performance, or a very, very good performance(both of which we have all experienced), is likely to be followed by a performancethat is not quite so bad, or not quite so good, respectively. Consider, forinstance, your best ever performance on a classroom examination. What did ittake to “nail” this test? It took, no doubt, a lot of hard work. But it is also likelythat some luck was involved. Everything has to work just right to produce anextremely good performance. If we are talking about an exam, then it is likelythat the material tested was that which you just happened to study the hardest,or the test format was one you particularly like, or it came at a time whenyou were feeling particularly confident, or all of these and more. Particularlygood performances are “extreme” because they are inflated (over our usual ortypical performance) by chance. Similarly, an especially bad test performance islikely to have occurred because of some bad luck. When tested again (followingeither a very good or a very bad performance), it is simply not likely that chancefactors will “gang up” the same way to give us that super score or that verypoor score. We will likely see a performance closer to the average of our overallscores. This phenomenon frequently is called regression to the mean. Statisticalregression is more likely when a test or measure is unreliable. When an unreliabletest is used, we can expect scores to be inconsistent over time.Now, consider an attempt to raise the academic performance of a group of collegestudents who performed very poorly during their first semester of college(the “pretest”). Participants are selected because of their extreme per formance