Chapter 6

lesting o (honge99to test a:omplexld selectsection,:ompanyapplic-3s in this:lusivelypact of ammonly;ide benedin thelnd com-: change.ence thatexample; of a test: whether,7 39 4ra new instrument to obtain ce1l samples from patients would be an improvementover the existing instrument. One of the measures of quality that was used inthe evaluation was the percentage of inadequate samples. An inadequate samplewas one that did not contain enough cells to proceed with the diagnostic test.Data were obtained on the percentage of inadequate samples before and after theuse of the new instrument. Collecting data over time makes it possible to seewhether patterns indicating improvement coincide with the time of the change.In the run chart in Figure 6.3, the reduction in inadequate samples coincideswith the use of the new instrument.Since the basis of comparison in a before-and-after test is historical experience,the test is vulnerable to misinterpretation if a special cause unrelated tothe change occurs at or about the same time that the change is made. perhaps aseminar was given on a new way to use ihe old instrument while the new instrumentwas being introduced. It is up to those conductlng the test to make thejudgement that the effect seen is due to the change being tested. There is a rationalbasis for this judgement if there are no obvious external events and the systemhas been stable in the past. In the example just presented, the physiciansbelieved that the improvement in the percentage ofinadequate samples was theresult of the new instrument. They decided to plan another cycle to test the useof the instrument under a wider range of conditions.The data collected during a before-and-after test is susceptible to another specialcause, one that is difficult to detect: the effect of conducting the test in thefirst piace, sometimes known as the Hawthorne or Sentinel effect. During thetest, the people involved may be more careful or diligent in thelr work. This maybe the cause of the improvement rarher than the change being tested. Thls difficultycan be mitigated by testing over time so that the initial novelty of the tesrwears off. Also, ifit is possible and ethical, people might not be informed thatthey are part of a test. Open communication, however, is often the better alternative.If the Hawthorne effect is suspected, the data collected during the testcan be used to provide feedback to people. This feedback allows them to changewhat they are doing so that the improvement seen during the test can be sustained.In the example illustrated in Figure 6.3, it was possible to collect a sequenceofdata during the test. In some situations, it might be practical to collect dataonly once before and once after the change. For example, scores on a pretest andposttest might be used to evaluate the effect of new audiovisual materials in ahistory class, or a group of patients undergoing rehabilitation might be asked toevaluate the amount of pain they are feeling before and after a new exercise program.In this case, displaying on a histogram the data collected prior to and aftera test is a useful way to do the analysis.