Contents

388 PART V: Analyzing and Reporting Researchalso reminds us that statistical inference can never replace replication as the best testof the reliability of an experimental outcome.EXPERIMENTAL SENSITIVITY AND STATISTICAL POWER• Sensitivity refers to the likelihood that an experiment will detect the effectof an independent variable when, in fact, the independent variable trulyhas an effect.• Power refers to the likelihood that a statistical test will allow researchers toreject correctly the null hypothesis of no group differences.• The power of statistical tests is influenced by the level of statistical significance,the size of the treatment effect, and the sample size.• The primary way for researchers to increase statistical power is to increasesample size.• Repeated measures designs are likely to be more sensitive and to have morestatistical power than independent groups designs because estimates oferror variation are likely to be smaller in repeated measures designs.• Type II errors are more common in psychological research using NHSTthan are Type I errors.• When results are not statistically significant (i.e., p .05), it is incorrect toconclude that the null hypothesis is true.Key ConceptThe sensitivity of an experiment is the likelihood that it will detect an effectof the independent variable if the independent variable does, indeed, have aneffect (see Chapter 7). An experiment is said to have sensitivity; a statisticaltest is said to have power. The power of a statistical test is the probability thatthe null hypothesis will be rejected when it is false. The null hypothesis is thehypothesis of “no difference” and, thus, is false and should be rejected whenthe independent variable has made a difference. Recall that we defined a TypeII error as the probability of failing to reject the null hypothesis when it is false.Power can also be defined as 1 minus the probability of a Type II error.Power tells us how likely we are to “see” an effect that is there and is anestimate of the study’s replicability. Because power tells us the probability ofrejecting a false null hypothesis, we know how likely we are to miss a real effect.For instance, if a result is not significant and power is only .30, we know that astudy with these characteristics detects an effect equal to the size we observedonly 3 out of 10 times. Therefore, 7 of 10 times we do this study we will missseeing the effect. In this case we may want to suspend judgment until the studycan be redone with greater power.The power of a statistical test is determined by the interplay of three factors:the level of statistical significance, the size of the treatment effect, and thesample size (Keppel, 1991). For all practical purposes, however, sample size is theprimary factor that researchers use to control power. The differences in sample sizethat are needed to detect effects of different sizes can be dramatic. For example,Cohen (1988) reports the sample sizes needed for an independent groups designexperiment with one independent variable manipulated at three levels. Ittakes a sample size of 30 to detect a large treatment effect; it takes a sample size