Contents

CHAPTER 12: Data Analysis and Interpretation: Part II. Tests of Statistical Significance and the Analysis Story 385NULL HYPOTHESIS SIGNIFICANCE TESTING (NHST)Key ConceptKey Concept• Null hypothesis testing is used to determine whether mean differencesamong groups in an experiment are greater than the differences that areexpected simply because of error variation.• The first step in null hypothesis testing is to assume that the groups do notdiffer—that is, that the independent variable did not have an effect (the nullhypothesis).• Probability theory is used to estimate the likelihood of the experiment’sob served outcome, assuming the null hypothesis is true.• A statistically significant outcome is one that has a small likelihood ofoc curring if the null hypothesis were true.• Because decisions about the outcome of an experiment are based on probabilities,Type I (rejecting a true null hypothesis) or Type II (failing to rejecta false null hypothesis) errors may occur.Statistical inference is both inductive and indirect. It is inductive because wedraw general conclusions about populations on the basis of the specific sampleswe test in our experiments, as we do when constructing confidence intervals.However, unlike the approach using confidence intervals, this form of statisticalinference is also indirect because it begins by assuming the null hypothesis.The null hypothesis (H0) is the assumption that the independent variable hashad no effect. Once we make this assumption, we can use probability theory todetermine the likelihood of obtaining this difference (or a larger difference) observedin our experiment IF the null hypothesis were true. If this likelihood issmall, we reject the null hypothesis and conclude that the independent variabledid have an effect on the dependent variable. Outcomes that lead us to reject thenull hypothesis are said to be statistically significant. A statistically significantoutcome means only that the difference we obtained in our experiment is largerthan would be expected if error variation alone (i.e., chance) were responsible forthe outcome (see Box 12.1).A statistically significant outcome is one that has only a small likelihood of occurringif the null hypothesis were true. But just how small is small enough? Althoughthere is no definitive answer to this important question, the consensus amongmembers of the scientific community is that outcomes associated with probabilitiesof less than 5 times out of 100 (or .05) if the null hypothesis were trueare judged to be statistically significant. The probability we elect to use to indicatean outcome is statistically significant is called the level of significance. Thelevel of significance is indicated by the Greek letter alpha (). Thus, we speak ofthe .05 level of significance, which we report as .05.Just what do our results tell us when they are statistically significant? Themost useful information we gain is that we know that something interestinghas happened. More specifically, we know that the smaller the exact probabilityof the observed outcome, the greater is the probability that an exact replicationwill produce a statistically significant finding. But we must be carefulwhat we mean by this statement. Researchers sometimes mistakenly say thatwhen a result occurs with p .05, “This outcome will be obtained 95/100 timesif the study is repeated.” This is simply not true. Simply achieving statistical