Learning Statistics with R - A tutorial for psychology students and other beginners, 2018a

Sampling Distribution for X if θ=.55
lower critical region
(2.5% of the distribution)
upper critical region
(2.5% of the distribution)
0 20 40 60 80 100
Number of Correct Responses (X)
Figure 11.4: Sampling distribution under the alternative hypothesis, for a population parameter value of
θ “ 0.55. A reasonable proportion of the distribution lies in the rejection region.
.......................................................................................................
11.8
Effect size, sample size and power
In previous sections I’ve emphasised the fact that the major design principle behind statistical hypothesis
testing is that we try to control our Type I error rate. When we fix α “ .05 we are attempting to ensure
that only 5% of true null hypotheses are incorrectly rejected. However, this doesn’t mean that we don’t
care about Type II errors. In fact, from the researcher’s perspective, the error of failing to reject the
null when it is actually false is an extremely annoying one. With that in mind, a secondary goal of
hypothesis testing is to try to minimise β, the Type II error rate, although we don’t usually talk in terms
of minimising Type II errors. Instead, we talk about maximising the power of the test. Since power is
defined as 1 ´ β, this is the same thing.
11.8.1 The power function
Let’s take a moment to think about what a Type II error actually is. A Type II error occurs when the
alternative hypothesis is true, but we are nevertheless unable to reject the null hypothesis. Ideally, we’d
be able to calculate a single number β that tells us the Type II error rate, in the same way that we can
set α “ .05 for the Type I error rate. Unfortunately, this is a lot trickier to do. To see this, notice that in
my ESP study the alternative hypothesis actually corresponds to lots of possible values of θ. In fact, the
alternative hypothesis corresponds to every value of θ except 0.5. Let’s suppose that the true probability
of someone choosing the correct response is 55% (i.e., θ “ .55). If so, then the true sampling distribution
for X is not the same one that the null hypothesis predicts: the most likely value for X is now 55 out
of 100. Not only that, the whole sampling distribution has now shifted, as shown in Figure 11.4. The
critical regions, of course, do not change: by definition, the critical regions are based on what the null
- 341 -