Learning Statistics with R - A tutorial for psychology students and other beginners, 2018a

hardly unusual: in my experience, most practitioners express views very similar to Fisher’s. In essence,
the p ă .05 convention is assumed to represent a fairly stringent evidentiary standard.
Well, how true is that? One way to approach this question is to try to convert p-values to Bayes
factors, and see how the two compare. It’s not an easy thing to do because a p-value is a fundamentally
different kind of calculation to a Bayes factor, and they don’t measure the same thing. However, there
have been some attempts to work out the relationship between the two, and it’s somewhat surprising. For
example, Johnson (2013) presents a pretty compelling case that (for t-tests at least) the p ă .05 threshold
corresponds roughly to a Bayes factor of somewhere between 3:1 and 5:1 in favour of the alternative. If
that’s right, then Fisher’s claim is a bit of a stretch. Let’s suppose that the null hypothesis is true about
half the time (i.e., the prior probability of H 0 is 0.5), and we use those numbers to work out the posterior
probability of the null hypothesis given that it has been rejected at p ă .05. Using the data from Johnson
(2013), we see that if you reject the null at p ă .05, you’ll be correct about 80% of the time. I don’t
know about you, but in my opinion an evidentiary standard that ensures you’ll be wrong on 20% of your
decisions isn’t good enough. The fact remains that, quite contrary to Fisher’s claim, if you reject at
p ă .05 you shall quite often go astray. It’s not a very stringent evidentiary threshold at all.
17.3.3 The p-value is a lie.
The cake is a lie.
The cake is a lie.
The cake is a lie.
The cake is a lie.
– Portal 10
Okay, at this point you might be thinking that the real problem is not with orthodox statistics, just
the p ă .05 standard. In one sense, that’s true. The recommendation that Johnson (2013) gives is not
that “everyone must be a Bayesian now”. Instead, the suggestion is that it would be wiser to shift the
conventional standard to something like a p ă .01 level. That’s not an unreasonable view to take, but in
my view the problem is a little more severe than that. In my opinion, there’s a fairly big problem built
into the way most (but not all) orthodox hypothesis tests are constructed. They are grossly naive about
how humans actually do research, and because of this most p-values are wrong.
Sounds like an absurd claim, right? Well, consider the following scenario. You’ve come up with a
really exciting research hypothesis and you design a study to test it. You’re very diligent, so you run
a power analysis to work out what your sample size should be, and you run the study. You run your
hypothesis test and out pops a p-value of 0.072. Really bloody annoying, right?
What should you do? Here are some possibilities:
1. You conclude that there is no effect, and try to publish it as a null result
2. You guess that there might be an effect, and try to publish it as a “borderline significant” result
3. You give up and try a new study
4. You collect some more data to see if the p value goes up or (preferably!) drops below the “magic”
criterion of p ă .05
Which would you choose? Before reading any further, I urge you to take some time to think about it.
Be honest with yourself. But don’t stress about it too much, because you’re screwed no matter what
you choose. Based on my own experiences as an author, reviewer and editor, as well as stories I’ve heard
from others, here’s what will happen in each case:
10 http://knowyourmeme.com/memes/the-cake-is-a-lie
- 564 -