Laurie Bauer - WordPress.com — Get a Free Blog Here

169 STATISTICS
in twenty. So p < 0.05, while statistically significant, might seem a relatively low
level of significance for some things.
Accordingly, there are two other levels of significance which are regularly
taken to be meaningful: p < 0.01 (there is only one chance in one hundred that
things turned out the way they did purely by accident) and p < 0.001 (there is
only one chance in a thousand that things turned out the way they did purely
by accident).
So imagine our research project: we’ve collected two samples of people from
two subpopulations, measured the values of a particular vowel, found a large
difference between the samples and concluded that we have evidence against
the null hypothesis (by calculating a small p-value from some appropriate statistical
test). This small p-value can arise in two ways: either there is a real
difference and our experiment has detected it, or alternatively we were
unlucky, and chose a sample with too many atypical people. The p-value gives
us a measure of how unlucky we would have to be if the null hypothesis was
actually true, but we found samples that differed as strongly as the ones we just
collected.
Alternatively we might have found samples that looked very similar, and
found a large p-value from a statistical test. This lack of evidence against the
null hypothesis can also come from two sources: there might really be no
difference and our experiment correctly found none, or alternatively our
sample was too small to detect the difference (e.g. we selected only two people
from each suburb). So we should be careful. Finding no evidence for a
difference can sometimes mean we just have not looked hard enough yet.
In summary, the lower the p-value the greater your confidence in dismissing
the null hypothesis, and thus the greater your confidence in the hypothesis that
you are dealing with two distinct populations. In other words, low values for p
support the notion that the speakers from different suburbs or people of
different heights do not speak the same way.
Some final warnings
Apart from the importance of using the term significant only in its proper
meaning if you are discussing sets of figures like these, there are two things to
note about figures of this kind.
The first is that, if you have a significance level of p < 0.05, and it took you
twenty tests to come to this conclusion, the answer to one of them is wrong, but
you don’t know which it is. After all, if there is a one in twenty chance of getting
the answer wrong and you try twenty times, then one of those answers should
be wrong by your own statistics. Twenty tests is not very many, which is another
reason for being a bit careful with low levels of significance. It also means that
you should not carry out too many tests in the hope that something will turn