Laurie Bauer - WordPress.com — Get a Free Blog Here

THE LINGUISTICS STUDENT’S HANDBOOK 168
to the same population, that each sample you have collected is a different
random sample from the same population. So the null hypothesis or the
sceptic’s position is that your two samples, though not identical, are not
different enough to be samples of different populations.
Statisticians have various tests available to them for deciding whether the
null hypothesis is or is not correct, and which test is appropriate depends
upon factors such as the kind of data you are using (e.g. whether it is measurements
on a physical scale or simple categorical answers to individual
questions – yes versus no, occupational types, and so on), and if you have a
physical scale whether the scale is extendable in both directions or just one.
As you read you will start to become familiar with the names of some of these
tests, such as Student’s t (one- or two-tailed t-tests), chi-squared (� 2 ),
Wilcoxon, etc. If you want to be well informed, you will check out when each
should be used. But if you assume that each has been appropriately used, you
will eventually come to a p-value, which is a statement of the probability that
your samples could have come from a population for which the null hypothesis
is true. A value of p will be given which will be between zero and one.
Large values close to p = 1 support the null hypothesis, low values close to p
= 0 indicate that the data set is inconsistent with the null hypothesis. This is
probably what you were hoping for when you asked the question. In practice
we never see p-values of exactly zero or one, but an answer near to zero is good
enough to conclude that there is a real difference between the two sets of
samples you have taken. But how close does it have to be to zero before you
can claim success?
If the probability is greater than 0.05, statisticians agree that it is not significant.
This is a technical usage of the word significant (though one based
on the ordinary language term). For any value greater than that, you cannot
assume that the null hypothesis is incompatible with the data from your
samples. Because significant is used in this very precise way in statistics, if you
do not mean it in its statistical sense you should avoid it in favour of some
synonym (important, indicative, suggestive, telling) when discussing results to
which you might in principle have applied a statistical test.
What does it mean if we say that p = 0.05, or, more likely p < 0.05 (‘p is less
than 0.05’)? What it means is that you have seen quite a large or quite a consistent
difference in the properties of the samples (e.g. the sample means), and
that there is only one chance in twenty that the two samples you took could be
samples from the same population and still look that different. If you think of
this in terms of betting money, those odds seem pretty good. If you put up £1,
and nineteen times out of twenty you win £1, and the other time you lose your
£1, you will be able to afford your own beer all evening. But if you think of it
in terms of something more serious, you would not like those odds. You would
not drive on a motorway if the odds of having a serious accident there were one