Statistical Methods in Medical Research 4ed

Significance tests
4.1 Statistical inference: tests and estimation 87
Data are often collected to answer specified questions, such as: (i) do workers in
a particular industry have reduced lung function compared with a control group?
or (ii) is a new treatment beneficial to those suffering from a certain disease
compared with the standard treatment? Such questions may be answered by
setting up a hypothesis and then using the data to test this hypothesis. It is
generally agreed that some caution should be exercised before claiming that some
effect, such as a reduced lung function or an improved cure rate, has been
established. The way to proceed is to set up a null hypothesis, that there is no
effect. So, in (ii) above the null hypothesis is that the new treatment and the
standard treatment are equally beneficial. Then an effect is claimed only if the
data are inconsistent with this null hypothesis; that is, they are unlikely to have
arisen if it were true.
The formal way of proceeding is one of the most important methods of
statistical inference, and is called a significance test. Suppose a series of observations
is selected randomly from a population and we are interested in a certain
null hypothesis that specifies values for one or more parameters of the population.
The question then arises: do the observations in the sample throw any light
on the plausibility of the hypothesis? Some samples will have certain features
which would be unlikely to arise if the null hypothesis were true; if such a sample
were observed, there would be reason to suspect that the null hypothesis was
untrue.
A very important question now is how we decide which sample values are
`likely' and which are `unlikely'. In most situations, any set of sample values is
peculiar in the sense that precisely the same values are unlikely ever to be chosen
again. A random sample of 5 from a normal distribution with mean zero and
unit variance might give the values (rounded to one decimal) 0 2, 1 1, 0 7, 0 8,
0 6. There is nothing very unusual about this set of values: its mean happens to
be zero, and its sample variance is somewhat less than unity. Yet precisely those
values are very unlikely to arise in any subsequent sample. But, if we did not
know the population mean, and our null hypothesis specified that it was zero, we
should have no reason at all for doubting its truth on the basis of this sample. On
the other hand, a sample comprising the values 2 2, 0 9, 2 7, 2 8, 1 4, the mean of
which is 2 0, would give strong reason for doubting the null hypothesis. The
reason for classifying the first sample as `likely' and the second as `unlikely' is
that the latter is proportionately very much more likely on an alternative
hypothesis that the population mean is greater than zero, and we should like
our test to be sensitive to possible departures from the null hypothesis of this
form.
The significance test is a rule for deciding whether any particular sample is in
the `likely' or `unlikely' class, or, more usefully, for assessing the strength of the