Statistical Methods in Medical Research 4ed

sample of permutations. If the null hypothesis is true, then the number of
permutations that will result in a statistic that is more extreme than the observed
value will have a binomial distribution with parameters NR and Pperm. Con-
sequently there is, approximately, a 95% probability that PR p
lies within
2 ‰fPperm…1 Pperm†g=NRŠ ofPperm.
As there is usually greater interest in smaller
significance levels, it is sensible to try to ensure that smaller values of Pperm are
estimated with greater absolute precision. Sensible values of NR might be found
by requiring the coefficient of variation of PR, which is
p
10.6Permutation and Monte Carlo tests 297
…1 Pperm†=…NRPperm†
, …10:14†
to be less than some prescribed value, such as 0 05 or 0 1. To obtain a coefficient
of variation of 0 1 when Pperm ˆ 0 5 requires NR ˆ 100, but this rises to
NR ˆ 1900 when Pperm ˆ 0 05. This is intuitively reasonable: many more simulations
will be required to determine, with given relative precision, the probability
of a rare event than the probability of a more common event.
Determining NR from (10 14) requires an estimate of Pperm: a conservative
approach would be to assume a small value and accept that if this is untrue some
unnecessary samples will be drawn. In many applications several thousand
permutations can be sampled very rapidly, so this is often acceptable. An alternative,
which can be very useful when evaluation of each statistic is timeconsuming,
is to adopt a sequential approach. The essential idea is that if the
observed statistic is not extreme then it is likely that a high proportion of the first
few random permutations will result in values of the statistic that are more
extreme than Gobs. For example, if 40 of the first 100 random permutations
gave more extreme statistics than the observed value, then it would be clear that
the test will not reject the null hypothesis. There would be no point in
continuing to sample permutations in order to obtain a more precise estimate
of a value which holds little further interest. On the other hand, if the observed
statistic is extreme, then it will be worthwhile continuing sampling until a
sufficiently precise estimate of Pperm is obtained. These informal ideas are put
on a firm foundation by Besag and Clifford (1991), who suggest that sampling
continues until h values of the statistic more extreme than the observed value
are obtained. If the number of samples needed to achieve this is l then the
estimate of Pperm is h=l. These authors suggest values for h of 10 or 20, although
smaller values could be used if the sampling and evaluations were particularly
onerous.
Monte Carlo significance levels are necessarily discrete: non-sequential values
being restricted to the set 0=NR,1=NR, ..., …NR 1†=NR, 1. However, for practical
purposes NR will generally be sufficiently large to ensure that this is of no
practical importance. An advantage of the sequential approach is that
significance levels obtained in this way are restricted to the set 1,