Clinical Trials

Clinical Trials: A Practical Guide ■❚❙❘Figure 1. The effect of multiple statistical tests on the overall Type I error rate. The chance of finding at leastone significant difference among different tests increases with the number of independent tests, even if there isno significant difference by design.0.8Overall chance of finding a significant result0.60.40.200 510 1520Number of tests with a 5% significance levelsingle test, there is more than a 20% chance that at least one of these tests willproduce a P-value ≤0.05. If 10 such tests are performed, the Type I error isincreased to about 0.40; and with 14 tests, the likelihood is higher than 0.50.The above results suggest that the use of multiple testing with a criticalsignificance level based on a single test (“Do any of the statistical tests reach thesignificance level of 0.05?”) is an inappropriate way of testing more than theoriginal null hypotheses. This suggests that results from the use of multiple testingwith a critical significance level based on a single test should be interpreted withcaution; approaches to this problem are discussed later in the chapter. Figure 1also highlights that using a statistical tool developed for a single defined questionneeds careful consideration when answers to more than one question are sought.In general, however, tests will tend to be correlated and the probabilities ofmaking at least one Type 1 error may be expected to be less than those shown inFigure 1 (see Figure 1 in Chapter 31 for simulation results).331