Clinical Trials

❘❙❚■ Chapter 23 | Subgroup AnalysisTable 2. The effect of multiple statistical tests on the overall Type I error rate.Number of tests with a 5% significance level Overall chance of a Type I error (%)1 5.002 9.753 14.264 18.555 22.6210 40.1320 64.15Secondly, when many subgroup analyses are undertaken in a clinical trial, thechance of finding a false-positive result (Type I error) by random chance aloneincreases if the significance level or threshold is kept the same (traditionally aP-value of 0.05; ie, a 1 in 20 chance). Table 2 shows results from a model usingtreatments of equal efficacy. As the number of subgroup analyses increases,so does the probability of finding at least one result meeting the 5% threshold,even though the treatments are of equal effect. For five subgroup analyses, theoverall chance of a Type I error inflates to 22.62% instead of the expected 5%.In other words, the odds of at least one false-positive result increases from 1 in 20to about 1 in 5, using a 0.05 level. Chapter 9 contains more on sample size and power.This method of multiple subgroup analyses has been termed ‘torturing the datauntil it confesses to something’, ‘a fishing expedition’ or ‘data dredging’ [3,4].Example 2A classic demonstration of a ‘fishing expedition’ was published by Yusuf andcolleagues, who looked at the results of the ISIS-2 (Second International Study ofInfarct Survival) trial [5,6]. The aim of this study was to determine whether the useof aspirin for patients after a heart attack was associated with a reduction in therisk of further heart attacks. While the overall study results found that aspirinreduced the risk of further heart attacks by 50%, which was a highly significantobservation (P < 0.0001), one subgroup analysis showed that aspirin was beneficialfor all astrological signs except Gemini and Libra. Such an association with birthsigns is likely to have arisen due to chance rather than have a plausible pathologicalassociation with the risk of heart disease.Patient imbalanceA further problem for subgroup analyses is that the balance created byrandomization might not be maintained in smaller subgroups. For example,while in the overall study men and women might be equally balanced by design,268