Fundamentals of epidemiology - an evolving text - Are you looking ...

That information gives you pause. Certainly you would not have been so impressed if he had told
you he could make a correct prediction in 30 tries. If the probability of a correct prediction (i.e., a
correct guess) in the absence of any special ability is 0.01, then the probability of one or more
correct guesses in 30 tries is 0.26 (1.0 minus the quantity 0.99 raised to the 30th power). Twenty-six
percent is still less than 50 percent, i.e., the probability of winning a coin flip, but not so
impressively. The evidence against the null hypothesis is now not nearly so strong. This change in
your interpretation illustrates the issue that arises in connection with multiple significance tests and
small studies bias.
It is possible, using statistical theory, to adjust significance levels and p-values to take into account
the fact that multiple independent significance tests have been done. But there are various practical
problems in applying such procedures, one of which is the lack of independence among multiple
tests in a particular set of data. For example, if your friend explained that he so rarely makes an
incorrect prediction that when he did he became so upset that it took him a whole hour (and 29
more predictions) to regain his predictive ability, then even if you remained skeptical you would be
hard-put to calculate an adjusted p-value for your test if you thought he was telling the truth.
Similarly, in a given dataset, does the fact that an investigator tested the same difference in various
ways (e.g., obesity as indexed by weight/height2 [Quetelet's index], weight/height3 [ponderal index],
percent above ideal weight, skinfold thickness, and body density) weaken the findings for each test?
If she also looked at blood pressure differences, would that weaken the credibility of statistical
significance of differences in obesity?
"You pays your money, and you takes your choice."
_____________________________________________________________________________________________
www.sph.unc.edu/courses/EPID 168, © Victor J. Schoenbach 14. Data analysis and interpretation – 497
rev. 11/8/1998, 10/26/1999, 12/26/1999