Modeling and Multivariate Methods - SAS

Chapter 3 Fitting Standard Least Squares Models 75
Estimates
Parameter Power
Suppose that you want to know how likely it is that your experiment will detect some difference at a given
α-level. The probability of getting a significant test result is called the power. The power is a function of the
unknown parameter values tested, the sample size, and the unknown residual error variance.
Alternatively, suppose that you already did an experiment and the effect was not significant. You think that
the effect might have been significant if you had more data. How much more data do you need?
JMP offers the following calculations of statistical power and other details related to a given hypothesis test:
• LSV, the least significant value, is the value of some parameter or function of parameters that would
produce a certain p-value alpha.
• LSN, the least significant number, is the number of observations that would produce a specified p-value
alpha if the data has the same structure and estimates as the current sample.
• Power values are also available. See “The Power” on page 81 for details.
The LSV, LSN, and power values are important measuring sticks that should be available for all test
statistics, especially when the test statistics are not significant. If a result is not significant, you should at least
know how far from significant the result is in the space of the estimate (rather than in the probability). YOu
should also know how much additional data is needed to confirm significance for a given value of the
parameters.
Sometimes a novice confuses the role of the null hypotheses, thinking that failure to reject the null
hypothesis is equivalent to proving it. For this reason, it is recommended that the test be presented in these
other aspects (power and LSN) that show the test’s sensitivity. If an analysis shows no significant difference,
it is useful to know the smallest difference that the test is likely to detect (LSV).
The power details provided by JMP are for both prospective and retrospective power analyses. In the
planning stages of a study, a prospective analysis helps determine how large your sample size must be to
obtain a desired power in tests of hypothesis. During data analysis, however, a retrospective analysis helps
determine the power of hypothesis tests that have already been conducted.
Technical details for power, LSN, and LSV are covered in the section “Power Calculations” on page 683 in
the “Statistical Details” appendix.
Calculating retrospective power at the actual sample size and estimated effect size is somewhat
non-informative, even controversial [Hoenig and Heisey, 2001]. The calculation does not provide
additional information about the significance test, but rather shows the test in a different perspective.
However, we believe that many studies fail to detect a meaningful effect size due to insufficient sample size.
There should be an option to help guide for the next study, for specified effect sizes and sample sizes.
For more information, see John M. Hoenig and Dinnis M. Heisey, (2001) “The Abuse of Power: The
Pervasive Fallacy of Power Calculations for Data Analysis.”, American Statistician (v55 No 1, 19-24).