Russel-Research-Method-in-Anthropology

Bivariate Analysis: Testing Relations 643
This is not perverse. All of us positivists out here know that it’s impossible
to absolutely, positively, prove any hypothesis to be forever unfalsifiably true.
So we do the next best thing. We try our very best to disprove our best ideas
(our research hypotheses) and hope that we fail, leaving us with the right to
say that our best guess is that we were right to begin with.
What this means in the real life of researchers is that statistical power is
the probability of avoiding both Type I and Type II errors: rejecting the null
hypothesis when it’s really true (Type I error) or accepting a null hypothesis
when it’s really false (Type II error). This probability depends on two things:
(1) the minimum size of the difference between two outcomes that you will
accept as a real difference and (2) the size of the sample. So, to achieve a
given amount of statistical power in any experiment or survey, you need to
calculate the size of the sample required, given the minimum size of the difference
between two outcomes—the effect size—that you will accept as a real
difference (Kraemer and Thiemann 1987; Cohen 1988).
This is a very important and subtle issue. Suppose you ask 100 men and
100 women, matched for socioeconomic status, race, and religion, to take the
Attitudes Toward Women Scale (AWS). The null hypothesis is that there is no
difference between the mean scores of the men and the mean scores of the
women on this scale. How big a difference do you need between the mean of
the men and the mean of the women on this scale to reject the null hypothesis
and conclude that, in fact, the difference is real—that men and women really
differ on their attitudes toward women as expressed in the AWS?
The answer depends on the power of the test of the difference in the means.
Suppose you analyze the difference between the two means with a t-test, and
suppose that the test is significant at the .05 level. Statistical power is the probability
that you are wrong to report this result as an indicator that you can
reject the null hypothesis.
The result, at the p .05 level, indicates that the difference you detected
between the mean for the men and the mean for the women would be expected
to occur by chance fewer than five times in 100 runs of the same experiment.
It does not indicate that you are 1 – p, or 95% confident that you have correctly
rejected the null hypothesis. The power of the finding of p .05 depends on
the size of the sample and on the size of the difference that you expected to
find before you did the study.
In the case of the AWS, there are 30 years of data available. These data
make it easy to say how big a difference you expect to find if the men and
women in your sample are really different in their responses to the AWS.
Many surveys, especially those done in foreign fieldwork, are done without
this kind of information available. You can offer a theory to explain the results
from one experiment or survey. But you can’t turn around and use those same