Russel-Research-Method-in-Anthropology

646 Chapter 20
[(2120)/2](.05)10.5
correlations significant at the 0.05 level and
[(2120)/2](.01)2.1
correlations significant at the 0.01 level by chance. There are 73 correlations
significant at the 0.05 level in table 20.20, and 42 of those correlations are
significant at the 0.01 level.
Kunitz et al. examined these correlations and were struck by the strong
association of the hysterectomy rate to all the variables that appear to measure
acculturation. I’m struck by it, too. This interesting finding was not the result
of deduction and testing; it was the result of shotgunning. The finding is not
proof of anything, of course, but it sure seems like a strong clue to me. I’d
want to follow this up with research on how acculturation affects the kind
of medical care that women receive and whether all those hysterectomies are
necessary.
The Problem with the Shotgun Approach
The problem with shotgunning is that you might be fooled into thinking
that statistically significant correlations are also substantively significant. This
is a real danger, and it should not be minimized (Labovitz 1972). It results
from two problems.
1. You have to be very careful about choosing a statistical measure of association,
depending on how the variables were measured in the first place. A significant
correlation in a matrix may be an artifact of the statistical technique used and not
be of any substantive importance. Running a big correlation matrix of all your
variables may produce some statistically significant results that would be insignificant
if the proper test had been applied.
2. There is a known probability that any correlation in a matrix might be the result
of chance. The number of expected significant correlations in a matrix is equal
to the level of significance you choose, times the number of variables. If you are
looking for covariations that are significant at the 5% level, then you only need
20 tests of covariation to find one such covariation by chance. If you are looking
for covariations that are significant at the 1% level, you should expect to find
one, by chance, in every 100 tries. In a matrix of 100 variables with 4,950 correlations,
you might find around 50 significant correlations at the 1% level by
chance.
This does not mean that 50 correlations at the 1% level in such a matrix are
the result of chance. They just might be. There could be 100 or more signifi-