Thinking and Deciding

168 HYPOTHESIS TESTING
given any other hypothesis (∼H). If the critical result is found, the probability of H
increases considerably, because the diagnostic ratio is so high. 5
For Platt, a good experiment is one that seeks a result with a probability of one,
given one set of hypotheses, H1, and a probability of zero, given some other set H2;
H2 is the same as ∼ H1 (everything not in H1). If the result is found, we know that
the truth is in H1; if not, we know that the truth is in H2. Moreover, Platt specifies
that the sets should be chosen so that p(H2) and p(H1) are about .5. By this choice
of hypotheses to test, we can ask the fewest questions to determine exactly where the
truth lies — that is, in which subhypothesis of H1 or H2.
More generally, however, we should look for results in which the conditional
probabilities for results, given the important hypotheses, differ greatly. Whether we
seek results with probabilities of one or zero will depend to some extent on why we
need to know and how sure we need to be. We do not always need absolute certainty
in order to take some practical action or in order to accept some scientific theory as
very likely true, for the purposes of planning our next experiment. A good heuristic
to keep us on the trail of such results is this: “Be sure to consider alternative hypotheses.”
If our planned experiment or observation cannot distinguish the hypothesis of
interest from the alternatives, it is not a good experiment. If no experiment can do
so, our hypothesis is untestable, and we ought to re-examine our goal in formulating
it.
Sometimes, results that are predicted by a new hypothesis, but improbable otherwise,
are already known but not noticed before the new hypothesis is stated. For
example, astronomers before Copernicus knew that retrograde motion of planets (opposite
to their usual direction through the stars) occurred only when the planets were
high in the sky at midnight (hence opposite the sun). This fact was implied by Copernicus’s
theory. In Ptolemy’s theory, it was an unlikely coincidence. It was not taken
as evidence against Ptolemy’s theory, though, until a better alternative was found
(Lightman and Gingerich, 1991). According to probability theory, this is reasonable:
The fact that evidence is improbable given a hypothesis does not weaken the
hypothesis unless the evidence is more probable given some alternative.
The psychology of hypothesis testing
A traditional theory in psychology (Bruner, Goodnow, and Austin, 1956), now largely
discredited, holds that knowing a concept amounts to knowing how to classify some
instance as a member of a certain category. It was argued, moreover, that we classify
instances on the basis of cues. For example, in card games, knowing the concept
“spade” amounts to knowing how to classify cards as spades or nonspades. In this
5 Popper would disagree with this account, because he regards probability as a poor criterion of the
value of a theory. He argues that the “most probable” theory is always the least interesting, citing, as an
example, a noncommittal statement such as this: “The light will be bent some amount, or possibly not at
all.” He does not, however, consider a change in the probability of a hypothesis as a criterion of the value
of a result, which is what is argued here.