Thinking and Deciding

EVALUATING PROBABILITY JUDGMENTS 119
say there is a 50% probability, and so forth. If my judgments are perfectly calibrated,
in the long run, these kinds of proportions will match exactly.
Note that my judgments can be coherent without being calibrated. For example,
I can say that the probability of heads is .90 and the probability of tails is .10. These
two are consistent with each other, but not with the facts. If my judgments are perfectly
calibrated, however, they must also be coherent, for whenever I say that the
probability of rain is 75%, I must also say that the probability of no rain is 25%.
Good calibration helps in making decisions. If the weather forecast is so poorly
calibrated that the probability of rain is really 1.00 when the forecast says .25, I will
get wet, if I regard .25 as low enough so as not to require an umbrella.
Calibration is a criterion for evaluation of probability judgments, not a method
for making them. Calibration serves as a criterion in hindsight, after we know what
happened. In principle, calibration could be assessed for judgments based on frequencies
or on the logical view, but this would be superfluous. If the assumptions
going into the judgments were correct, calibration would have to be perfect, and we
would know what the probability judgments were without asking a judge.
It may seem that calibration is related to the frequency view of probability. In
a way it is. Calibration certainly captures the idea that we want our probability
judgments to correspond to something about the real world, which is perhaps the
major idea behind the frequency view. By the personal view, however, a judgment
can be justifiable even before the frequency data are in. For the frequency view,
the judgment is meaningless unless the data are in. In addition, a personalist would
argue that calibration, though always desirable, is often impossible to assess. When
someone predicts the probability of a nuclear-power plant explosion, there are simply
not enough data to assess calibration. 3
Scoring rules
Coherence does not take into account what actually happens. A judge can make
completely coherent judgments and still get all of the probabilities exactly backward.
(When such a judge said .80, the observed probability was .20; when the judge said
1.00, the event never happened.) Moreover, the idea of calibration does not tell us
how to measure degree of miscalibration. A set of probability judgments is either
calibrated or not; however, degrees of error surely matter.
More important, calibration ignores the information provided by a judgment. A
weather forecaster’s predictions would be perfectly calibrated if they stated that the
probability of rain was .30 every day, provided that it does actually rain on 30% of
the days. Such a forecast would also be coherent if the forecaster added that the
probability of no rain was .70. These forecasts would be useless, however, because
they do not distinguish days when it rains from days when it does not rain.
3 It is an important empirical question whether we can assess a judge’s calibration in general,sothatwe
could draw conclusions about the judge’s calibration in one area from calibration in another. The answer
to this question, however, has nothing to do with the question of whether judgments are in principle
justifiable in the absence of frequency data.