Thinking and Deciding

ACCURACY OF PROBABILITY JUDGMENTS 143
Is it humanly possible to be well calibrated? Do errors result of an inherent
limitation on our accuracy? We do not know the answer to this in general. We do
know the answer for expert judges who make probability judgments every day as
part of their work. The answer is that such people can be amazingly well calibrated.
Murphy and Winkler (1977) studied the calibration of some 25,000 weather forecasts
made in the Chicago area over a four-year period ending in 1976. Since 1965,
weather forecasts have included probability information for rain, snow, and other
conditions. Weather forecasters use a variety of information, including statistical
tables of events of past years, but they do make their predictions on the basis of personal
judgment; weather forecasts are made by people, not computers. Murphy and
Winkler found that calibration was nearly perfect. When the forecasters said that
there was a 100% chance of rain, it rained about 98% of the time, and so on down
the line.
This does not mean that the forecasts were always very informative. Again, if
you know that it rains on one out of every ten days, your forecasts can be perfectly
calibrated if you give .10 as the probability of rain, every day. Of course, this is not
what the forecasters did.
Most important, this result shows that inappropriate extreme overconfidence can
be overcome. Whatever their accuracy in predicting the weather itself, weather forecasters
are extremely accurate in assessing the confidence that should be placed in
their own predictions. Quite possibly they learn this from years of feedback.
Another cause for the phenomenon of “overconfidence when confidence is high”
(aside from regression to the mean) is a bias that I have emphasized earlier: the tendency
to seek evidence in favor of an initial belief, as opposed to evidence against
it. Koriat, Lichtenstein, and Fischhoff (1980) gave subjects the same sort of twoalternative
questions used in the studies described earlier and asked for confidence
judgments. Some subjects were asked to give reasons for and against their favored
answer before assigning a confidence judgment. Other subjects gave only for reasons,
others against reasons, and others none at all (as in the original studies). The
overconfidence phenomenon was reduced (but not completely eliminated) in those
subjects who were asked for both for and against reasons and in those subjects who
were asked for against reasons alone. Apparently subjects were failing to think of
such criticisms on their own, without the explicit instruction to do so. Subjects who
were asked to give for reasons did not differ from the control group that gave no reasons
at all. Apparently subjects think of for reasons on their own, without prompting.
In a related study, Hoch (1985) asked graduating business students, just beginning
their search for jobs, to assign probabilities to various outcomes of their job
search, such as “What is the probability that your starting salary will exceed $ ?”
or “What is the probability that you will receive more than job offers by the end
of the school year?” Actual data were collected on the same subjects at the end of the
school year from computer records of the placement office. Subjects generally had
been overconfident, although they were more overconfident when asked about high
salaries as opposed to low ones, and about few offers as opposed to many. Subjects
who were told to think of reasons why the event in question might not occur showed