Thinking and Deciding

METHODS FOR EMPIRICAL RESEARCH 47
develop new treatment methods often proceed on the basis of imperfect research.
Similarly, educators and decision scientists can develop new ways to improve thinking
and decision making on the same basis.
General issues
A few general issues arise repeatedly in discussions of research on judgment and
decision making.
Within-subject versus between
Often a researcher wants to examine the effect of some manipulation on a judgment.
For example, consider the “outcome bias” experiments of Baron and Hershey (1988).
The idea was to show that subjects judged the quality of a decision by its outcome,
even when the outcome was determined by luck and when the subjects knew everything
that the decision maker knew. Subjects made judgments about hypothetical
scenarios, in which someone made a decision that had either a good or bad outcome.
For example, a surgeon decided to do an operation, with a known probability of success,
and the operation either succeeded or not. Subjects judged the surgeon to be a
better decision maker when the operation succeeded. The success, of course, was a
matter of luck. The surgeon had taken a “calculated risk.”
One way of doing this kind of experiment is to present the good and bad outcomes
next to each other. For example, “Suppose that the operation succeeded and the
patient recovered. Please rate the surgeon’s decision. Now suppose that the operation
failed and the patient died. Please rate the decision again.” When this is done, most
subjects give the same rating. They understand that using the outcome is a judgment
made in hindsight and that the surgeon knew only the probability of success. This is
called a within-subject design because the experimental manipulation is done within
each subject. It is also a transparent design, because the subject can easily see the
manipulation.
Another way to do the experiment is to present the good and bad outcomes to
different subjects. This is called a between-subject design. In this kind of research,
these two methods answer somewhat different questions. The within-subject method
asks what subjects think they should do. In this case, it asks whether they think
that outcome ought to matter. It is obvious, when the two outcomes are next to
each other, that this is what they are being asked. The between-subject method asks
whether people are affected by success versus failure when they are unaware that the
difference is being manipulated. This is more like the real-world situation.
Subjects think that success-failure should not matter, yet they are affected by it
in a between-subject design. This seems to be a clear demonstration of a violation of
a normative model, since the subjects themselves endorse the normative model that
outcome should not matter. At the very least, the subjects are being inconsistent with
their own judgments. But what of those few subjects who show the effect even in a
within-subject design? Can they too be said to be violating a normative model?