User Interface Design and Ergonomics - National Open University of ...

If you are led to develop more and more elaborate approximations to the real system for
testing purposes you need to think about controlling costs. Simple mockups are cheap,
but prototypes that really work, or even Wizard of Oz setups, take substantial
implementation effort.
Some of this effort can be saved if the prototype turns out to be just part of the real
system. As we will discuss further when we talk about implementation, this is often
possible. A system like Visual Basic or Hypercard allows an interface to be mocked up
with minimal functionality but then hooked up to functional modules as they become
available. So don't plan for throwaway prototypes: try instead to use an implementation
scheme that allows early versions of the real interface to be used in testing.
3.1.4 DECIDING WHAT DATA TO COLLECT
Now that we have people, tasks, and a system, we have to figure out what information to
gather. It is useful to distinguish PROCESS DATA from BOTTOM-LINE data. Process
data are observations of what the test users are doing and thinking as they work through
the tasks. These observations tell us what is happening step-by-step, and, we hope,
suggests WHY it is happening. Bottom-line data give us a summary of WHAT happened:
how long did users take, were they successful, how many errors did they make.
It may seem that bottom-line data are what you want. If you think of testing as telling you
how good your interface is, it seems that how long users are taking on the tasks, and how
successful they are, is just what you want to know.
We argue that process data are actually the data to focus on first. There's a role for
bottom-line data, as we discuss in connection with Usability Measurement below. But as
a designer you will mostly be concerned with process data. To see why, consider the
following not-so-hypothetical comparison.
Suppose you have designed an interface for a situation in which you figure users should
be able to complete a particular test task in about a half-hour. You do a test in which you
focus on bottom-line data. You find that none of your test users was able to get the job
done in less than an hour. You know you are in trouble, but what are you going to do
about it? Now suppose instead you got detailed records of what the users actually did.
You see that their whole approach to the task was mistaken, because they didn't use the
frammis reduction operations presented on the third screen. Now you know where your
redesign effort needs to go.
We can extend this example to make a further point about the information you need as a
designer. You know people weren't using frammis reduction, but do you know why? It
could be that they understood perfectly well the importance of frammis reduction, but
they didn't understand the screen on which these operations were presented. Or it could
be that the frammis reduction screen was crystal clear but they didn't think frammis
reduction was relevant.
180