User Interface Design and Ergonomics - National Open University of ...

It's very easy to shape the comments users will give you, and what they do in the test, by
asking questions and making suggestions. If someone has missed the significance of
some interface feature a word from you may focus their attention right on it. Also,
research shows that people will make up an answer to any question you ask, whether or
not they have any basis for the answer. You are better off, therefore, collecting the
comments people offer spontaneously than prodding them to tell you about things you are
interested in.
On helping, keep in mind that a very little help can make a huge difference in a test, and
you can seriously mislead yourself about how well your interface works by just dropping
in a few suggestions here and there. Try to work out in advance when you will permit
yourself to help. One criterion is: help only when you won't get any more useful
information if you don't, because the test user will quit or cannot possibly continue the
task. If you do help, be sure to record when you helped and what you said.
A consequence of this policy is that you have to explain to your test users that you want
them to tell you the questions that arise as they work, but that you won't answer them.
This seems odd at first but becomes natural after a bit.
c. Recording
There are plain and fancy approaches here. It is quite practical to record observations
only by taking notes on a pad of paper: you write down in order what the user does and
says, in summary form. But you'll find that it takes some experience to do this fast
enough to keep up in real time, and that you won't be able to do it for the first few test
users you see on a given system and task. This is just because you need a general idea of
where things are going to be able to keep up. A step up in technology is to make a video
record of what is happening on the screen, with a lapel mike on the user to pick up the
comments. A further step is to instrument the system to pick up a trace of user actions,
and arrange for this record to be synchronized in some way with an audio record of the
comments. The advantage of this approach is that it gives you a machine readable record
of user actions that can be easier to summarize and access than video.
A good approach to start with is to combine a video record with written notes. You may
find that you are able to dispense with the video, or you may find that you really want a
fancier record. You can adapt your approach accordingly. But if you do not have a video
setup, do not let that keep you from trying the method.
d. Summarizing the Data
The point of the test is to get information that can guide the design. To do this you will
want to make a list of all difficulties users encountered. Include references back to the
original data so you can look at the specifics if questions arise. Also try to judge why
each difficulty occurred, if the data permit a guess about that.
182