User Interface Design and Ergonomics - National Open University of ...

a. Choosing the Order of Test Tasks
Usually you want test users to do more than one task. This means they have to do them in
some order. Should everyone do them in the same order, or should you scramble them, or
what? Our advice is to choose one sensible order, starting with some simpler things and
working up to some more complex ones, and stay with that. This means that the tasks that
come later will have the benefit of practice on the earlier one, or some penalty from test
users getting tired, so you can't compare the difficulty of tasks using the results of a test
like this. But usually that is not what you are trying to do.
b. Training Test Users
Should test users hit your system cold or should you teach them about it first? The answer
to this depends on what you are trying to find out, and the way your system will be used
in real life. If real users will hit your system cold, as is often the case, you should have
them do this in the test. If you really believe users will be pre-trained, then train them
before your test. If possible, use the real training materials to do this: you may as well
learn something about how well or poorly it works as part of your study.
c. The Pilot Study
You should always do a pilot study as part of any usability test. A pilot study is like a
dress rehearsal for a play: you run through everything you plan to do in the real study and
see what needs to be fixed. Do this twice, once with colleagues, to get out the biggest
bugs, and then with some real test users. You'll find out whether your policies on giving
assistance are really workable and whether your instructions are adequately clear. A pilot
study will help you keep down variability in a bottom-line study, but it will avoid trouble
in a thinking-aloud study too. Don't try to do without!
d. What If Someone Does not Complete a Task?
If you are collecting bottom-line numbers, one problem you will very probably encounter
is that not everybody completes their assigned task or tasks within the available time, or
without help from you. What do you do? There is no complete remedy for the problem. A
reasonable approach is to assign some very large time, and some very large number of
errors, as the "results" for these people. Then take the results of your analysis with an
even bigger grain of salt than usual.
e. Keeping Variability Down
As we've seen, your ability to make good estimates based on bottom-line test results
depends on the results not being too variable. There are things you can do to help, though
these may also make your test less realistic and hence a less good guide to what will
happen with your design in real life. Differences among test users is one source of
variable results: if test users differ a lot in how much they know about the task or about
the system you can expect their time and error scores to be quite different. You can try to
187