Designing a persistent online strategy game - Department of ...

40 Chapter 4. Models & methods for evaluating games
playing and comment on their experience. A moderator observes their play and takes
notes, but should try not to influence their play. To find areas where players are stuck
or have problems as little help as possible should be given.
After the session the players are interviewed about their experience. Other methods
that can be used are feedback forms or open ended discussions.
4.8.3 Consumer Playtest
The Consumer Playtest (CP) [14] is a quantitative method which purpose is to identify
gameplay issues with a game. The focus is to gather players’ attitudes towards different
aspects of a game. CP is performed with 25-35 participants in a lab. The first step
is to answer a questionnaire that is designed to ensure that they belong to the target
group of the game. Step two is to let them play the game for a period of time, they
are instructed to play it as they would do at home and are also given all material that
they would have if they bought it. In step three players answers a questionnaire about
their experience. The questions can concern aspects such as perceived graphics, sound,
controls, and overall fun. Some questions are game specific, such as for example the
puzzle elements in a puzzle game.
With a small sample of users in evaluation it is hard to generalise their opinions and
experiences to the whole target group. The data gathered there is only representative
for the users in the evaluation. In CP the sample of players is large enough to enable
statistical analysis on the data gathered. This allows for the comparison between different
CP sessions. Different games and different version of the same game can thus
be compared statistically. This encourages iteration when designers can see how the
changes affect players’ opinions and experience of a game.
4.8.4 Heuristic evaluation
Heuristic evaluation is one of the discount methods described by Nielsen [43]. When
using discount methods evaluators sacrifice some degree of certainty and statistical significance
in favor of a low cost. The argument for using discount methods is that some
usability evaluation is better than none [43]. People do not have to be usability experts
to do a heuristic evaluation; novices can also identify some usability issues. It however
requires some expertise to know how to apply the heuristics so experienced evaluators
usually finds more issues than novices.
In heuristic evaluation one or several experts review the system with the help of the
heuristics. Every occurrence where the system violates the heuristics is noted down and
graded. Usually the expert also gives an example of a solution to the problem. If there
are several experts it is important that the evaluation is performed separately. This is
to ensure that as many problems as possible are identified. After evaluation the experts
meet to discuss the problems they found before they write a report for the designers
[43].
Games can also benefit from heuristic evaluation. Usability heuristics has been shown
to provide new and useful data in game development [34]. But the differences between