Evaluating User Experience in Games: Concepts and Methods - Lirmm
Evaluating User Experience in Games: Concepts and Methods - Lirmm
Evaluating User Experience in Games: Concepts and Methods - Lirmm
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
13 Us<strong>in</strong>g Heuristics to Evaluate the Overall <strong>User</strong> <strong>Experience</strong> 249<br />
Table 13.2 The results of the evaluation ranked accord<strong>in</strong>g to issues found, po<strong>in</strong>ts obta<strong>in</strong>ed <strong>and</strong><br />
compared to the results of Metacritic.com<br />
Rank<br />
Rank<strong>in</strong>g accord<strong>in</strong>g to<br />
found issues<br />
Rank<strong>in</strong>g accord<strong>in</strong>g to<br />
po<strong>in</strong>ts Metacritic.com rank<strong>in</strong>g<br />
1 Team Fortress 2 (18) Team Fortress 2 (82.9%) Team Fortress 2 (92%)<br />
2 Sam <strong>and</strong> Max, GRID (22) Die Siedler (79.65%) GRID (87%)<br />
3 GRID (77.93%) Sam <strong>and</strong> Max (82%)<br />
4 Die Siedler (26) Sam <strong>and</strong> Max (77.7%) Die Siedler (80.6%)<br />
5 Sacred 2 (29) Sacred 2 (75.17%) Sacred 2 (78%)<br />
First, the researchers reviewed each game after play<strong>in</strong>g it, us<strong>in</strong>g the heuristics to<br />
rank the found issues accord<strong>in</strong>g to Nielsen <strong>and</strong> Mack’s severity scale (Nielsen <strong>and</strong><br />
Mack 1994), which led to the number of total usability issues found per game as<br />
displayed <strong>in</strong> Table 13.2:<br />
0. Not a usability problem at all.<br />
1. Cosmetic problem only: It does not have a profound impact on the game.<br />
2. M<strong>in</strong>or problem: It has a slight impact on the game <strong>and</strong> <strong>in</strong>fluences the experience<br />
a bit.<br />
3. Major problem: This problem has a severe impact on the game <strong>and</strong> negatively<br />
<strong>in</strong>fluences the user experience.<br />
4. Usability catastrophe: This problem has to be fixed <strong>in</strong> order to allow for a decent<br />
user experience.<br />
Second, the evaluators assigned a score from 1 to 5 (1 be<strong>in</strong>g worst, 5 be<strong>in</strong>g best)<br />
to every s<strong>in</strong>gle heuristic to determ<strong>in</strong>e how well the game fulfilled each of them.<br />
For this rat<strong>in</strong>g, the severity rank<strong>in</strong>g of the found issues was used as an <strong>in</strong>dicator<br />
for the degree of fulfilment. In general, the problems <strong>and</strong> their severity, which were<br />
found dur<strong>in</strong>g the rat<strong>in</strong>g accord<strong>in</strong>g to the above-mentioned scale, helped to determ<strong>in</strong>e<br />
which heuristics were the least satisfied ones. After the rank<strong>in</strong>g of the heuristics,<br />
the evaluators met aga<strong>in</strong> <strong>and</strong> discussed possible <strong>in</strong>consistencies <strong>in</strong> their evaluation.<br />
These problems were resolved through discussions, <strong>and</strong> when necessary evaluation<br />
<strong>and</strong>/or rank<strong>in</strong>g was adapted. The achieved overall score was obta<strong>in</strong>ed by the summation<br />
of the rat<strong>in</strong>gs by the s<strong>in</strong>gle evaluators <strong>and</strong> the calculation of an average<br />
rank<strong>in</strong>g. This score was then converted <strong>in</strong>to a percentage scale <strong>in</strong>dicat<strong>in</strong>g to which<br />
degree the game complied with the heuristics (100% would mean the achievement<br />
of maximum po<strong>in</strong>ts). The result<strong>in</strong>g rank<strong>in</strong>g is shown <strong>in</strong> Table 13.2.<br />
To compare the results of the expert-based heuristic evaluation, we chose to select<br />
at least 10 game reviews (on average 20) for each game to avoid bias<strong>in</strong>g of the s<strong>in</strong>gle<br />
reviewers <strong>and</strong> therefore guarantee a more objective rat<strong>in</strong>g. Metacritic.com 17 exactly<br />
fulfils these requirements by accumulat<strong>in</strong>g scores from different review<strong>in</strong>g sites <strong>and</strong><br />
17 http://www.metacritic.com/