Outline

CMPT 106 2/25/2010
CMPT 106
PLAYTESTING
Outline
What is playtesting
What playtesting is not
Who to test
A playtesting session
2
CMPT 106
2/25/2010
Lecture 1 1

CMPT 106 2/25/2010
Playtesting
Microsoft Games User
Research
over 3000 hours of
playtesting
more than 600 players
3 CMPT 106
2/25/2010
A conversation about playtesting
4
CMPT 106
2/25/2010
Lecture 1 2

CMPT 106 2/25/2010
A conversation about playtesting
About your process... I was really
impressed with the way that the
narrative and the game design was
well implemented. I'm wondering
how you came to that.
KS: A lot of it came out of the fact
that we playtested our game every
single week. We were watching
people play the game and seeing
how they reacted to certain
situations, and tried to play up how
they were feeling at the time with
the dialogue and different turning
points in the story. We definitely had
our players and our customers in
mind when we were making the
game.
5 CMPT 106 http://www.gamasutra.com/view/feature/3585/still_alive_kim_swift_and_erik_.php
2/25/2010
A conversation about playtesting
What really impressed me specifically
about the story was the way that the
two were really one and the same, kind
of. There was no exposition, really. The
way that you found out about the
world was by things that were there. I
was wondering if that was designdriven
or writer-driven, or was it a
combination of the two
EW: The combination of the two. It goes
back to the playtesting again. The
playtesting helps test the gameplay, but
at the end of the playtest, we'd also ask
people to tell us the story back. "Tell us
everything that you remember about the
story."
If enough people weren't picking up on
things, generally speaking, it was an
indication to us that they just didn't care
about it, which meant we should
probably either cut it or in some cases
where we didn't really want to let it go...
KS: Make it better.
6 CMPT 106
2/25/2010
Lecture 1 3

CMPT 106 2/25/2010
A conversation about playtesting
I've been telling people that I feel like
Portal is one of the more sophisticated
narrative-driven games, even though
it's hard to call it that, really, because
it's just actually well-realized.
KS: It definitely has a lot to do with
process and the fact that we sat down to
watch people play our game. It's really
incredible, how much information you can
get just from watching someone play your
game that hasn't played it before.
Seeing what they think about the game
and watching the expressions on their
face, you can tell if they're happy with
what they're feeling, or if they're bored
or if they're sad. We were always very
deliberately watching our players to see
how we could make the experience better
for them.
7 CMPT 106
2/25/2010
Playtesting – What is it
Playtesting is the single most important activity that
a designer engages in
BUT the one that designers understand the least
Performed through the entire design process
• Is the game functioning the way that you intended
• Is the game internally complete
• Is the game balanced and fun to play
8
CMPT 106
2/25/2010
Lecture 1 4

CMPT 106 2/25/2010
What playtesting is not
Just play it and gather feedback
Playing is just part of the process
When the designer and team talk about the feautres
Internal design review
Rigorously test each part of the software
Quality assurance
Marketing watches users through one-way glass
Focus group testing
Analyze how users interact with the interface
Usability testing
9
CMPT 106
2/25/2010
Iterative Design
You can‟t change game
design right before
shipping!
Continuous process of
playtesting, evaluating,
and revising
Testing cycle gets tighter
close to ship dates
Should we wait for beta
Absolutely NOT
Too late to make changes
Stuck with a dud
10 CMPT 106
2/25/2010
Lecture 1 5

CMPT 106 2/25/2010
Who to test
Self-testing
When you are building a
working version of your game
Work with your team both as
a group and as individuals
Valuable in the foundation
stage (experimenting with
fundamental concepts)
Goal: to make the game work
by solving glaring problems
As game development
progresses, continue to test
yourself, but rely more on
outside testers
11 CMPT 106
2/25/2010
Who to test
Confidants
People you know well like
friends and family
They bring fresh eyes and
uncover different things
Might need to explain things
because the prototype may be
incomplete
Goal: get a version that can be
played without much intervention
Good for getting some
feedback, but need to move
away from friends as game
matures
• They are too harsh or too
forgiving
12 CMPT 106
2/25/2010
Lecture 1 6

CMPT 106 2/25/2010
Who to test
Strangers
Hard to take criticism
from people you have
just met
You will gain fresh
perspective and insight
Objective opinions
untainted by
knowledge of the
game or personal ties
13 CMPT 106
2/25/2010
Who to test
Target audience
Want testers who
represent who will spend
money on your game
Get more relevant
feedback
Compare your game to
others they have played
(market research)
Know what they like and
dislike and will be able
to articulate
But don‟t be too narrow!
14 CMPT 106
2/25/2010
Lecture 1 7

CMPT 106 2/25/2010
Who to test
15
CMPT 106
2/25/2010
Who to test
16
CMPT 106
2/25/2010
Lecture 1 8

CMPT 106 2/25/2010
Conducting a playtesting session
Do not start by:
Telling players about the game, how it works, plans for future development, your
hopes and dreams
Defeats the purpose of a fresh perspective!
• You can never go back and get their natural first impression
Also, you do not come in the box!
Change your role:
You are no longer designer, but investigator and objective observer
Lead the player, record their actions, analyze their responses
Don‟t interfere
Give minimal explanation
You must allow them to make mistakes!
See how they approach the game
If they get stuck, let them figure it out – you will learn from their mistakes
If you can, have an objective person run the test while you observe
Make a test script to keep you on track
17
CMPT 106
2/25/2010
The golden rule
TREAT YOUR PLAYTESTERS WITH RESPECT
18
CMPT 106
2/25/2010
Lecture 1 9

CMPT 106 2/25/2010
Playtest process
1. Introduction (2-3 minutes)
Yourself and the process
You are not testing them, but are testing the game
Audio and video, logging
2. Warm-up discussion (5 minutes)
Relax the player
Find out about the tester
• Tell me about some of the games you play
• What do you like most about these games
• What was the last game you purchased
19
CMPT 106
2/25/2010
Playtest process
3. Play session (15-20 minutes)
Remind them that difficulties they have will improve
the game
Observe their play from a distance
Think-aloud
• What choices are they making
• What uncertainties they have when playing
• Running monologue of the player‟s thoughts
20
CMPT 106
2/25/2010
Lecture 1 10

CMPT 106 2/25/2010
Playtest process
4. Discussion of game experience (15-20 minutes)
Have one-on-one discussion with playtesters
Develop a set of questions
• Open-ended questions are more valuable at this stage
• E.g., Overall what are your thoughts about the game
• Were you able to learn how to play quickly
• What is the objective of the game
• Is there any information that would have been useful to you before
starting
• Is there anything that you don‟t like about the game
• Was anything confusing Please take me through what you found
confusing.
• More specific questions later in the design process
5. Wrap-up
Thank them!
21
CMPT 106
2/25/2010
The golden rule
TREAT YOUR PLAYTESTERS WITH RESPECT
22
CMPT 106
2/25/2010
Lecture 1 11

CMPT 106 2/25/2010
Observation methods
Three common approaches
Simple observation
Think-aloud
Co-discovery learning
23
CMPT 106
2/25/2010
Simple Observation
Player is give a task (or not), and evaluator just
watches the user
Problem: no insight into the player’s decision process or attitude
24
CMPT 106
2/25/2010
Lecture 1 12

CMPT 106 2/25/2010
Think-aloud Protocol
Players are asked to say
what they are
thinking/doing:
What they believe is
happening
What they are trying to
do
Why they took an action
Gives insight into what
the player is thinking
25 CMPT 106
2/25/2010
Think-aloud Protocol
Problems:
Awkward/uncomfortable for
subject (thinking aloud is not
normal!)
“Thinking” about it may alter
the way people perform their
task
Hard to talk when they are
concentrating on problem
Interferes with game play
Still is the most widely used
method in general industry
26 CMPT 106
2/25/2010
Lecture 1 13

CMPT 106 2/25/2010
Other Problems with Think-aloud
27
CMPT 106
2/25/2010
Retrospective Think-aloud
Problems with Think-aloud:
Awkward/uncomfortable for
player (thinking aloud is not
normal!)
“Thinking” about it may alter
the way people perform their
task
Hard to talk when they are
concentrating on problem
Interferes with game play
Solution: Videotape the
experience and perform a
retrospective think-aloud
Has its own problems
Awkwardness of watching
themselves on video
Awkwardness of reliving
mistakes
Reflection of the experience
rather than in context
More about „what
happened‟ than how they
felt about it
28 CMPT 106
2/25/2010
Lecture 1 14

CMPT 106 2/25/2010
Co-discovery Learning
Two people play
together
Normal conversation
between the two
players is monitored
Removes awkwardness
of think-aloud, more
natural
Provides insights into
thinking process of
both players
29 CMPT 106
2/25/2010
Recording Observations
Paper and pencil
primitive but cheap
evaluators record events, interpretations, and extraneous
observations
evaluator seems disengaged
problem: writing is slow
• prepared coding schemes can help; just tick off events
Audio recording
capture discussion (think aloud, co-discovery)
hard to synchronize streams (e.g., interface actions)
• (expensive) tools exist to help
transcription is slow and difficult!
• tools exist to help
30
CMPT 106
2/25/2010
Lecture 1 15

CMPT 106 2/25/2010
Recording Observations
Video recording
can see what a user is doing
(good to use one camera/scan converter for screen +
one for subject)
can be intrusive (at least initially)
analysis can be challenging
• annotation is so time consuming and very very boring
Companies often build “usability labs” with one-way
mirrors, video cams, etc.
31
CMPT 106
2/25/2010
A conversation about playtesting
32
CMPT 106
2/25/2010
Lecture 1 16

CMPT 106 2/25/2010
Sidhe Griffths – Wii SpeedRacer
http://www.gamasutra.com/view/feature/3894/practical_game_playtesting_a_.php
“The first time players get their hands on
the game always results in problems --
and when it comes time to write up the
report, I realize with each soul-destroying
point that it's my job to then present this
information to the developers.
However, the light at the end of the tunnel
is that we can find and tackle these
problems while the game is still in our
hands. Once it has been released, it is
too late, which is far from a good thing.”
One of the ways we were able to create
such a solid game was through
playtesting on those players who would
be playing the game (in this case the
target demographic was 9 - 14 year
olds). Once they were in, every aspect of
the session was recorded; from the
gameplay to the player's expressions and
the way they use the input device. While
it may seem overkill, everything can
potentially be used to fine-tune a game.
33 CMPT 106
2/25/2010
Sidhe Griffths – Wii SpeedRacer
http://www.gamasutra.com/view/feature/3894/practical_game_playtesting_a_.php
Using the Wiimote for a driving game was
something we thought would instantly be
intuitive. While the game supported the Mario
Kart wheel add-on, it could also be played
without it. What we didn't foresee was the way
people would take it upon themselves to use the
device.
The player was given a small animation
showing how to hold the Wiimote, and we
assumed that it was straightforward enough.
However, what happened was that the majority
of players perceived the animation as if it were
from the top-down, and so would control the
car as if they were using the steering wheel of
a pickup truck.
Therefore, for the loading screen we put in a
simple black and white image of a television
screen, with the Wiimote in front of it. It was
hoped that this image would act like a frame of
reference for holding the Wiimote. And it
worked! The next time we ran a playtesting
session the players picked up the Wiimote and
were able to instantly drive.
34 CMPT 106
2/25/2010
Lecture 1 17

CMPT 106 2/25/2010
Sidhe Griffths – Wii SpeedRacer
http://www.gamasutra.com/view/feature/3894/practical_game_playtesting_a_.php
The solutions we found for the above issues
were also adapted and used in one way or
another for many of the others found within the
game, and by spending time playtesting, which
did not tax the development process, as
sessions can be run concurrently with
development itself, we removed a huge number
of issues that would otherwise have potentially
remained within the game after it had shipped.
You need to ask what would have happened if
that had been the case. It is certain that not
only would the reviews have been a lot lower,
but the word of mouth between the players
themselves would have been extremely
negative too.
The final part of playtesting is to verify that the
changes you have made actually work. This is a
relatively easy matter and can be carried out
when running other sessions. Observing if issues
come up or not will quickly determine whether
the problems have been solved.
35 CMPT 106
2/25/2010
Write a playtesting script
Helps you ignore your ego
Learn to take criticism
without emotion
Do not answer criticisms,
just write them down
Your goal is not to get
validation, but to make
your game better
Players want to please
you and will respond
according to your attitude
Discourage snark
36 CMPT 106
2/25/2010
Lecture 1 18

CMPT 106 2/25/2010
Methods of playtesting
Groups dynamics can be good for generating ideas
and sometimes bad for evaluating ideas
One-on-one testing: Sit down with individuals and
observe
Group testing: Get a group together to play your game
Feedback forms: Give the testers a standard list of
questions to answer (can feed into database)
Interview: Sit down face-to-face with testers
Open discussion: Conduct a one-on-one discussion or
group discussion
Data Hooks: Log events in the game itself
37
CMPT 106
2/25/2010
Styles of Questions: Open-ended
Open-ended questions ask for opinions
Good for general subjective information
but, difficult to analyze rigorously
E.g., “Can you suggest any improvements to the inventory
interface”
38
CMPT 106
2/25/2010
Lecture 1 19

CMPT 106 2/25/2010
Styles of Questions: Closed
Restricts responses by supplying the choices for answers
Can be easily analyzed …
But, questions must be well designed!
Did you access the journal:
O never O 1-3 times O 4-5 times O more than 5 times
How many hours a day do you spend playing computer games
O over 4 hrs a day
O between 2 and 4 hrs daily
O between 1 and 2 hrs daily
O less than 1 hr a day
39
CMPT 106
2/25/2010
Styles of Questions: Likert Scale
Measure opinions, attitudes, and beliefs
Asks user to judge a statement on a numeric scale
Scale usually corresponds to agreement or
disagreement with a statement
The text on the computer screen was hard to read:
strongly agree
strongly disagree
1 2 3 4 5
40
CMPT 106
2/25/2010
Lecture 1 20

CMPT 106 2/25/2010
Styles of Questions: Semantic Anchors
Opposing attitudes anchor the response
The fighting minigame is:
fun 1 2 3 4 5 boring
easy 1 2 3 4 5 hard
41
CMPT 106
2/25/2010
Styles of Questions: Multi-Choice
respondent offered a choice of explicit responses
How do you most often get help with games (tick one)
O on-line manual
O paper manual
O ask a friend
Which types of games do you enjoy (tick all that apply)
O action/adventure
O RPG
O sports
O puzzle
42
CMPT 106
2/25/2010
Lecture 1 21

CMPT 106 2/25/2010
Styles of Questions:
Combining Open-ended & Closed Questions
gets specific response, but allows room for user‟s opinion
The boss battle was difficult:
disagree agree comment
1 2 3 4 5
I couldn’t get the rocket launcher working!
43
CMPT 106
2/25/2010
The Play Matrix
To help playtesters and
players give context to their
discussions
Horizontal – Skill to chance
Vertical – Metal to Physical
Core aspects of game
experiences
Chess
Blackjack
Guitar Hero
Tetris
Whack-a-mole
Halo
Civilization
44 CMPT 106
2/25/2010
Lecture 1 22

CMPT 106 2/25/2010
The Play Matrix
Good for stimulating
discussion and analyzing
gameplay
Get your playtesters to
think about the game and
verbalize their feelings
Pattern in the play matrix
Use the play matrix to
plot five of your favorite
games
What pattern do you see
What does this tell you
45 CMPT 106
2/25/2010
The Play Matrix
Ask your testers to plot your
game
Is the outcome determined by skill
or chance
More by mental skill or physical
dexterity
Would they move the game one
way
• Note that different audiences will
feel differently
You can change aspects of your
game (e.g., move a variable
determined by chance to one
determined by player choice
• E.g., Remove dice and add cards
that they can play
46 CMPT 106
2/25/2010
Lecture 1 23

CMPT 106 2/25/2010
Observations and Playtester comments
Basic techniques (see Figure 9.10 in text)
Do not lead
• If players ask a question, give it back to them
• If they reach an impasse, you have found a problem
Remind them to think out loud if possible
• Commentaries provide a window into players‟ expectations
and choices
Quantitative data
• Don‟t just take notes, use forms to gather numbers
47
CMPT 106
2/25/2010
Test control situations
Test specific aspects of the game
The end of the game
A random event that rarely takes place
A special situation within the game
A particular level of a game
New features
48
CMPT 106
2/25/2010
Lecture 1 24

CMPT 106 2/25/2010
Resources
Valve's Faliszek: Playtesters Aren't Idiots, It's You
http://gamasutra.com/php-bin/news_index.phpstory=19523
Best Of GDC: The Secrets Of Portal's Huge Success
http://gamasutra.com/php-bin/news_index.phpstory=17625
GDMag Q&A: How Halo 3 Got Legendary
http://www.gamasutra.com/phpbin/news_index.phpstory=17636
Future Play 2007: Indie Dev Frozen North On Pushing The
Prototype
http://www.gamasutra.com/phpbin/news_index.phpstory=16269
49
CMPT 106
2/25/2010
Lecture 1 25