GOMS, KLM, CogTool - Classes

Evaluation Types
GOMS and KLM
CS352

Announcements
• This week:
– Evaluating designs
• Reading: ch12.3-5 (just skim 12.4), ch15.2-4
• Next week:
– “Concepts and Prototypes 1”
• due 11:30pm, July 18 th (next Monday)
– Midterm: July 20 th (next Weds)
– Poster session: Concepts and Protos (next Thurs)
2

Concepts and Prototypes
• Due next Monday; poster session on Thursday
– Very brief summary of the task and the users
– Some concepts (i.e. more than one different idea)
• Why they fit your users/task
– One prototype, consisting of:
• Storyboard; should show majority of functionality
• Justifications of your design decisions, in terms of your
research
– Physically:
• “Poster” tells you how big it should be, but laying
normal sheets of paper out on the desks here is fine
• Don’t use CogTool yet. Hand sketches or bad drawing
software are fine.

“Justification” in later assignments
• Some assignments ask you to “justify” in
terms of your user study
• If your user study wasn’t enough, you can do
mini-follow ups:
– observations
– interviews
• Just write a few sentences describing them,
and that will work.
Because we all know “WATERFALL DOESN’T WORK”, right?

Where we are in PRICPS:
• Predispositions: Did this in Project Proposal.
• RI: Research was studying users. Hopefully led
to Insights.
• CP: Concept and initial (very low-fi) Prototypes
due some time next week
• Evaluate throughout, repeat iteratively!!

Evaluation
• Analytical – based on your head
• Empirical – based on data
– Formative
• inFORMs design
• what is (still) needed?
– Summative
• did it work?

Analytical methods
• You follow established
guidelines/procedures/models to decide (in
your head) how good your design is.
• Examples:
– GOMS/KLM – for skilled users, no errors.
• evaluating efficiency of regular use.
– Heuristic Evaluation
– Cognitive Walkthrough – for first-time users.
• evaluating ease of learning.

GOMS (and KLM)
• GOMS: a family of models.
• Predict user performance.
• Useful for predicting actual time a user will
take in UI.
• Useful for comparing different UIs.

• Goal: “what”.
• Method: “how” steps
(learned).
• Operators: Cognitive
processes + physical
actions to DO it.
• Selection rules: rules
saying which method
to select.
GOMS Constructs
Roughly correspond
to Norman’s “gulf of execution”
stages. Here’s my guess…

GOMS example:
Delete a word
• Goal: delete a word in a sentence.
• Method #1: use the menu
– Recall that the word has to be highlighted.
– Recall that the command is “cut”.
– Recall that “cut” is in the Edit Menu.
– Accomplish goal of selecting and executing “cut”.
– Return: goal accomplished.

GOMS example (cont.)
• Method #2: use the delete key
– Recall where to position cursor in relation to word
to be deleted.
– Recall which key is delete key.
– Press “delete” key to delete each letter.
– Return: goal accomplished.
• Operators used in these methods
– Click mouse, Drag cursor over text, Select menu,
Move cursor, Press KB key, Recall, ...

GOMS example (cont.)
• Selection rules:
– Use mouse/menu method (#1) if there’s a lot of
text to delete.
– Else use “delete” key (method #2).

Applications of GOMS
• Various application and fields
– Telephone operator workstation using CPM-GOMS
– CAD system for ergonomic design using NGOMSL
– Intelligent tutoring system using NGOMSL
– Mouse driven text editor using KLM
– Bank deposit reconciliation system using KLM
– Space operations database system using KLM

Limitations of GOMS [Card et al. (1980) ]
• applied to skilled users, not to beginners or
intermediates.
• doesn't account for either learning of the system or
its recall after a period of disuse.
• doesn't account for errors.
• does not address the amount and kind of fatigue
• individual differences among users is not accounted
for in the model.

KLM (a low-level variant of GOMS)
• Keystroke Level Model.
• Simple, but accurate. Widely used.
• Scope:
– skilled users
– doing a task error-free.
– using a specific method in a UI.
• CogTool has this built-in.

KLM Operators
• User Operators:
– K (keystroke), P (point), H (homing), D (drawing),
M (mental: think).
– Times for each are provided to you
• based on extensive research/empirical data.
• System Operator:
– R (respond).

• 27 ways to
open a file
in Windows!
• Methods as
states +
transitions.
• (High op
granularity
combines
low-level ops.)
KLM/GOMS example 1:
Open a file

KLM/GOMS example 1 (cont.)
• Used to:
– Understand cost
of each.
– (break down
participants
actual costs).
– “RL”:recent lists.
– “WS/WX”:search.
• And to...

KLM/GOMS example 1 (cont.)
• Use the breakdowns to compare costs to new
UI alternative (new feature “FP”)
actual
calc’d

Hick’s Law
• GOMS’s “S” stands for “Selection”
• Hick’s Law predicts the time to make a choice
between multiple choices, n is the # of choices:

Fitts’ Law
• Predicts that the time required to rapidly move to a
target area is a function of the distance to and the
size of the target

In practice…
• GOMS not used often
• Fitts’ law often used for determining best case
for new kinds of input methods

http://cogtool.hcii.cs.cmu.edu/
Some features
– Mock up interfaces and tasks
– KLM cost of the task in this UI.
– Comparing the cost across different
designs
– ACT/R Simulation of the task
– Import/Export to HTML

What’s coming up next:
• Heuristic Evaluation
• Cognitive Walkthroughs
• Empirical Studies