1st Year Cohort programs at SFU Surrey - IAT 432: Design Evaluation

IAT 432
Week 11
Usability Evaluation
Can Be Harmful

…there were two
HCI researchers:
Saul Greenberg
and
Bill Buxton.
Once upon a time…

Saul Greenberg
• Full Professor in the
Department of Computer
Science at the University of
Calgary
• HCI, CSCW, PUI or “NUI”
• Badass mountaineer
• Bonus Q: SIAT connection?

Bill Buxton
• Principal Researcher at
Microsoft Research
• Multitouch and Music in the
1970’s
• Worked for Alias Wavefront
(before Maya was Maya)
• Also a badass mountineer
• Bonus Q: SIAT connection?

The would attend conferences…
• Premier HCI conference
• Attended by academia and industry
• 2600+ attendees, not easy to present
5

… and speak to industry…
• Most large IT companies have internal Usability teams
– Test product under iterative development
– Produce list of Usability issues to be fixed in the next dev
round
6

… and they noticed something very, very wrong.
• Blind usability
– “gives meaningless and trivial results”
– Misdirects or prematurely kills future design directions
7

So they wrote a paper. The End.
8

What’s wrong with Usability
• Usability methods are a way to engage in user-centered
design – T/F?
• Weak science
• Objectivity vs. Subjectivity
• Evaluating Early Designs
• Predicting Cultural Adoption
9

Usability is Weak Science
• Scientific methods != good science
– Little discussion of methods, big focus on Quant over qual
• Quant gets an easy pass compared to Qual
– Researchers typically choose the method, then adjust
usability question to fit the method
10

• Ideal HCI science:
Usability is Weak Science
– “When performing a series of tasks, the use of the new
technique leads to increased human performance when
compared to the old technique.”
• Reality HCI science:
– There is at least one case where my design performs
better than the old way.
11

Usability is Weak Science
• Existence Proof vs Risky Hypotheses
– If a hypothesis holds in spite of trying to refute it, it is
stronger science.
• Lack of replication
– “It’s been done before” -> bias towards new and shiney
– Non experts in your subdomain are more likely to think
it’s been done before!
12

Most Usability methods try to be objective…
Objectivity vs. Subjectivity
…when objectivity may not help improve design.

• How do:
– Graphic designers
– Industrial designers
– Web designers
– Car designers
– Craftspeople
• …tell if a design is good?
Objectivity vs. Subjectivity
14

• Designer-as-assessor
Objectivity vs. Subjectivity
– Snodgrass and Coyne – see paper for full ref
• “[Design evaluation] is not haphazard because the
assessor has acquired a tacit understanding of design
value and how it is assessed, a complex set of tacit
norms, processes, criteria and procedural rules,
forming part of a practical know-how.”
15

Objectivity vs. Subjectivity
• Subjective is just as valid (if not more valid) than
objective, depending on context
• Design is not math, physics, chemistry, biology, etc.
• Design is embedded in incredibly complex and highly
contextual processes
16

Usability devalues early-stage design
• Brainstorming
– “Your idea is not very user friendly, therefore it is
dumb!”
• Early design is under-specified in ways that often make
it “un-usable”
• Don’t mistake a sketch for a prototype!
17

Getting the Design Right (prototyping)
Vs Getting the Right Design (sketching)

Usability devalues early-stage design
• Most Usability methods make sense on prototypes, but
not on sketches.
• Use your judgment as a designer!
20

Usability is a poor predictor for cultural adoption
• Useable or Useful?
• Most customers (and many designers) don’t understand
how radically new tech can be useful
21

Usability is a poor predictor for cultural adoption
• Creativity is socially constructed
– People have to recognize and make sense of new stuff…
– …but is has to be different enough for them to see the
value
• Big shifts in tech are more likely to change the world
but are hard to eval w. Usability
22

Usability is a poor predictor for cultural adoption
• Consumers and pop media have a short memory and
even shorter attention span.
– Memex – Vannevar Bush, 1945
• http://www.youtube.com/watch?v=c539cK58ees
– Sketchpad – Ivan Sutherland, 1962
• http://www.youtube.com/watch?v=495nCzxM9PI
– NLS (aka “The Mother of All Demos”) – Dougles Engelbart,
1968
• http://www.youtube.com/watch?v=JfIgzSoTMOs
23

If Usability is so harmful,
why do we need this course?
• Tech is cheap, experience matters
• Sign of a maturing field
• Useful for building standards with broad acceptance
and agreement
• Still better than nothing (sometimes)
24

Other Approaches to Design and Evaluation
• Better understanding of requirements
• Consider cultural aspects that Usability would miss
• Investigating new interface possibilities through
technical innovation
25

Other Approaches to Design and Evaluation
• Develop and show design alternatives
– Good design process is sometimes like grinding in an
RPG
– More point in a design space == greater chance of
better designs!
• Reflect on Pros and Cons of usability at a particular
point in design cycle
• Use Usability when it makes sense!
26

• Anatomy of an HCI paper
– Abstract
Assignment 5
– Intro -> Here’s the problem
– Argument I -> problem
– Argument II -> problem
– Argument III…etc.
– What to do? -> fix problems
– Conclusion
– References
27

• Due Wednesday, April 11 th
Assignment 5
– One week before final exam
• Be persuasive!
• You have some time – use it wisely.
– Proof read your submission at least twice
– Discuss Q’s early, let ideas percolate
– Email Andrew or I with questions that can be answered
quickly.
28

• Assignment 4 Q’s?
House Keeping
• Email Assignment 5 partner to Andrew by the end of
the week!
• Exam Q Review – Lecture or Lab?
– The vote is for Lecture - AB
29

Lab
• This week’s lab cancelled due to illness
– (Maybe email Andrew and tell him to toughen up/get
better?)
• Next week:
– Guest lecture by Veronica Zammitto (EA)
– Lab: Usability Can Be Harmful exercises
• RTFA!
• I mean, pretty-please read the Greenberg and
Buxton article.
30