Controlled Lab Experiments - Allen Bevans

IAT 432
Week 6
Controlled Experiments 1/3
(Assignment 3)
Scientific Method and
Hypothesis Testing

• Examples?
What is Science?
2

• Examples?
What is Science?
• How is that different what Designers do?
3

• Systematic inquiry
What is Science?
• Community standards (rigor)
• Repeatable
4

Descriptions vs Explanations
• Description: Qualitative
– Words
– Summarized via top-down or bottom-up analysis, written
summary.
– “I observed these statements…”
• Description: Quantitative
– Numerical data sets
– Summarized via stats, graphs, etc.
– “I observed these measurements…”
5

Descriptions vs Explanations
• Explanation: Qualitative
– Top-down or bottom-up analysis
– Shaped by what we expect
– “These statements suggest…”
• Description: Quantitative
– Statistical analysis
– Shaped by what we expect
– “These measurements suggest…”
6

Descriptions vs Explanations
• Explanation: Qualitative
– Top-down or bottom-up analysis
– Shaped by what we expect
– “These statements suggest…”
• Description: Quantitative
– Statistical analysis
– Shaped by what we expect
• How do we know what to expect?
– “These measurements suggest…”
7

• Theory
• Causes and effects
Scientific Method
– A causes B effect or A affects B.
• Testable hypothesis
• Controlled Experiment with subset of population
• Evidence to support hypothesis?
• Generalize to population
8

Experimental Method
• Scientific Method � Empirical Method
• “Evidence”
• Claims/hypotheses
• Quantitative Data
• Objective
9

A � B
Causes and Effects In Usability
• A = cause = input or interface feature
• B = effect = human performance, preference,
experience
• Common form:
– A1 is better than A2 for causing B
10

Controlled Experiment Approach
A controlled experiment for usability evaluations is good
for certain kinds of questions …
• Is one design better in terms of usability than another?
• Does a change in interface feature change usability?
– i.e. performance (effectiveness/efficiency) or preference
(satisfaction)
11

Mouse Size Example
• Theory: Mouse size affects children’s performance on
selection tasks
• Hypothesis: Children can select targets faster with a
small mouse compared to a regular mouse.
• Comparative controlled experiment
– Cause: Mouse size -- e.g., MS Optical mouse 15 cm L x 10
cm W x 7 cm H vs 10 cm L x 7 cm W x 4 cm H
– Effect: task speed from beginning to select
– Population: children age 4-6
12

• Empirical
Term
= Relying on or derived from observation
For example, a user-based usability study is an
empirical study.
13

• Hypothesis (singular)
• Statement
• Causes and effects
Hypotheses
• Interface design A causes B
• Comparing two designs
– A1 is better than A2 for B
14

• Example
True or Not? The Logic of Proofs
– Statement: All swans are white.
– Observe 10,000,000 swans.
– All the observed swans are white.
– Proof?
15

True or Not? The Logic of Proofs
• Can NEVER prove a hypothesis …
• Can NEVER prove a statement with observations.
• Can only find support for or disprove.
• Example
– Statement: All swans are white.
– Observe 10,000,000 swans.
– All the observed swans are white.
– Proof? No. Support: Yes.
16

Logic of Proofs cont’
Prove by disproving opposite statement.
How to disprove … If all swans are white, then no swans are
black.
Opposite statement: No swans are black (blue, pink,
orange)
Find 1 black swan!
No swans are black is disproven.
17

Logic of Proofs cont’
In Scientific method – the approach is to disprove a “null”
hypothesis.
All we can say, then, is “Evidence supports…”
18

Apply logic of proofs to hypotheses
• Opposite is “no effect” … “no better”
• Called “null” hypothesis …
• E.g. Smelliness and dating
• General = Smell effects number of dates.
• Null = Smell does not effect number of dates.
19

Apply logic of proofs to hypotheses
• Opposite is “no effect” … “no better”
• Called “null” hypothesis …
• E.g. Smelliness and dating
• General = Smell effects number of dates.
• Null = Smell does not effect number of dates.
• Directional = Smell gets more dates.
– Better smell gets more dates?
20

Operationalizing the Hypothesis
• Hypothesis must be testable
• Operationalize �
1. Isolate & specify cause (e.g. interface feature)
2. Measure effect (performance, preference,
experience) on some kind of activity/task
3. Specify population
21

Mouse Size Example
• Theory: Mouse size affects children’s performance on
selection tasks
• Hypothesis: Children can select targets faster with
small mouse than regular mouse.
• Comparative two designs in a controlled experiment
– Cause: Mouse size -- e.g., MS Optical mouse 15 cm L x 10
cm W x 7 cm H vs 10 cm L x 7 cm W x 4 cm H
– Effect: task speed from beginning to select
– Population: children age 4-6
22

Mouse Size example cont’
• Measure task time for small and regular groups
• Quantitative Data
– Two data sets, one for small mouse and one for regular
mouse group
– For each -- average time value across 10 tasks for 20
children.
• On average, is a smaller mouse faster?
• If it is, did we prove hypothesis? NO…
23

Mouse Size Example
• Say we found on average, that children were faster
with smaller mouse.
• Null Hypothesis: Mouse size does not affect children’s
speed on target selection tasks.
• But we found, on average for 20 children that they
were faster with smaller mouse …
• So Null can’t be true … disprove Null.
• Original hypothesis is “supported” (not proved)
24

More terms … Variables
A variable is something that changes and can have
different values that can be specified or measured
Examples
Font size = 8, 10, 12 (varies & can be 8 or 10 or 12)
Colour = red, green, blue (varies & can be one of … )
Time = n seconds (n varies & can be 0 – 600 seconds)
Error rate = x% (x varies & can be 0 – 100%)
Subject type = novice or expert, male or female
25

Kinds of Variables
• Cause � Independent Variable (IV)
– The input or interface feature you have different designs
for (e.g., mouse size = small or regular)
– Characteristics of users (novice/expert)
• Effect � Dependent Variable (DV)
– the human behaviors or experiences that you measure for
each level of the IV.
– E.g., task time = 0 – 600 seconds
26

Independent Variable
• The thing experimenter change or manipulate
independent of users’ behavior to see how it affects
users’ behavior
– Often some small aspect of an interface feature
– Can also arise from grouping users (expert/novice)
• Examples
– Font size 8, 10 or 12
– Keyboard layout style phonepad or alphabetic
– Expert vs novice users
27

Dependent Variable
• A variable that depends on users’ behaviors
• The thing you measure
• So, a dependent variable is some aspect of behavior
that changes/varies and can be measured like “task
time” or “rating”
28

Hypothesis
• Related independent and dependent variables
• A change in indep var causes an effect on dep variable
• Indep var = mouse size; dep var = task time
• Hyp: A smaller mouse size improves task time for
children age 4-6 on a target selection task.
• Null: Mouse size does not affect task time …
29

• Hypothesis
Basic Form of CE for Usability Study
• Independent variable = interface aspect(s)
• Dependent variable = human performance/preference
• Sample Population
• Select levels of IV that are varied between groups
• Measure DV(s) for each group
30

About reading comprehension…
31

About reading comprehension…
32

About reading comprehension…
33

About reading comprehension…
34

This Week’s Studio
• Work through Assignment 3
• Install and run software
• Data collection
• Meet new team
35

Next Lectures
• Week 7: More on controlled experiments
– Validity and Reliability
• Week 7: Review of methods to date
• Week 8: Analysis/Statistics … how do you know from
the data sets that the DVs for groups are different?
36

Read
• Martin, Doing Experiments in Psychology
• Chapters 1,2,7,8
• Glossary
• Dix Chapter 9 Handout 9.4.2
37