Online Conjoint Analysis: - YouGov

Online Conjoint Analysis:
The faster, the worse?
Sabine Pützfeld (psychonomics AG)
Dr. Torsten Melles (psychonomics AG)
Bielefeld, March 22 nd 2006

Agenda
I. Introduction: Characteristics of Conjoint
Analysis
II.
Research objectives
III. Methodology
IV. Reliability during survey progress
V. Effects of age and product category
VI. Conclusions and directions for future research
2

4 basic characteristics of Conjoint Analysis (I)
1. Respondents evaluate multiattributive product
stimuli.
paired comparison of car 1 and car 2
3

4 basic characteristics of Conjoint Analysis (II)
2. Multiple decisions are
made.
4

4 basic characteristics of Conjoint Analysis (III)
3. Product attributes are
combined systematically.
5

4 basic characteristics of Conjoint Analysis (IV)
4. The utility of each attribute level („partworth“) is statistically
derived from all judgments.
Partworth 1
+
+
Partworth 2
Partworth 3
Overall decision
6

Agenda
I. Introduction: Characteristics of Conjoint
Analysis
II.
Research objectives
III. Methodology
IV. Reliability during survey progress
V. Effects of age and product category
VI. Conclusions and directions for future research
7

Background: duration per paired comparison in seconds
duration per paired
comparison in seconds
45
40
35
30
25
20
15
10
5
0
warm-up
1 5 10 15 20 25
survey progress (paired comparisons)
On average, the first 5 to 6
paired comparisons of a
conjoint analysis take more
time.
Independent of the total
number of paired
comparisons, the duration
of the following paired
comparisons is nearly at the
same constant level.
study 1 (n=528)
study 2 (n=163)
study 3 (n=303)
study 4 (n=306)
study 5 (n=262)
8

Problem Formulation
Effect of exhaustion? – Reliability
of the answers declines
or
Effect of routine? – Reliability of
the answers does not decline
9

Agenda
I. Introduction: Characteristics of Conjoint
Analysis
II.
Research objectives
III. Methodology
IV. Reliability during survey progress
V. Effects of age and product category
VI. Conclusions and directions for future research
10

Methodology: Database
Peer analysis of 5 Online Conjoint Analyses:
(1) „checking account“ – November 2004 – 528 respondents from
online panel – 27 paired comparisons (2 versions)
(2) „wall paints“ – January 2005 – 163 respondents from online panel
and recruitment by telephone – 18 paired comparisons
(3) „mobile phone price plans“ – January 2005 – 303 respondents
from online panel – 16 paired comparisons
(4) „building society savings“ – February 2005 – 306 respondents
from online panel – 20 paired comparisons
(5) „car study“ – March 2005 – 262 respondents from online panel –
27 paired comparisons
Order of the paired comparisons was
randomized in each study
11

Methodology: Analysis of reliability
Relying on the partworths (derived from responses to all paired comparisons) for
each respondent, an individual pattern of preference will be generated.
At a second stage, the partworths are used to calculate the theoretically correct
answer of each paired comparison, provided that the statements are totally
consistent.
perfect consistency of answer = response on a paired comparison corresponds
to the one calculated on the basis of the individual pattern of preference
Measure of reliability: consistency of answer I
correlation between response on a paired comparison corresponds to the one
calculated on the basis of the individual pattern of preference
Measure of reliability: consistency of answer II
(squared) deviation of response on a paired comparison corresponds to the one
calculated on the basis of the individual pattern of preference
12

Agenda
I. Introduction: Characteristics of Conjoint
Analysis
II.
Research objectives
III. Methodology
IV. Reliability during survey progress
V. Effects of age and product category
VI. Conclusions and directions for future research
13

Consistency of answer during survey progress I
(correlation)
consistency of answer
(correlation between actual given answer and
calculated answer)
1
0,9
0,8
0,7
0,6
warm-up effect
Except for the first paired
comparison the correlations
of the considered studies
are on a constantly high
(0.84 – 0.92) level of
consistency.
No decline of consistency
during survey progress
Effect of exhaustion cannot
be confirmed
0,5
1 5 10 15 20 25
survey progress (paired comparisons)
study 1 (n=528)
study 2 (n=163)
study 3 (n=303)
study 4 (n=306)
study 5 (n=262)
14

Analysis of consistency of answers
Relying on the partworths (derived from responses to all paired comparisons) for
each respondent an individual pattern of preference will be generated.
On a second step the partworths are used to calculate the theoretically correct
answer of each paired comparison, provided that the statements are totally
consistent.
perfect consistency of answer = response on a paired comparison corresponds
to the one calculated on the basis of the individual pattern of preference
Measure of reliability: consistency of answer I
correlation between response on a paired comparison corresponds to the one
calculated on the basis of the individual pattern of preference
Measure of reliability: consistency of answer II
shows, how far the
answer differs from
predicted value
(squared) deviation of response on a paired comparison corresponds to the one
calculated on the basis of the individual pattern of preference
15

Consistency of answer during survey progress II
(deviation)
first 6 paired comparisons of all considered studies
6%
5%
rel. frequencies
4%
3%
2%
1%
0%
-3 0 3
deviation of actual given answer and
calculated answer
16

Consistency of answer during survey progress II
(deviation)
last 6 paired comparisons of all considered studies
6%
5%
rel. frequencies
4%
3%
2%
1%
0%
-3 0 3
deviation of actual given answer and
calculated answer
17

Consistency of answer during survey progress II
(deviation)
first and last 6 paired comparisons of all
considered studies
6%
No significant differences
while comparing the
distributions of the
deviations regarding the
first and the last 6 paired
comparisons
rel. frequencies
5%
4%
3%
2%
Answers of both survey
parts represent the same
deviation from predicted
value
Effect of exhaustion cannot
be confirmed
1%
0%
-3 0 3
deviation of actual given answer and
calculated answer
18

Agenda
I. Introduction: Characteristics of Conjoint
Analysis
II.
Research objectives
III. Methodology
IV. Reliability during survey progress
V. Effects of age and product category
VI. Conclusions and directions for future research
19

Further questions
Based on all respondents, no effects of the consistency
of answer could be determined, which indicate a kind of
exhaustion during progress of paired comparisons.
Does the consistency of answer depend on the age of
the respondent resp. do older people become
exhausted earlier than younger people?
Does the progress of consistency of answer depend on
the product that is tested during the conjoint analysis?
20

Age effects within consistency of answer
1,6
1,4
older respondents
answer less consistently
on principle
Averaged squared deviations of
all paired comparisons
older than 45 years
1,17
squared deviations
1,2
1
0,8
0,6
0,4
0,2
0
1 5 10 15 20 25
survey progress (paired comparisons)
up to 25 years (n=427)
26 to 35 years (n=535)
younger respondents
answer more consistently
on principle
36 to 45 years (n=351)
older than 45 years (n=244)
36 to 45 years
26 to 35 years
up to 25 years
21
0,77
1,05
0,99
An effect of exhaustion
cannot be confirmed in any
age group

Further questions
Based on all respondents no effects of the consistency
of answer could be determined, which indicate a kind of
exhaustion during progress of paired comparisons.
Does the consistency of answer depend on the age of
the respondent resp. do older people become
exhausted earlier than younger people?
Does the progress of consistency of answer depend on
the product that is tested during the conjoint analysis?
22

Differentiation according to the tested product
squared deviations
2,5
2
1,5
1
0,5
car study
checking account
Consistency of answer
depends on the tested
product (resp. on complexity).
However no differences
during the survey progress
can be determined. That
means no effects of
exhaustion could be found in
any of the researched
studies.
0
1 5 10 15 20 25
survey progress (paired comparisons)
study 1 (n=528)
study 2 (n=163)
study 3 (n=303)
study 4 (n=306)
study 5 (n=262)
23

Agenda
I. Introduction: Characteristics of Conjoint
Analysis
II.
Research objectives
III. Methodology
IV. Reliability during survey progress
V. Effects of age and product category
VI. Conclusions and directions for future research
24

Conclusion
The analyses indicate, that a decline in reliability during survey progress
of an online conjoint analysis does not appear at all.
Even after 27 paired comparisons, the answers of the respondents are
as consistent as at the beginning of the survey. That applies to
different age groups
different tested products
On the other hand a clear warm-up effect occurs, which confirms the
results of former studies. The respondents require 2 to 3 tasks to reach
the maximum of reliability within theirs statements.
As long as no systematic deviation of the preference statement exists in
this context (but merely a lower reliability), it is not necessary to exclude
the first paired comparisons from the utility estimation. Consequently, it
is not necessary to include special warm-up tasks.
25

Directions for future research
The reliability of the conjoint analysis results has been found to depend
on the age of the respondents and on the tested product. That means:
In order to draw conclusions with the same reliability, different amounts of
data (case number, number of statements per respondent) according to the
age group and to the product to be tested will be necessary. Therefore a
further quantification would be reasonable.
Considering age, one should investigate whether there is a possibility of
creating the tasks for the older people in a different way, so that the answers
will be more reliable.
Considering product category, reasons for differences in reliability should be
explored. Reasons may be e.g. the familiarity with the product or the
involvement or the motivation to participate in the survey.
26

Thank you for your attention!