Using Utility Constraints to Improve the Predictability of Conjoint ...

Sawtooth SoftwareRESEARCH PAPER SERIESUsing Utility Constraintsto Improve the Predictabilityof Conjoint AnalysisBryan K. OrmeandRichard M. Johnson,Sawtooth Software,1997© Copyright 1997 - 2002, Sawtooth Software, Inc.530 W. Fir St.Sequim, WA 98382(360) 681-2300www.sawtoothsoftware.com

A Dissenting OpinionThe May 1995 Journal of Marketing Research included an article by Allenby, Arora and Ginter(hereafter, AAG) entitled, “Incorporating Prior Knowledge into the Analysis of Conjoint Studies.” AAGreported that prohibiting sign reversals in ACA resulted in significant improvements. AAG proposed aninteresting new method using the Gibbs sampler to estimate constrained part worths. They “held out” thelast three pairs from an ACA interview for external validation. AAG measured performance with a meansquared error measure using draws from the posterior distribution of model parameters, finding themedian MSE for the held out pairs to be about 4.60 for standard ACA estimates and 3.52 whenconstraints were imposed. AAG concluded that imposing utility constraints via the Gibbs sampler hadimproved the quality of ACA utilities.Another Look at AAG’s FindingsJohnson and Pinnell (1995) examined the same data in terms of the more commonly accepted validationmeasure of holdout hit-rates. AAG provided their part worth estimates, for both standard and Bayesmethods. Johnson and Pinnell found that hit rates for the held out pairs were 95% for standard ACA and86% for Bayes (t=10.99). Constraints had actually been harmful to prediction. The data set also includedfour holdout choice tasks that AAG had not considered. Hit rates for those additional holdouts were83.2% for standard ACA and 83.3% for Bayes. The difference is not significant (t=.36). Johnson andPinnell concluded that the Bayes method for imposing utility constraints had not significantly improvedthe predictability of ACA utilities.It is important to note the AAG imposed order constraints for attributes such as brand (which do not havea universal order) based upon stated preferences from the priors portion of the ACA interview. Wesuspect that stated preferences might not always represent “truth” for every respondent. Somerespondents may have been confused by the stated preference question, thus providing bad informationfor use in constraints. We expect that Bayesian methods may provide modest improvement for ACA datasets when used only for constraining strong a priori attributes and look forward to more evidence of theirusefulness in the future.Suggestions for PracticeWe think it is reasonable to correct reversals for attributes with strong a priori ordering no matter theconjoint method. Our full-profile system (CVA) lets the researcher prescribe order constraints undereither OLS or monotone regression. The CBC system can impose order constraints only under the LatentClass add-on module.ACA is less susceptible to reversals than full-profile methods, but reversals still can occur. The currentversion of ACA influences, but does not strictly constrain, utility orders. We may include suchconstraints in future releases. For the time being, ACA users should be aware of the issue and examinetheir data sets. Counting reversals by respondent can provide an additional data point beyond the“correlation” recorded in the utility file for judging respondent reliability. You may find it useful todiscard the most unreliable respondents. For those cases that remain, simply tieing offending levels, ifthere are any, can be a simple yet effective remedy.3

ReferencesAllenby, Greg M., Neeraj Arora, and James L. Ginter (1995), “Incorporating Prior Knowledge into theAnalysis of Conjoint Studies,” Journal of Marketing Research, (May), 152-62.Herman, Steve and Rob Klein (1995), “Improving the Predictive Power of Conjoint Analysis,”Marketing Research, (Fall) Vol. 7 No. 4, 29-31.Johnson, Richard M. and Jonathan Pinnell (1995), “Comment on “Incorporating Prior Knowledge into theAnalysis of Conjoint Studies,” Working Paper, Sawtooth Software, Sequim, WA.Moore, William L., Raj B. Myhta and Teresa M. Pavia (1994), “A Simplified Method of ConstrainedParameter Estimation in Conjoint Analysis,” Marketing Letters 5:2, 173-81.Orme, Bryan K., Mark Alpert and Ethan Christensen (1997), “Assessing the Validity of ConjointAnalysis—Continued,” Working Paper, Sawtooth Software, Sequim, WA.Srinivasan, V., Arun K. Jain, and Naresh K. Malhotra (1983), “Improving Predictive Power of ConjointAnalysis by Constrained Parameter Estimation,” Journal of Marketing Research, (November), 433-38.van der Lans, Ivo A., Dick R. Wittink, Joel Huber and Marco Vriens (1992), “Within- and Across-Attribute Constraints in ACA and Full Profile Conjoint Analysis,” Sawtooth Software ConferenceProceedings, 365-79.4