Rules of Engagement - PMRG

Rules of Engagement:The war against poorly engaged respondents;Guidelines for eliminationSteve Gittelman and Elaine TrimarchiSampleSolutions, LLC.

We need to better understand the two sides of respondentengagement….Respondents become engaged in our questionnairesand then move on in their lives.We become married to their data.Is it a bad marriage?

Madama ButterflyGiacomo PucciniThe marriage between a sailor and a geisha.Those marriages never last.

If we pass along bad data to anuninformed client we become entangled ina bad marriage.Perhaps we should consider breaking offthe engagement before wedding day.Engagements are complicated. They canbe deeply rooted in tradition.

The Burden is ours….Poorly engaged respondents provide us withquestionable data.…How do we decide who we shall retain inour data sets and who shall be removed?…It is an age old question.

A biblical perspectiveThink of Noah and the Ark.All those souls crammed into such a small space.He brought the animals into the ark two bytwo…he had to choose who would stay and whowould go.But Noah had his share of problems, let’s listen.

It is no wonder….…our questionnaires are long andboring.We ask so many questions.Perhaps, too many.

If we had the chance to ask God thequestions we ask of ourrespondents,would he answer?

We need to establish guidelines….…for sample cleaning of the poorlyengaged.

Applying the rules….• We examined 50 online studies our firm conducted. Foreach we successfully created post-hoc qualityengagement metrics with four or more variables in 41 ofthe 50 studies examined.• No modification of the questionnaires was allowed…andno questions were added to create an engagementmodel.• The remaining 9 questionnaires had 1 to 3 variables.

We believe that everyone can do it.• It is even easier if a littleadvance effort is invested.• But we won’t go there. Weknow how expensivequestionnaire real estate can be.

The tools of the trade….the engagement variables.• Speeding - Less than one half the median time.• Open-Ends - Less than one half the median numberof words.• Trap Questions – requests that the respondentprovide a specific answer.• Inconsistencies - pairs of logically inconsistentquestions.• Straight Lining (Non-differentiation) - An unusuallytightly grouped response rate.• Rare Items - Often absent from the original draft of asurvey, common and rare items are groupedtogether, respondents who indicate they own asubstantive number of rare products fail the test.

Inconsistencies: ExampleStrongly DisagreeStrongly AgreeMy health has improved over the past 30 days: 1 2 3 4 5My health has declined over the past 30 days: 1 2 3 4 5

An example of a rare items testDo you or anyone in your household currentlytake any of the following medications?Vicodin yes noPrilosec yes noClaritan yes noPlavix yes noJakafi yes noKalydeco yes noStablon yes noDormalin yes noThe first four items are fairly common, and thelast four are experimental drugs or treatmentsfor rare diseases. Those that mention evenone of the rare items tend to commit otherquality faults.

Rare Items Test.Respondents failing the rare items test show shortercompletion times (t-test, p

Creating a MetricNo particular combination of variables appearsto work best, differences between questionnairesappear to drive our choice.We seek diversity in the measures used.We prefer to base our decisions on at least fourmeasures.

Who shall stay and who shall go?Engagement Fault + Meaningful Change in Data = Deletion

Study Examples1. Menopause Symptom Relief2. Antihistamine Satisfaction3. Upper Respiratory Treatments4. Antihistamine IHUT (In Home Usage Test)

Menopause SymptomRelief

Menopause Symptom Relief Study___________________________

Antihistamine SatisfactionStudy

Antihistamine Study_______________________________________

Upper Respiratory TreatmentStudy

Upper Respiratory Treatment Study_________________________

Antihistamine IHUT(In Home Usage Test)

Antihistamine IHUT Callback___________________

Antihistamine IHUT Screener________________________________

Grounding our engagement metrics:_________________________What does it all mean?• We must be on the watch for drift in our sample frame.• Tying into metrics grounds our studies.• Importantly, we need to know if the deletion of sample brings usmore or less accuracy.

Using data from an online double opt in panel, n=5000 we examinedthe relationship between quality metrics and drift (toward or awayfrom) various benchmarks.

Quality & Health Benchmarks:______________________________Last Routine Checkup

Quality & Health Benchmarks:________________________Exercise

Quality & Health Benchmarks:_____________________________Smoke Everyday

Quality & Health Benchmarks:_____________________________General Health

Quality & Health Benchmarks:_____________________________BMI

Buyer Behavior___________________________________________

Have faith…we can change.We choose between tragedyand comedy.…and for those who can’t……

Many thanks for…Your engagementand attentiveness.Steve Gittelman, Ph.D.Email: Steve@MktgInc.com