Media Mix Modeling vs. ANCOVA - Merkle

Media Mix Modeling vs. ANCOVAAn Analytical Debate

What is the best way to measureincremental sales, or “lift”,generated from marketinginvestment dollars?2

Measuring ROI From Promotional SpendWhere possible to implement, an experimental design thatuses a randomly selected holdout group provides the moststatistical power and reliability in marketing measurementSample Direct Marketing Campaign Plan: 2012June July August September October November DecemberTMSDM1TMSEM1TMSEM 2DM3DM4DM2EventMarket Events and External Factors3

Use Randomization to Increase Statistical Power• Holdout sample should be selected from the lowest level of the experimental designand each treatment cell should have a corresponding randomly selected holdout groupSegment 1 TargetsN=150,600Segment 2 TargetsN=350,200Segment 3 TargetsN=102,000TREATMENTGROUP (TEST)RANDOMCONTROLTREATMENTGROUP (TEST)RANDOMCONTROLTREATMENTGROUP (TEST)RANDOMCONTROLNOTE: Control group can be much smaller than treatment group. Use sample size calculator todetermine minimum possible sample size for control groupSample Size Calculator4Option to applyfinite populationcorrectionadjustment

Verify Pre-Campaign Holdout ValidityPRE PERIOD CAMPAIGN RUNS POST PERIOD• Random selection of the control group from the target group should guarantee that thetest and control groups are equivalent on key metrics prior to the campaign start, but…• Statistical testing of the difference between pre-period test and control groups on keymetrics is an important validation step and adds to confidence in the post-periodmeasurementP-Values for the Difference Between Test and Control GroupMean Values on Key Metrics in Pre-PeriodSample Metrics Group1 Group2 Group3Pre-Period Trends:Test and Control GroupsAvg. Sales$ .45 .52 .35Media Impressions .82 .84 .61Other PromotionalSpend.25 .38 .27Competitor Spend .31 .45 .24Demographics .28 .19 .245p >.15 is non-significant

Measure Pre-Post Launch Change• If the test group and control group are statistically equivalent prior to the campaignlaunch, then the difference in sales between the groups after the campaignrepresents the incremental sales contribution of the campaign• ANCOVA (Analysis of Covariance) test willmeasure the significance of the differenceand also control for other potential factorsthat could differentially impact test and controlgroups during the campaign periodPRE PERIOD CAMPAIGN POSTTestControlCampaignEffect ??Average Weekly Sales per Target:Test v Control, Pre and Post0.700.600.500.400.300.200.10-Pre-PeriodPost-PeriodGrowthFrom Preto PostTest 0.45 0.65 0.20Control 0.44 0.50 0.06Test-Control 0.01 0.15 0.14ANCOVA Adjusted 0.11CampaignEffect6Sales by Week: Test and Control GroupsANCOVA Adjusted difference is aftercontrolling for covariates and, if significant (pvalueless than .15), is the measure of trueincremental sales from the campaign

Summary: ANCOVA for Marketing MeasurementBenefits• Extremely reliable results• Conservative test• Control for other factors that may impact volume growth of target relative toholdout• Able to scale to calculate overall ROI from marketing program• Expect replicable results if same conditions and weights apply in repeatedtreatment7

There is another option…Media Mix Modeling can overcome many limitations ofANCOVA-based analysis8

Limitations of ANCOVA• Feasibility of holdout group• Opportunity cost of being out of market with incremental media• Selection of test period length is subjective• Difficult to measure mass & digital media• No guidance on cross-tactic decisions• Does not provide insight into future budget allocation decisions• Does not explain “base” factor contributionsAs we will see, Media Mix Modeling will overcome all of these limitations…9

How is Media Mix Modeling Different?Media Mix Models can be used to understand the incremental, layered effect of crosstacticmarketing over time…Incremental MediaContributionBase10

RevenueRevenueIncremental RevenueIncremental RevenueWhat are the Requirements and Process?Input DataStatistical ModelsResponse Curves &OptimizationTV3503002502001501005000 5 10 15 20 25 30TimeDirect Mail1200100080060040020000 5 10 15 20 25 30Time605040302010ytpi 1Sales DecompositionixpititTVDirect MailRadioPrintBase2001801601401201008060402000.0 24.0 48.0Investment AmountPrintRadioDirect MailTVRadio2502001501005000 5 10 15 20 25 30Time180160140120100806040200PrintDirect Mail11%Radio8%01 3 5 7 9 11 13 15 17 19 21 23Print14%Segment 1TV16%600500400300200100Base51% -Time87 7162 4942467373280 298Segment 1 Segment 2TVDirect MailRadioPrintBase2001801601401201008060402000.0 24.0 48.0Investment AmountInvestmentPrintRadioDirect MailTV0 5 10 15 20 25 30Time11Time

How Does Media Mix Modeling Work?Functional forms of model equations…Estimation of equations…12lLy tXyy01llt20 1x12x10 1x11y x ...01 2y x x ...y0exp(1102x11)kx Lx Lkˆˆˆ01yini 1( xˆˆini 11xi( xx)(yiix)2y)uiyiyiyi0 1ni 1uˆ2ini 1( yiˆ0ˆˆ1ˆxix)i2

Functional Forms of Equations• Functional form of a relationship between response and explanatory variables isdetermined by factors such as diminishing/increasing returns to scale, (a)symmetryin response, etc.• Some of the most frequently used functional forms are:Functional Form Representation Return to ScaleLinearQuadraticPower additiveMultiplicative(log-log)lLy tXyy01llt20 1x12x10 1x11 2y x x ...012x LkConstantDiminishingDiminishingDiminishing13Log-Reciprocalyexp(0x11)S-Shaped

Estimation of Equations• Example equationy 1xi0• Estimation of the ‘betas’ˆˆ01yn• Residualˆii1( xinˆi11xi( xˆiix)(yix)u2iy)uiyiyiyi0 1ˆˆPopulation equation indicating relationship betweenx and y; estimated using sample of data representingthe populationIntercept equals the sample average of y plus the sampleestimate of xSample covariance between x and y divided by the samplevariance of xxiDifference between actual and predicted, estimate ofthe unknown error in the population equation14• Sum of Squared Residualsni 1uˆ2ini 1( yiˆ0ˆ1xi)2Ordinary Least Squares estimates minimize the sumof squared residuals

Application of Parameter Estimates• How do we calculate contribution for each variable in the model?– Multiply coefficient from model (“beta”) by weekly model inputs(impressions)– Sum weekly values to get total contribution attributable to each media• Model Coefficient (“Beta”) for Display: 0.0000486431WeekDisplayImpressionsContribution5/30/2009 1,972,606 966/6/2009 2,226,734 1086/13/2009 2,483,358 121…5/7/2011 5,550,921 2705/14/2011 7,016,425 341Sum contribution acrossweeks to get totalincremental sales due toDisplay… 53,4155/21/2011 4,937,705 24015

GRPsGRPsMeasurement of time-varying impacts• “Adstock” refers to the effect of advertising extending several periods after the originalexposure• Estimate using Distributed Lag Model1. Estimate model with lagged effects for all media terms – coefficients represent % decay at each lag2. Smooth with estimation of gamma distribution to the lagged effect coefficients4003002001000• Estimate using various exponential decaysTVTV Decay - Step 1 Lag ModelTV Decay - Step 2 Gamma Distr1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28Week4003002001000TVAdstocktTVAdstock( TV1x0.#0)1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28Weektt16TV Adstock (20%) Adstock (40%) Adstock (60%) Adstock (80%)

Toolset of econometric methodologiesMethodologies used in MMM Analyses– Ordinary Least Squares (OLS)– Mixed (Bayesian Shrinkage, RandomCoefficients)– Unobserved Components Models(UCM)– Two Stage: UCM-Mixed– Seemingly Unrelated Regression (SUR)– Structural Equation Modeling (SEM)Time SeriesData(i.e. National x Week)OLSMethodology SelectionUCMPanel Data(i.e. DMA x Week)MixedHierarchical RelationshipsTwoStage:UCM -MixedSURSEM17

Case Study ComparisonsWhat does each approach offer in these instances?18

NRx per Physician per WeekCase Study 1: Direct Campaign• Typical multi-channel campaign to physicians with mix of tactics deployed in rapidsuccession across long timeframe• Capitalizes on use of “universal control group” of non-marketed holdout0.450.400.35EM1EM2DM20.300.250.20DM1Tele-detailDM2EM1Resend 1EM1Resend 20.150.100.05EM2Resend 1-19

NRx per Physician per WeekCase Study 1: ANCOVA APPROACH1. Define multiple pre-post periods2. Conduct holdout-validity tests for each pre-period and each set of test/control groups3. Measure ANCOVA-Adjusted change in volume using double difference0.450.400.35PRE PERIOD 2 POST PERIOD 2EM1 EM2 DM20.300.250.200.150.100.05ANCOVA:PREPERIOD1DM1Tele-detailANCOVA:POSTPERIOD1DM2EM1Resend 1EM1Resend 2EM2Resend 1-20

Case Study 1: ANCOVAAncova 1: Time Period 1 Ancova 2: Time Period 2Change in Per Physician PrescriptionVolume from Pre1 to Post 1Change in Volume fromPre2 to Post2Test +1.4 +2.5Control +0.4 -0.2Difference 1.0 2.7ANCOVA-Adjusted 0.8 2.2Significance .05 p

Case Study 1: Mixed Modeling Approach1. Use correlogram approach fittedwith gamma curves to calculatedecay curves per channel2. Transform input variables toaccount for decay3. Build model at the physicianweeklevel over 130 weeks ofhistory and all physicians,whether targeted or not incampaign4. Fit model using best functionalform5. Calculate response curves foreach tactic6. Input into planning tool foroptimization22

Case Study 1: Campaign Planning From MMO Output• MMO equation creates outputs that can be used in a scenario planning tool to testthe impact of different investment levels by tactic and calculate expected ROI fromvarying budget levels23… and ANCOVA confirmed lift estimates

Case Study 2: Web Support Program• Situation: Launched consumer support website where consumers register onlinefor product support and information. Consumers only provide zip code in onlinesupport registration. Sales not able to be tied directly to consumers but only togeography (zip code)• Key question: Does consumer support program drive future sales?ANCOVA Approach:1) Match consumer registered zip codeto most likely purchase zip code2) Identify control zip codes with noconsumer registrations in proximity3) Test “lift” after web program launches0.200.150.100.05-Volume per household per month – Pre LaunchSep-09 Oct-09 Nov-09 Dec-09 Jan-10 Feb-10ControlTestWeb Visits per HH per MonthMarket Share – Pre Launch0.500.400.300.200.10-Sep-09 Oct-09 Nov-09 Dec-09 Jan-10 Feb-10ControlTest15%10%5%0%Sep-09 Oct-09 Nov-09 Dec-09 Jan-10 Feb-10ControlTest24

Case Study 2: Possible ANCOVA OutputANCOVA may demonstrate a link between sales and web support program use• Test pre-post period differencesbetween zip codes with registrationsand with no registrations• Control for covariates that mightinfluence test zip codes25

Case Study 2: Mixed Model Approach1. Collect zip-level data on all programs in place, by week, over long time period2. Calculate contribution of each of the tactics, including the web registrations3. Compare relative contribution to sales and relative ROI levels of each tacticInput Model ROIModel Output: Quadratic FormEffect Estimate StdErr tValue ProbtIntercept 0.126 0.009 13.433 0.000log_trend 0.025 0.004 6.719 0.000total_mkt_grp 0.411 0.012 35.333 0.000sq_total_mkt_grp (0.139) 0.005 -27.309 0.000decay_register 5.037 0.755 6.672 0.000sq_decay_register (8.051) 2.934 -2.744 0.006decay_activation1 7.056 0.550 12.820 0.000sq_decay_activation1 (4.106) 1.586 -2.589 0.010decay_activation2 0.603 0.077 7.823 0.000sq_decay_activation2 0.052 0.040 1.297 0.194Baseline 52%TV GRP 25% 2:1Registrations 3% 4:1Activation 1 6% 3:1Activation 2 7% 6:1Promotion 1 7% 1:126Effect Estimate StdErr tValue ProbtIntercept 0.138 0.009 14.6

Case Study 3: Cross-Tactic MeasurementSituation: Large advertising spend – objective is tooptimize spend by tactic and geography27Base43%Contribution %FY 2010Incremental57%Media mix modeling indicates incrementalityof media along with indication of ROI acrosstactics…Spend – Incremental Sales$200M FY 20115%120 10%8%7% 28019%StreamingVideo55%5%20%5018015030%9%32%Spend % Increm. Sales %ROI IndexNewspaperDirect MailTVRadioDisplay

Case Study 3: MMM provides insights intopromotional performance by regionMedia mix modeling indicates promotional messaging is more effective in theMidwest than all other regions…100% Promo Tactics by Region90%80%303FY 201170% 1231227611480 101 81 6760%50%40%30%20%10%0%3% 4% 3%17%13% 11% 13%9%12%7% 9% 10% 12% 11% 12% 14%6%449%4% 4%28% Cost % Increm Sales ROI Index

A Best Practice Approach.. Taking marketing measurement to the next level29

Comparison of Methods = Winner on thisAttributeANCOVAMedia Mix ModelingCostHidden CostsComplexity of ExecutionData Requirements (Depends on # groups)Cost of withholding promotionfrom controlStatistics simpler, but testdesign more complexMeasurement Ability Scenario Planning Only to repeat exact Best for...30

Best Practice Measurement FrameworkMedia Mix Modeling gives best practice estimates of media impacts – both overall and at the vehicle level. Themethodology is also extensible to the tactic level, and can be applied in cases where indirect or direct attributionis not feasible. Indirect/Direct Attribution is best employed in relative analyses within a media vehicle, at levels ofgranularity not possible via traditional mix modeling (i.e. search keywords).MeasurementLevel:Legend:CategoryMediaVehicleTVMediaDRTV1-800DisplayBaseSearchDirectMeasure usingMedia MixModelingTactic•Creative•Daypart•Duration•Creative•Daypart•Duration•Creative•Site•LandingPage•Site•Keyword•Creative•SegmentMeasure usingIndirect/DirectAttribution• Last-click• In-market testing(ANCOVA)• Ad tracking31

MerkleAnalyticsBrian DemitrosAssociate DirectorIntegrated Media Optimization PracticeMerkle Analytics443.542.4438BDemitros@merkleinc.comLynda S GordonSenior DirectorLife Sciences Analytics PracticeMerkle Analytics440.476.0351LSGordon@merkleinc.com