Learning toExperimentRay Duch explains theappeal <strong>of</strong> experimentalresearch in the socialsciences, <strong>and</strong> introducesthe work <strong>of</strong> the Centrefor Experimental SocialSciences (CESS) atNuffield CollegeExperimental research has experienceda significant renaissance in the socialsciences. It is no longer confinedto psychology but has increasingly beenembraced by other social science disciplines(including economics, political science <strong>and</strong>sociology). The results <strong>of</strong> experimentalresearch can be found informing decisionmaking in the retail industry, finance,health care, <strong>and</strong> government. For example,experiments that measure skin conductanceresponse <strong>and</strong> hormonal data have beenconducted on pr<strong>of</strong>essional traders <strong>of</strong>financial instruments in order to underst<strong>and</strong>the physiological basis for risk aversion.Field experiments have been conducted inrural India in order to help design insurancepolicies for agriculture crops. And laboratoryexperiments have been developed toassess the impact on vote choice <strong>of</strong> differentrace-based campaign ads. This increasedinterest in experimental research by socialscientists led to the founding <strong>of</strong> the Centrefor Experimental Social Sciences (CESS) atNuffield College in 2008, with funding providedby Nuffield Governing Body for a five-yearperiod.What has inspired the interest inexperimentation? The overriding factor is theconcern with claims <strong>of</strong> causality – or causalpriority – made in social science, specificallythe extent to which we can have confidencethat the causal variable <strong>of</strong> interest in a typicalempirical model is independent <strong>of</strong> confoundingfactors. If subjects (aka individuals) areassigned in a truly r<strong>and</strong>om fashion totreatments <strong>of</strong> interest then we can be confidentthat any variation in behaviour (typicallychoice in social science experiments) acrossthe treatments can be attributed to treatmenteffects rather than any other confoundingvariables (such as education or social class).A classic example from Political Scienceconcerns the effect <strong>of</strong> media messages <strong>and</strong>campaign contact on political behavior, beit voting turnout, vote choice, or campaigncontributions. Work in this area was traditionallybased on survey data – respondents reportingthe contact they had with campaign workers ortheir exposure to media messages. However,a now classic study <strong>of</strong> voter turnout by AlanGerber <strong>and</strong> Donald Green – ‘The Effects <strong>of</strong>Canvassing’ in American Political ScienceReview (2000) – pointed out that the regressionmodels using these kinds <strong>of</strong> survey-basedself-reports were very likely to generatespurious relationships between contact <strong>and</strong>voting turnout. The problem here is that eligible8Photo: Keiko Ikeuchi
voters are not r<strong>and</strong>omly assigned to votermobilisation campaign efforts. Clearly inthe ‘observational data’ world, the politicallyengaged, <strong>and</strong> hence those most likely to turnout <strong>and</strong> vote, are the voters who are morelikely to be exposed to these campaignspromoting voter turnout. The ‘treatment’here is confounded with the ‘effect’ (that is,voter turnout). The traditional approach todealing with such problems associated withobservational data is to introduce variablesinto the estimation model designed to controlfor these effects. Gerber <strong>and</strong> Green (<strong>and</strong>others) argued that control variables in modelssuch as these were not a convincing way <strong>of</strong>accounting for confounding effects.The Gerber <strong>and</strong> Green study nicely illustrateshow these problems <strong>of</strong> confounding variables<strong>and</strong> causality might be overcome byemploying experimental techniques. Theyimplemented an innovative field experiment inwhich 30,000 individuals were identified usingvoter registration lists (containing considerableinformation on partisan registration <strong>and</strong> pastvoting behaviour). These individuals werer<strong>and</strong>omly assigned to different types <strong>of</strong>mobilisation efforts – mailings, telephone, <strong>and</strong>personal contact. This r<strong>and</strong>om assignment<strong>of</strong> the treatments makes it much more likely,compared with observational data, thatthe type <strong>of</strong> mobilisation effort is effectivelyexogenous in their regression estimation.A similar concern to boost confidence in thecausal variable <strong>of</strong> interest, has led politicalscientists to show increasing interest inthe experimental laboratory. One <strong>of</strong> theareas in which this is relevant involvesunderst<strong>and</strong>ing the effect <strong>of</strong> different votingrules or mechanisms for aggregatingindividual preferences. (This example isdrawn from Rebecca Morton <strong>and</strong> KennethWilliams, Experimental Political Science <strong>and</strong>the Study <strong>of</strong> Causality.) A researcher mightbe interested in testing a formal model thatpredicts that in plurality rule elections wherethere are no majority requirements, votersmay be more inclined to vote strategicallyfor their second preferred c<strong>and</strong>idate whenthere are more than two c<strong>and</strong>idates than isthe case in plurality elections with majorityrequirements. The researcher testing thismodel has two major challenges. First, shehas to induce rational behaviour on thepart <strong>of</strong> the subjects, <strong>and</strong> this is typicallyaccomplished by financial incentives (at CESSwe require that all experiments include subjectfinancial incentives) that induce preferenceorderings over the c<strong>and</strong>idates. Secondly, thelab setting allows the investigator to createthe appropriate institutions or voting rulesthat correspond to the theoretical model – thetreatments in effect. And it is these institutionaltreatments, over which the investigator has fullcontrol, to which the subjects are r<strong>and</strong>omlyassigned.Lab experiments are one <strong>of</strong> the principalactivities <strong>of</strong> the Nuffield CESS. The CESSexperimental lab, located in George StreetMews, consists <strong>of</strong> 25 experimental stations.Currently the lab has a subject pool <strong>of</strong> 3,000– about 75 percent are students <strong>and</strong> theremainder are non-students from the Oxfordarea. In a typical term the CESS lab might host10 experiments amounting to 60 experimentalsessions.One <strong>of</strong> the principal criticisms <strong>of</strong> traditional labexperiments is that they may not generalise toa general population (<strong>of</strong> consumers, voters,business managers, <strong>and</strong> so on). The CESSinitiative recognises this possible shortcoming,<strong>and</strong> also promotes experimental research thatis not conducted in the traditional experimentallab setting. These include field experiments,virtual lab experiments, <strong>and</strong> experimentsembedded in traditional public opinion <strong>and</strong>internet panel surveys.Virtual experimental ‘sessions’ take placewherever the subject has access to the“What has inspiredthe interest inexperimentation?The overridingfactor is the concernwith claims <strong>of</strong>causality ... made insocial science.”internet. Hence, subject recruitment is notconfined to a defined narrow subject pool asis the case with most traditional labs but canbe as broad as any individual in the worldwith internet access. Advances in computing<strong>and</strong> communication have made designing<strong>and</strong> implementing these experiments simpler<strong>and</strong> less expensive. Subjects participatingin virtual lab experiments are typically drawnfrom convenience samples (that is, nonrepresentativesamples) although one c<strong>and</strong>esign virtual subject sampling frames thatare representative <strong>of</strong> a population.At CESS we have conducted these virtualexperiments both with subjects from the CESSsubject pool, <strong>and</strong> with more representativesamples <strong>of</strong> the general population.CESS is actively engaged in implementingexperiments that are embedded in large-scaleinternet surveys. Strictly speaking, the latterare not a r<strong>and</strong>om probability sample <strong>of</strong> thepopulation. However, in a growing number<strong>of</strong> countries it is now possible to constructinternet samples that, with appropriateweighting strategies, are representative <strong>of</strong>the general population. The extent to whichopt-in internet samples approximate a r<strong>and</strong>omprobability sample <strong>of</strong> the population dependson the population being sampled (somepopulations have denser internet penetrationthan others), sampling design methods <strong>and</strong>weighting strategies employed. Hence, byembedding experiments in internet surveyswe can generalise to a broader population;something one cannot do with a conventionallab experiment.As part <strong>of</strong> the 2009 British Election campaign,CESS implemented a number <strong>of</strong> experimentsembedded in a six-wave internet panel surveywith over 10,000 respondents. Leadinguniversities around the world participatedin this CESS initiative. In one <strong>of</strong> the studies,together with Shanto Iyengar at Stanford, Idesigned an experiment that measures Britishattitudes about immigration. The experiment isunique in that it distinguished abstract policypreferences (Are more open immigrationpolicies generally bad for the country?) fromwillingness to admit individual immigrants,especially those deemed deserving <strong>of</strong> entryon both economic <strong>and</strong> cultural grounds.We are able to gauge attitudes towardindividual immigration situations by exposingrespondents to r<strong>and</strong>om treatments thatconsisted <strong>of</strong> images <strong>and</strong> descriptions <strong>of</strong>specific immigrant applications for entry intoBritain. The British are both more tolerant withrespect to admitting individual immigrants– although this varies by race <strong>and</strong> region <strong>of</strong>emigration – than their general policy positionswould suggest, <strong>and</strong> less tolerant in this regardthan American respondents to a similarexperiment we conducted in the US.We have also implemented the ImplicitAttitude Test (IAT) as part <strong>of</strong> British <strong>and</strong>German large scale internet surveys. IATs areexperimental vignettes designed to recoverimplicit attitudes – for example racial attitudes– that respondents are unlikely to volunteer intraditional survey questions. The experimentalprotocol was developed by psychologists atHarvard <strong>and</strong> is now widely used. Embeddingthis experiment in a large internet surveyallows social scientists to underst<strong>and</strong> betternot only the nature <strong>of</strong> racial bias in a society,but also how it affects a range <strong>of</strong> politicalpreferences <strong>and</strong> choices.The criticism levelledagainst causal inferences from modelsbased on conventional observational datahas resulted in social scientists increasinglyembracing experimental methods. TheNuffield CESS is committed to providingthe Oxford social science community withthe facilities <strong>and</strong> the advice necessary forexecuting experiments in both lab <strong>and</strong> non-labcontexts.Raymond DuchUniversity Pr<strong>of</strong>essor <strong>of</strong> Quantitative PoliticalScience, Nuffield College9
Change language