Lages <strong>and</strong> JaworskaHow predictable are voluntary decisions?APPENDIXTable A1 | Replication study 1: left/right key press in spontaneous motor decision task.Subject Resp. bias left/right Bino. test p-value No of Runs Runs test p-value Lilliefors p-value SVM pred acc[1] 21/11 0.1102 11 0.1194 0.0957 [0.5875]2 16/16 1.0 17 1.0 0.1659 0.39733 13/19 0.3771 16 1.0 0.1116 0.2672[4] 11/21 0.1102 15 1.0 0.0017** [0.4970]5 14/18 0.5966 26 0.0009*** 0.4147 0.82186 15/17 0.8601 15 0.6009 0.0893 0.5607[7] 22/10* 0.0501(*) 17 0.4906 >0.5 [0.4259]8 20/12 0.2153 18 0.5647 >0.5 0.6295[9] 10/22* 0.0501(*) 15 1.0 0.001*** [0.2845]10 18/14 0.5966 21 0.1689 0.0936 0.6789[11] 31/1*** 0.0001*** 3 1.0 >0.5 [NA]12 19/13 0.3771 20 0.2516 0.4147 0.579013 20/12 0.2153 22 0.0280(*) 0.3476 0.726714 14/18 0.5966 21 0.1689 0.1659 0.609815 16/16 1.0 21 0.2056 0.1116 0.6330[16] 4/28*** 0.0001*** 5 0.0739 0.0017*** [0.7513]17 17/15 0.8601 22 0.0989 >0.5 0.7026[18] 11/21 0.1102 16 0.9809 0.0893 [0.2985]19 12/20 0.2153 7 0.00092*** 0.001*** 0.7891[20] 11/21 0.1102 16 0.9809 0.0936 [0.2844]Tot/Avg 4.25 # 4 (2) 16.2 3 (2) 4 (4) 0.5539Participant excluded according to Soon et al.’s (2008) response criterion (binomial test p < 0.11); # average deviation: Σ i |x i –n/2|/N for n = 32 <strong>and</strong> N = 20.*p < 0.05, **p < 0.01, ***p < 0.001; ( ·) number of violations adjusted for multiple tests after Sidak–Dunn.Table A2 | Replication study 2: addition/subtraction in hidden intention task.Subject Resp. bias add/sub Bino. test p-value No of Runs Runs test p-value Lilliefors p-value SVM pred acc1 11/21 0.1102 9 0.0184(*) 0.0112(**) 0.77252 14/18 0.5966 10 0.0215(*) 0.0271(*) 0.71243 11/21 0.1102 8 0.0057(**) 0.1153 0.69914 14/18 0.5966 9 0.0071(**) 0.001*** 0.78475 21/11 0.1102 12 0.2405 0.001*** 0.59326 2/30*** 0.0001*** 5 1.0 >0.5 N/A7 17/15 0.8601 18 0.8438 0.3267 0.60748 20/12 0.2153 12 0.1799 0.0016** 0.59049 6/26*** 0.0005*** 4 0.0006*** 0.4443 0.835110 18/14 0.5966 16 0.9285 0.2313 0.397211 9/23* 0.0201(*) 13 0.8488 0.4030 0.443412 9/23* 0.0201(*) 10 0.1273 >0.5 0.6131Tot/Avg 5.33 # 4 (2) 10.5 5 (1) 5 (3) 0.6408*p < 0.05, **p < 0.01, ***p < 0.001; # average deviation: Σ i |x i –n/2|/N for n = 32 <strong>and</strong> N = 12. ( ·) Number of violations adjusted for multiple tests after Sidak–Dunn.<strong>Frontiers</strong> in Psychology | Quantitative Psychology <strong>and</strong> Measurement March 2012 | Volume 3 | Article 56 | 149
METHODS ARTICLEpublished: 25 September 2012doi: 10.3389/fpsyg.2012.00354Using classroom <strong>data</strong> to teach students about <strong>data</strong><strong>cleaning</strong> <strong>and</strong> <strong>testing</strong> assumptionsKevin Cummiskey 1 *, Shonda Kuiper 2 <strong>and</strong> Rodney Sturdivant 11Department of Ma<strong>the</strong>matical Sciences, United States Military Academy, West Point, NY, USA2Department of Ma<strong>the</strong>matics <strong>and</strong> Statistics, Grinnell College, Grinnell, IA, USAEdited by:Jason W. Osborne, Old DominionUniversity, USAReviewed by:Steven E. Stemler, WesleyanUniversity, USAFern<strong>and</strong>o Marmolejo-Ramos,University of Adelaide, AustraliaMartin Dempster, Queen’s UniversityBelfast, UK*Correspondence:Kevin Cummiskey, Department ofMa<strong>the</strong>matical Sciences, UnitedStates Military Academy, West Point,NY, USA.e-mail: kevin.cummiskey@usma.eduThis paper discusses <strong>the</strong> influence that decisions about <strong>data</strong> <strong>cleaning</strong> <strong>and</strong> violations ofstatistical assumptions can have on drawing valid conclusions to research studies. The<strong>data</strong>sets provided in this paper were collected as part of a National Science Foundationgrant to design online games <strong>and</strong> associated labs for use in undergraduate <strong>and</strong> graduatestatistics courses that can effectively illustrate issues not always addressed in traditionalinstruction. Students play <strong>the</strong> role of a researcher by selecting from a wide variety of independentvariables to explain why some students complete games faster than o<strong>the</strong>rs.Typicalproject <strong>data</strong> sets are “messy,” with many outliers (usually from some students taking muchlonger than o<strong>the</strong>rs) <strong>and</strong> distributions that do not appear normal. Classroom <strong>testing</strong> of <strong>the</strong>games over several semesters has produced evidence of <strong>the</strong>ir efficacy in statistics education.The projects tend to be engaging for students <strong>and</strong> <strong>the</strong>y make <strong>the</strong> impact of <strong>data</strong><strong>cleaning</strong> <strong>and</strong> violations of model assumptions more relevant. We discuss <strong>the</strong> use of oneof <strong>the</strong> games <strong>and</strong> associated guided lab in introducing students to issues prevalent in real<strong>data</strong> <strong>and</strong> <strong>the</strong> challenges involved in <strong>data</strong> <strong>cleaning</strong> <strong>and</strong> dangers when model assumptionsare violated.Keywords: Guided Interdisciplinary Statistics Games <strong>and</strong> Labs, messy <strong>data</strong>, model assumptionsINTRODUCTIONThe decisions that researchers make when analyzing <strong>the</strong>ir <strong>data</strong> canhave significant impacts on <strong>the</strong> conclusions of a scientific study.In most cases, methods exist for checking model assumptions,but <strong>the</strong>re are few absolute rules for determining when assumptionsare violated <strong>and</strong> what to do in those cases. For example,when using t -tests or ANOVA, decisions about normality, equalvariances, or how to h<strong>and</strong>le outliers are often left to <strong>the</strong> discretionof <strong>the</strong> researcher. While many statistics courses discuss modelassumptions <strong>and</strong> <strong>data</strong> <strong>cleaning</strong> (such as removing outliers or erroneous<strong>data</strong>), students rarely face <strong>data</strong> analysis challenges where<strong>the</strong>y must make <strong>and</strong> defend decisions. As a result, <strong>the</strong> impacts of<strong>the</strong>se decisions are rarely discussed in detail.The topics of <strong>data</strong> <strong>cleaning</strong> <strong>and</strong> <strong>testing</strong> of assumptions are particularlyrelevant in light of <strong>the</strong> fact that <strong>the</strong>re have been severalhigh-profile retractions of articles published in peer-reviewed psychologyjournals because of <strong>data</strong> related issues. In June 2012, Dr.Dirk Smeesters resigned from his position at Erasmus University<strong>and</strong> had a paper retracted from <strong>the</strong> Journal of Experimental Psychologyafter it was determined his <strong>data</strong> was statistically highlyunlikely. He admitted to removing some <strong>data</strong> points that did notsupport his hypo<strong>the</strong>sis, claiming that this practice is commonin psychology <strong>and</strong> marketing research (Gever, 2012). Simmonset al. (2011) show that even when researchers have good intentions,<strong>the</strong>y control so many conditions of <strong>the</strong> experiment that<strong>the</strong>y are almost certain to show statistically significant evidencefor <strong>the</strong>ir hypo<strong>the</strong>sis in at least one set of conditions. These conditionsinclude <strong>the</strong> size of <strong>the</strong> sample, which outliers are removed,<strong>and</strong> how <strong>the</strong> <strong>data</strong> is transformed. They argue that <strong>the</strong> ambiguityof how to make <strong>the</strong>se decisions <strong>and</strong> <strong>the</strong> researcher’s desire toobtain significant results are <strong>the</strong> primary reasons for <strong>the</strong> largenumber of false positives in scientific literature (Simmons et al.,2011).Replication studies are designed to ensure <strong>the</strong> integrity of scientificresults <strong>and</strong> may help detect issues associated with <strong>the</strong> <strong>data</strong> <strong>and</strong>model assumptions. However, replication is not a panacea. Miller(2012) shows that <strong>the</strong> probability of replicating a significant effectis essentially unknowable to <strong>the</strong> researcher so <strong>the</strong> scientific communitymay not always correctly interpret replication study results.Fur<strong>the</strong>ring <strong>the</strong> difficulty in assessing <strong>the</strong> quality of researchers’decisions involving <strong>data</strong> is <strong>the</strong> fact that relatively few replicationstudies are done on published research. Journals prefer to publishoriginal research <strong>and</strong> rarely publish replications of previous studieseven if <strong>the</strong> replication shows no effect. Therefore, <strong>the</strong>re is littleincentive for researchers to replicate o<strong>the</strong>rs’ results which increases<strong>the</strong> likelihood that studies resulting in false positives are acceptedinto scientific journals. Dr. Brian Nosek <strong>and</strong> a group working on<strong>the</strong> ambitious Reproducibility Project are attempting to replicateevery study published in three major psychology journals in 2008(Barlett, 2012).Through student generated <strong>data</strong>sets, we demonstrate how studentsin <strong>the</strong> same class, with <strong>the</strong> same raw <strong>data</strong>set, <strong>and</strong> using <strong>the</strong>same statistical technique can draw different conclusions withoutrealizing <strong>the</strong> assumptions <strong>the</strong>y made in <strong>the</strong>ir analysis. There ismuch truth to Esar’s (1949) humorous saying “Statistics [is] <strong>the</strong>only science that enables different experts using <strong>the</strong> same figuresto draw different conclusions.” Instead of having our studentsbelieve that statistics is simply a set of step-by-step calculations,www.frontiersin.org September 2012 | Volume 3 | Article 354 | 150
Change language