Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

García-PérezStatistical conclusion validitywhich the COAST rule was reportedly used, although unintendedly.And not a single citation is to be found in WoS from papersreporting the use of the extensions and modifications of Botellaet al. (2006) or Ximenez and Revuelta (2007). Perhaps researchersin psychology invariably use fixed sampling, but it is hard to believethat “data peeking” or “data monitoring” was never used, or thatthe results of such interim analyses never led researchers to collectsome more data. Wagenmakers (2007, p. 785) regretted that “itis not clear what percentage of p values reported in experimentalpsychology have been contaminated by some form of optionalstopping. There is simply no information in Results sections thatallows one to assess the extent to which optional stopping hasoccurred.” This incertitude was quickly resolved by John et al.(2012). They surveyed over 2000 psychologists with highly revealingresults: Respondents affirmatively admitted to the practices ofdata peeking, data monitoring, or conditional stopping in ratesthat varied between 20 and 60%.Besides John et al.’s (2012) proposal that authors disclose thesedetails in full and Simmons et al.’s (2011) proposed list of requirementsfor authors and guidelines for reviewers, the solution tothe problem is simple: Use strategies that control Type-I errorrates upon repeated testing and optional stopping. These strategieshave been widely used in biomedical research for decades(Bauer and Köhne, 1994; Mehta and Pocock, 2011). There is noreason that psychological research should ignore them and give upefficient research with control of Type-I error rates, particularlywhen these strategies have also been adapted and further developedfor use under the most common designs in psychologicalresearch (Frick, 1998; Botella et al., 2006; Ximenez and Revuelta,2007; Fitts, 2010a,b).It should also be stressed that not all instances of repeated testingor optional stopping without control of Type-I error ratesthreaten SCV. A breach of SCV occurs only when the conclusionregarding the research question is based on the use of thesepractices. For an acceptable use, consider the study of Xu et al.(2011). They investigated order preferences in primates to findout whether primates preferred to receive the best item first ratherthan last. Their procedure involved several experiments and theydeclared that “three significant sessions (two-tailed binomial testsper session, p < 0.05) or 10 consecutive non-significant sessionswere required from each monkey before moving to the nextexperiment. The three significant sessions were not necessarilyconsecutive (. . .) Ten consecutive non-significant sessions weretaken to mean there was no preference by the monkey” (p. 2304).In this case, the use of repeated testing with optional stopping ata nominal 95% significance level for each individual test is partof the operational definition of an outcome variable used as acriterion to proceed to the next experiment. And, in any event,the overall probability of misclassifying a monkey according tothis criterion is certainly fixed at a known value that can easilybe worked out from the significance level declared for eachindividual binomial test. One may object to the value of the resultantrisk of misclassification, but this does not raise concernsabout SCV.In sum, the use of repeated testing with optional stoppingthreatens SCV for lack of control of Type-I and Type-II errorrates. A simple way around this is to refrain from these practicesand adhere to the fixed sampling assumptions of statistical tests;otherwise, use the statistical methods that have been developed foruse with repeated testing and optional stopping.PRELIMINARY TESTS OF ASSUMPTIONSTo derive the sampling distribution of test statistics used in parametricNHST, some assumptions must be made about the probabilitydistribution of the observations or about the parameters ofthese distributions. The assumptions of normality of distributions(in all tests), homogeneity of variances (in Student’s two-samplet test for means or in ANOVAs involving between-subjects factors),sphericity (in repeated-measures ANOVAs), homoscedasticity(in regression analyses), or homogeneity of regression slopes(in ANCOVAs) are well known cases. The data on hand may ormay not meet these assumptions and some parametric tests havebeen devised under alternative assumptions (e.g., Welch’s test fortwo-sample means, or correction factors for the degrees of freedomof F statistics from ANOVAs). Most introductory statisticstextbooks emphasize that the assumptions underlying statisticaltests must be formally tested to guide the choice of a suitable teststatistic for the null hypothesis of interest. Although this recommendationseems reasonable, serious consequences on SCV arisefrom following it.Numerous studies conducted over the past decades have shownthat the two-stage approach of testing assumptions first and subsequentlytesting the null hypothesis of interest has severe effects onType-I and Type-II error rates. It may seem at first sight that this issimply the result of cascaded binary decisions each of which has itsown Type-I and Type-II error probabilities; yet, this is the result ofmore complex interactions of Type-I and Type-II error rates thatdo not have fixed (empirical) probabilities across the cases that endup treated one way or the other according to the outcomes of thepreliminary test: The resultant Type-I and Type-II error rates ofthe conditional test cannot be predicted from those of the preliminaryand conditioned tests. A thorough analysis of what factorsaffect the Type-I and Type-II error rates of two-stage approachesis beyond the scope of this paper but readers should be aware thatnothing suggests in principle that a two-stage approach might beadequate. The situations that have been more thoroughly studiedinclude preliminary goodness-of-fit tests for normality beforeconducting a one-sample t test (Easterling and Anderson, 1978;Schucany and Ng, 2006; Rochon and Kieser, 2011), preliminarytests of equality of variances before conducting a two-sample ttest for means (Gans, 1981; Moser and Stevens, 1992; Zimmerman,1996,2004; Hayes and Cai,2007),preliminary tests of both equalityof variances and normality preceding two-sample t tests for means(Rasch et al., 2011), or preliminary tests of homoscedasticity beforeregression analyses (Caudill, 1988; Ng and Wilcox, 2011). Theseand other studies provide evidence that strongly advises againstconducting preliminary tests of assumptions. Almost all of theseauthors explicitly recommended against these practices and hopedfor the misleading and misguided advice given in introductorytextbooks to be removed. Wells and Hintze (2007, p. 501) concludedthat “checking the assumptions using the same data thatare to be analyzed, although attractive due to its empirical nature,is a fruitless endeavor because of its negative ramifications on theactual test of interest.” The ramifications consist of substantial butFrontiers in Psychology | Quantitative Psychology and Measurement August 2012 | Volume 3 | Article 325 | 20