Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

More documents

Recommendations

Info

TressoldiPower replication unreliabilitythe minimum ES that is likely to be detected in a study using a givensample size.In most experimental designs, the accepted probability of makinga Type I error is α = 0.05 and the desired power is not lessthan 0.80. However, in order to define how to obtain such a levelof power, it is necessary to know the ES of the phenomena beingidentified. It is intuitive that the smaller the phenomenon, thegreater should be the means to detect it. This analogy is similar tothe signal/noise relationship. The smaller the signal, the strongermust be the means to detect it in the noise. In psychological experiments,these means are the number of participants taking part inthe study and the number of trials they are requested to perform.Given that Power = 1−β = ES ∗ √ N/SD ∗ α, if we know the estimatedES of a phenomenon, after the definition of the desiredpower and the α level, the only free parameters is N, that is thenumber of participants or trials.A review of 322 meta-analyses published before 1998, summarizing25,000 studies referred to 474 social psychological effects,reports that the mean ES reported is r = 0.21 and the mode was lessthan r = 0.10 (Richard et al., 2003). For this observed mean ES, toobtain a statistical power for independent sample t tests = >0.9,the sample size for each group should be at least 90, a numberrarely observed in the studies.Setting aside the strong criticisms of the use of NHST (Cohen,1994; Kline, 2004), a neglected aspect of this approach, is the controlof how much statistical power is necessary to detect what theresearcher aims to find 2 .This problem is not new and has already been raised bySedlmeier and Gigerenzer (1989), Cohen (1992), Bezeau andGraves (2001), and Maxwell (2004) among others. Howeverthe widespread adherence to “The Null Ritual” as discussed byGigerenzer et al. (2004), which consists in: (a) Set up a statisticalnull hypothesis of “no mean difference” or “zero correlation.” (b)Don’t specify the predictions of your research hypothesis or ofany alternative substantive hypotheses; (c) Use 5% as a conventionfor rejecting the null; (d) If significant, accept your researchhypothesis (e) Always perform this procedure, seems to preventmost researchers taking into account such fundamental statisticalparameter. In Gigerenzer et al. (2004) check, covering the years2000–2002, and encompassing with some 220 empirical articles,only nine researchers who computed the power of their tests werefound.Using the results of four recent meta-analyses related to “controversial”or “elusive” psychological phenomena, we illustrate theimportance of using the available ESs to derive the appropriatenumber of participants to achieve a power = >0.90. Only if areplication fails with this level of power, it is legitimate to raisedoubts about the reality of the phenomena under investigation.CAN SUBLIMINALLY PRESENTED INFORMATION INFLUENCEBEHAVIOR?This question is one of the most controversial questions in psychologyand it remains an intriguing and strongly debated issue. Apartfrom the controversies relating to the controls that information2 Even if power estimate is important in meta-analyses (i.e., Valentine et al., 2009),in this paper we focus only on power estimates in single studies.(priming) was effectively masked and the identification of seriousmethodological flaws which caused great doubt as to the existenceof subliminal processing, the debate is even hotter around the topicof the level of influence of this unconscious (subliminal) information.Hitherto, the debate as to whether subliminal priming reflectsgenuine semantic processing of the subliminal information or theformation of automatic S–R mappings remains unresolved.The meta-analysis of Van den Bussche et al. (2009) tried toshed light on these questions by analyzing all the available literaturebetween 1983 and December 2006. Their analysis was carriedout separately from the two most studied protocols, subliminalpriming for semantic categorization and subliminal priming forlexical decision and naming. If semantic subliminal priming facilitatedthe subsequent categorization of targets belonging to thesame semantic category, it suggests that the primes were unconsciouslycategorized and processed semantically. The same effect ispostulated if lexical decision and naming are faster or more accurateto semantically subliminal related prime–target pairs than tounrelated pairs. A naming task is similar to the lexical decision taskexcept that the targets are all words, and participants are asked toname the targets aloud.The synthesis of the main results is reported in Table 1.DOES INCUBATION ENHANCE PROBLEM SOLVING?The “incubation period” is the temporary shift away from anunsolved problem in order to allow a solution to emerge in themind of the individual, seemingly with no additional effort, afterhe or she has put the problem aside for a period of time, havingfailed in initial attempts to solve it.Among the questions under debate there is the problem ofwhether the nature of the discovery of the solution is really unconsciousand if it is qualitatively different from that used to tackleproblems that do not require such insight.The meta-analysis of Sio and Ormerod (2009) tried to shed lighton this and other related questions, analyzing all the available literaturefrom 1964 to 2007. The main findings of their meta-analysisare reported in Table 1.UNCONSCIOUS THOUGHT THEORYThe key assumption of Unconscious Thought Theory (UTT, Dijksterhuiset al., 2006) is that unconscious thought and consciousthought are characterized by different processes. That is, “unconsciousthought” processes have a relatively large capacity – hence,they allow for an optimal decision strategy in which all attributesof chosen alternatives are weighted according to their importance.These unconscious processes require time, therefore the qualityof decisions increases with the duration of unconscious thought.“Conscious thought” processes on the other hand, have a smallcapacity and therefore only allow for simplified decision makingstrategies. As summarized by Dijksterhuis et al., 2006, p. 105):“When a decision strategy warrants the careful and strict applicationof one specific rule, as in a lexicographic strategy, useconscious thought. When matters become more complicated andweighting is called for, as in the weighting strategy, use unconsciousthought.”As expected, among the questions raised by this theory, a criticalproblem is whether really optimal decisions are obtained afterFrontiers in Psychology | Quantitative Psychology and Measurement July 2012 | Volume 3 | Article 218 | 55
TressoldiPower replication unreliabilityTable 1 | Descriptive statistics of the four meta-analyses, related to Unconscious Semantic Priming, Incubation effect, UTT, and NLP with theestimated power of a typical study, the mean and 95% CI power calculated from all studies included in the meta-analysis and the number ofparticipants necessary to obtain a Power = 0.90.Phenomena Protocols Source NstudiesAveragedN × study(range)ES *(mean and95% CI)pEstimatedpower ofa typicalstudyObservedpower(mean and95% CI)EstimatedN toachievepower = 0.90SemanticprimingIncubationeffectUnconsciousthought theoryNon-localperceptionSemanticcategorizationLexical andnamingRemote VisionGanzfeldForced-choicewith normal stateof consciousnessVan denBussche etal. (2009)Van denBussche etal. (2009)Sio andOrmerod(2009)Strick et al.(2011)Milton(1997)Storm et al.(2010)Storm et al.(in press)23 20 (6–80) 0.80 ± 0.2 n.a. 0.96 0.90 (±1.7)32 33 (9–132) 0.47 ± 0.11 n.a. 0.84 0.69 (±0.4.6) 40117 31 (7–278) 0.29 ± 0.09 n.a. 0.21 0.43 (±3.3) 10092 26 (14–55) 0.22 ± 0.08 1.2 × 10 −8 0.20§ 0.19 (±0.72) 400§78 34 (1–74) 0.16 ± 0.06 6 × 10 −9 0.55 n.a. 73108 40 (7–128) 0.13 ± 0.04 8.3 × 10 −11 0.46 0.45 (±3.8) 11072 128 (12–887) 0.011 ± 0.004 5.3 × 10 −7 0.07 0.065 (±0.81) 3450*Random effect Cohen’s d; §two groups comparison.a period of distraction from deliberate conscious mental activityfor the same amount of time as would be the case had the decisionsbeen taken deliberately.The meta-analysis of Strick et al. (2011), aimed to give ananswer to this and other related questions by analyzing all theavailable evidence up to May 2011. The main findings are reportedin Table 1.NON-LOCAL PERCEPTIONNon-local perception (NLP) is based on the hypothesis that thehuman mind may have quantum like properties, that is, that someof its functions, such as perceptual abilities, reasoning, etc., maybe analyzed using quantum formalism.The main non-local properties which are studied within therealm of quantum physics and which are supported by “extraordinaryevidence” (see Genovese, 2005, 2010), are “entanglement”and“measurement interference.”The first property, entanglement,allows two or more physical objects to behave as one even if theyare separated in space and time. This “strange” property allowsa form of immediate communication of the objects’ characteristicsover distances between or among the entangled objects, ashas been observed in teleportation experiments (i.e., Bouwmeesteret al., 1997). The possibility that quantum-like properties may beobserved not only in physics but also even in biology and psychologyhas not only been studied theoretically (von Lucadouet al., 2007; Khrennikov, 2010; Walach and von Stillfried, 2011)but also experimentally (see Gutiérrez et al., 2010 for biology andBusemeyer et al., 2011, for psychology).One of the main concerns about the studies related to differentaspects of NLP, is their inconsistency in obtaining results satisfyingthe statistical cut off criteria to refute the null hypothesis that extrasensory perception does not exist, usually setting α < 0.05.This problem is recognized by some researchers involved in thisfield of research (Kennedy, 2001) and even more by all deniers ofNLP evidence (e.g., Alcock, 2003).The main findings of three meta-analysis related to three NLPdifferent protocols, Remote Vision, that is NLP using a free-choiceresponse (Milton, 1997), NLP in a Ganzfeld state using free-choiceresponse (Storm et al., 2010) and NLP in a normal state of consciousnessusing a forced-choice response (Storm et al., in press),covering all evidence available from 1964 to 1992, 1974 to 2009,and 1987 to 2010 respectively, are reported in Table 1POWER ESTIMATIONA synthesis of the descriptive statistics related to the four phenomenadescribed above is presented in Table 1 in decreasing order ofmagnitude of ESs. For each meta-analysis, the retrospective statisticalpower with α = 0.05, achieved by a typical study using themean of the number of participants of all studies included in themeta-analysis was estimated.For all but one meta-analysis, it was also possible to calculatethe mean and 95% CI post hoc power, using the number ofparticipants of each study included in the meta-analyses and theestimated random ES, setting α = 0.05.Furthermore, for each of the four psychological phenomena,the number of participants necessary to obtain a statisticalwww.frontiersin.org July 2012 | Volume 3 | Article 218 | 56
Page 2 and 3:
FRONTIERS COPYRIGHTSTATEMENT© Copy
Page 4 and 5:
Table of Contents05 Is Data Cleanin
Page 7 and 8: OsborneAssumptions and data cleanin
Page 9 and 10: ORIGINAL RESEARCH ARTICLEpublished:
Page 12 and 13: Hoekstra et al.Why assumptions are
Page 20 and 21: García-PérezStatistical conclusio
Page 30 and 31: Sheng and ShengEffect of non-normal
Page 42 and 43: REVIEW ARTICLEpublished: 12 April 2
Page 44 and 45: Nimon et al.The assumption of relia
Page 58 and 59: TressoldiPower replication unreliab
Page 62 and 63: FinchModern methods for the detecti
Page 72 and 73: MINI REVIEW ARTICLEpublished: 28 Au
Page 74 and 75: NimonStatistical assumptionsand Del
Page 76 and 77: NimonStatistical assumptionsFor exa
Page 78 and 79: Kraha et al.Interpreting multiple r
Page 94 and 95: SmithsonComparing moderation of slo
Page 102 and 103: REVIEW ARTICLEpublished: 01 March 2
Page 104 and 105: Flora et al.Factor analysis assumpt
Page 106 and 107:
Flora et al.Factor analysis assumpt
Page 108 and 109:
Page 110 and 111:
Page 112 and 113:
Page 114 and 115:
Page 116 and 117:
Page 118 and 119:
Page 120 and 121:
Page 122 and 123:
Page 124 and 125:
Kasper and ÜnlüAssumptions of fac
Page 126 and 127:
Page 128 and 129:
Page 130 and 131:
Page 132 and 133:
Page 134 and 135:
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Lages and JaworskaHow predictable a
Page 146 and 147:
Page 148 and 149:
Page 150 and 151:
Page 152 and 153:
Cummiskey et al.Testing assumptions
Page 154 and 155:
Page 156 and 157:
Page 158:
show all

Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

Create successful ePaper yourself

Delete template?

Save as template?