Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

More documents

Recommendations

Info

Cummiskey et al.Testing assumptions with classroom dataport” (Box, 1953). Levene’s test of homogeneity of variance is lesssensitive to departures from normality (Levene, 1960; Brown andForsythe, 1974). For the cleaned data, Levene’s test gives a p-valueof 0.023, indicating there is evidence of unequal variances.A commonly used informal test is to reject the equal varianceassumption when the ratio of SD (largest SD over the smallestSD) is greater than 2 (Moore and McCabe, 2003). Using this ruleof thumb on our data, the ratios for the raw and cleaned data are1.02 and 1.45, respectively. Using this informal rule, we fail to rejectthe assumption of equal variances for both the raw and cleaneddata.Whether or not the researcher decides to assume the two populationshave equal variances will contribute to the choice of whichstatistical test to perform and has a surprisingly large impact onthe p-value of the two sample t -test. Without assuming equalvariances, the p-value of the two sample t -test on the cleaned datais 0.109, which is considerably larger than the p-value of 0.058found when assuming equal variances. Note that the p-value forthe t -test assuming equal variances and the one-way ANOVA aremathematically equivalent and result in the same p-value.To explain the impacts of this equal variance assumption, weneed to recognize the influence of unequal sample sizes. Whenthe group with the smallest sample size has a larger SD, the meansquare error (or pooled SD) is likely to underestimate the truevariance. Then ANOVA is likely to incorrectly reject the nullhypothesis (conclude that there are differences when there reallyare no differences between group means).A second assumption that a researcher should validate is thatboth samples are from normally distributed populations. From theboxplots in Figure 2, a student should suspect that the populationdistributions are not normal. Additional tools such as histogramsand normal probability plots clearly show that the sample data isnot normally distributed. For both athletes and non-athletes, theShapiro–Wilks test for normality rejects the null for both sampleswith p-values less than 0.001, providing further evidence that theassumption of normality is not valid (see Table 2 for a summaryof the results of various Shapiro–Wilks tests).When faced with data that indicates the normality assumptionis not valid, transforming the data is one method to allow the analystto proceed with the analysis. In this case, taking the log of thecompletion times results in plots that appear much closer to theshape of the normal distribution 2 . Figure 3 shows a boxplot of thecleaned data after the log transformation. It is more appropriateto conduct the statistical test using the transformed data since thenormality assumption is more closely met. When the two samplet -test (unequal variances) is performed on the transformed data,the resulting p-value is 0.307 which would lead us to concludethat there is not a significant difference between athletes and nonathletes.These results are somewhat different from the p-value of0.478 obtained using the raw data, although the difference in thep-value did not change the conclusion of the test.Researchers using the cleaned data face similar issues. Even afterremoving the outliers, the cleaned data still is strongly skewed tothe right. Once again, a log transform improves the normalityassumption. Conducting the two sample t -test on the cleaned logtransformed data results in a p-value of 0.18.We have shown that even when students start their analysis withthe same raw dataset, decisions involving data cleaning and validationof model assumptions cause p-values to vary between 0.058and 0.478. Table 3 summarizes the different p-values based onthe assumptions we discussed in the last two sections. This clearlydemonstrates that model assumptions need to be checked beforeany statistical conclusions are drawn. It also shows that a researcherdetermined to find significant results can do so by choosing a setof assumptions resulting in the smallest p-value.In introductory statistics courses, this dataset can be used tofocus on tests that are based on the equal variance assumptionand the normality assumption (t-tests and ANOVA) and how theviolation of these assumptions can influence p-values as shownin Table 3. However, there are several other statistical techniquesthat are typically beyond the scope of an introductory course thatcan be discussed with this dataset. In addition to Levene’s test andthe Shapiro–Wilks test shown above, instructors could discuss thefollowing:• More advanced methods, such as the Box-Cox power transformation,can be used to find a better transformation (Osborne,2002; Olivier and Norberg, 2010).2 While the log transformation is helpful, both groups still show some evidence oflack of normality. Other, less common, transformations did better fit the normalityassumption, however, the resulting p-values were fairly similar.Table 2 | Summary of Shapiro–Wilks normality test under variousconditions.AthleteNon-athleteRaw data
Cummiskey et al.Testing assumptions with classroom dataTable 3 | Summary of p-values for the difference in means betweenTangrams completion times of athletes and non-athletes undervarious assumptions.Two-sample t-test assuming equalvariancesTwo-sample t-test withoutassuming equal variancesTwo-sample t-test assuming equalvariance using the log-transformeddataTwo-sample t-test assuming equalvariance using the log-transformeddataRawdata(p-value)Cleaned data(p-value)0.478 0.0580.475 0.1090.307 0.1390.323 0.180• Response time is often modeled with the exponentially modifiedGaussian distribution (ex-Gaussian distribution). This gametypically provides a relevant example of data that is better modeledwith a distribution other than the normal distribution(Marmolejo-Ramos and González-Burgos, 2012).• Many students believe that if the sample size is greater than30, the Central Limit Theorem guarantees that tests based onthe normal distribution are appropriate to use. Tim Hesterberghas written an easy-to-understand article that is freely availableonline that challenges this assumption (Hesterberg, 2008). Havingstudents read this article helps them understand that whilethe “sample size greater than 30” is a nice guideline, it is not anabsolute rule.• The data can be used to demonstrate better tools to enhance thevisibility of data that is not normally distributed. For example,the shifting boxplot and the violin plot with confidence intervalaround the mean provide additional information not displayedin the basic boxplot (Marmolejo-Ramos and Matsunaga, 2009;Marmolejo-Ramos and Tian, 2010).• The analysis of the Tangrams data is equally suited for coursesthat emphasize other concepts, such as Bayesian approaches todata analysis (Kruschke, 2011).• The data can be analyzed with techniques that are not basedon the normality assumption, such as randomization andpermutation tests.RANDOM SAMPLINGIn our experience, the assumptions of statistical tests such as thosediscussed in the previous section are at least covered in typical statisticstextbooks and classes. An assumption that is addressed farless often but that should always be validated before any statisticalconclusions are drawn about populations is that each observationis a sample randomly selected from the population. Homeworkstyleproblems do not give enough information about how the datawas collected for students to consider the quality of the sample.When students conduct their own experiment, the students aremore aware of this issue in the resulting data.Some sources of bias are easily identifiable. For example, thissample gathered at West Point is most certainly not generalizableto all colleges nationwide, as only about 15% of cadets are female.In addition, it is important to discuss the impact of a researcheracting as a subject within their own research study. Other sourcesof bias are not so easily identifiable but warrant discussion. This isespecially the case with observational studies such as ours wherethe students enrolled in the course are acting as subjects in thestudy. For example, it could be possible that there are other factorsthat the athletes in our sample share that were the true reasonfor differences in their times when playing the puzzle game. Onepossibility is that the athletes have early morning practice and aretherefore more tired than the non-athletes. Also, because studentsdecided and knew which variables were being investigated, thereis the possibility of stereotype threat, where groups are primed toperform better or worse.From our experience, the following discussion questions areeffective at addressing some of the important issues involved withrandom sampling and designing experiments.• If you had a chance to play the game under different circumstances,would you be able to perform better? Describe anyfactors that may have kept you from doing your best whileplaying the game.• Do you think outside factors (like the ones you or others mentionedin the previous question) could impact the results of astudy? Should researchers provide some type of motivation fortheir subjects in various studies to do their best?• If you were conducting this study again, how would youcontrol for any key factors that may influence each subject’sperformance?• Ask small groups to design their own study, addressing other notso obvious conjectures.- Are there any variables (such as academic major or gender)that explain why some players solve Tangrams puzzles fasterthan others?- What type of student improves the most when playing thistype of game?- Is it helpful for a second student to provide hints?ADDITIONAL DATA SETSThe Tangrams game and corresponding labs are designed to beflexible so that students can design studies related to their interests.In another semester at West Point, we investigated the relationshipbetween academic major and Tangrams performance. Studentsmajoring in math, science, and engineering disciplines (MSE) werecompared to those majoring in other disciplines (non-MSE). Inthis study, the raw data resulted in a p-value of 0.353 for thetwo sample t -test. When outliers were removed from the data,the p-value decreased to 0.071. Other classes have tested twowayor three-way factorial designs. For any variables that studentsare interested in testing, the nature of the Tangrams game tendsto produce positively skewed data and outliers. Thus, any studyconducted by students with this game provides opportunities todemonstrate the importance of data cleaning and model assumptions.Table 4 contains a list of suggested independent variablesthat could be used to explain Tangrams performance.In this paper, we focused on the time to complete the puzzleas the response or dependent variable. The Tangrams game offerswww.frontiersin.org September 2012 | Volume 3 | Article 354 |154
Page 2 and 3:
FRONTIERS COPYRIGHTSTATEMENT© Copy
Page 4 and 5:
Table of Contents05 Is Data Cleanin
Page 7 and 8:
OsborneAssumptions and data cleanin
Page 9 and 10:
ORIGINAL RESEARCH ARTICLEpublished:
Page 12 and 13:
Hoekstra et al.Why assumptions are
Page 14 and 15:
Page 16 and 17:
Page 18 and 19:
Page 20 and 21:
García-PérezStatistical conclusio
Page 22 and 23:
Page 24 and 25:
Page 26 and 27:
Page 28 and 29:
Page 30 and 31:
Sheng and ShengEffect of non-normal
Page 32 and 33:
Page 34 and 35:
Page 36 and 37:
Page 38 and 39:
Page 40 and 41:
Page 42 and 43:
REVIEW ARTICLEpublished: 12 April 2
Page 44 and 45:
Nimon et al.The assumption of relia
Page 46 and 47:
Page 48 and 49:
Page 50 and 51:
Page 52 and 53:
Page 54 and 55:
Page 56 and 57:
TressoldiPower replication unreliab
Page 58 and 59:
TressoldiPower replication unreliab
Page 60 and 61:
Page 62 and 63:
FinchModern methods for the detecti
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
MINI REVIEW ARTICLEpublished: 28 Au
Page 74 and 75:
NimonStatistical assumptionsand Del
Page 76 and 77:
NimonStatistical assumptionsFor exa
Page 78 and 79:
Kraha et al.Interpreting multiple r
Page 80 and 81:
Page 82 and 83:
Page 84 and 85:
Page 86 and 87:
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
Page 94 and 95:
SmithsonComparing moderation of slo
Page 96 and 97:
Page 98 and 99:
Page 100 and 101:
Page 102 and 103:
REVIEW ARTICLEpublished: 01 March 2
Page 104 and 105: Flora et al.Factor analysis assumpt
Page 124 and 125: Kasper and ÜnlüAssumptions of fac
Page 144 and 145: Lages and JaworskaHow predictable a
Page 152 and 153: Cummiskey et al.Testing assumptions
Page 156 and 157: Cummiskey et al.Testing assumptions
Page 158: Cummiskey et al.Testing assumptions
show all

Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

Create successful ePaper yourself

Delete template?

Save as template?