Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

More documents

Recommendations

Info

Cummiskey et al.Testing assumptions with classroom dataTable 4 | Candidate independent variables to explain Tangramsperformance.VariableGenderAcademic majorType of high school attendedAthletePolitical affiliationAcademic performanceResearch questionDo males or females perform better attangrams?Do students majoring in science,technology, engineering, and mathematicsperform better at tangrams than otherstudents?Do students who attended private or publichigh schools perform better at tangrams?Do college athletes perform better attangrams than non-athletes?Do students who affiliate with thedemocratic, republican, or other partiesperform better at tangrams?Do students who made the dean’s listperform better at tangrams than those thatdid not?many more dependent variables that can be investigated. Table 5contains a list of some of the other dependent variables that canbe investigated using Tangrams.In more advanced courses, small groups of student researchershave used these online games to develop more complex researchstudies. For example, one group built upon Butler and Baumeister’s(1998) findings that a friendly observer (compared to neutralor hostile observers) had a detrimental effect on the ability of participantsto accurately complete difficult tasks. They conducted astudy with a repeated measures design to determine whether thiseffect would be the same if the friendly observer actively offeredadvice on how to solve Tangrams puzzles.We have developed a series of online games and associated labslike the one discussed in this paper. Multiple independent variablescan be used to test memory or spatial reasoning skills. Studentscan choose variables (such as using hints, amount of time allowed,or number of pieces) that are very likely to result in small p-valueseven with small sample sizes. Other independent variables, suchas gender or major, are less likely to have small p-values. Whenmultiple studies are conducted, we typically find no more thanone or two of the studies will show significant differences amonggender. This can lead to very interesting discussions of publicationbias that can occur when only research studies with smallp-values are published. Each game-based lab or research projectallows for quick, anonymous, and automated data collection thatcan be used to demonstrate the importance of impacts of datacleaning and model assumptions on the results of a study.STUDENT COMMENTS AND ASSESSMENTIn statistics education, it is often challenging to have studentsexperience the true complexities of conducting a large researchstudy. Designing an experiment, collecting and analyzing data,and deciding upon and carrying out the appropriate statisticaltest are usually too time-consuming, costly, or impractical for anTable 5 | Candidate dependent variables.VariablePuzzle completion timePuzzle success or failureNumber of movesTime to quitTime to receive a hintNumber of puzzles solvedDescriptionTime to complete a tangrams puzzleGiven a fixed amount of time, whether ornot a student can complete the puzzleNumber of moves (a flip or rotation)required to solve the puzzleTime before a student quits a puzzle thatis impossible to solveTime until a student asks the game for ahintGiven a fixed amount of time, the numberof puzzles that a student can solveintroductory statistics course. As a result, most students learn statisticalcalculations without a tie to the context of the scientificresearch and become disinterested in, and even cynical toward,statistics. An alternative to lectures and textbook style problems isto incorporate research-like experiences in the classroom.The Classroom Undergraduate Research Experience (CURE)survey of undergraduate science evaluated courses that containresearch-like experiences. Research-like experiences are activitiesthat contain “Group work, reading primary literature, data collectionand analysis. . . students conduct research in which theoutcome is not known (even to the course instructor) and studentshave at least some input into the research topic and design ofthe methodological approach.” Results of the CURE survey showthat “Students in high research-like courses report learning gainssimilar in kind and degree to gains reported by students in dedicatedsummer research programs” (Lopatto, 2010). In addition,Wei and Woodin (2011) found “Evidence is emerging that theseapproaches are introducing more underrepresented minorities toscientific research in the classroom.” These game-based labs aredesigned so that, with technology, students can gain many of thebenefits of research-like experiences.A formal assessment of student attitudes and learning was conductedthroughout the 2011–2012 school year. These materialswere taught in five sections of an introductory statistics course.When we asked students what they liked best about the lab, typicalresponses were:• The data set was real and we played a part in creating it.• To be able to see an actual scenario where what we learned canbe used.• The fact that we could collaborate. . .• . . .work at own pace, ask questions as needed.• I liked that getting the data was very quick and easy.• Playing the game!As a group, students enjoyed playing the games. Even thoughthe online game is fairly simple with plain graphics, it was considereda welcome break from normal classroom activities. The levelof interest in trying to explain potential biases in Tangrams performancewas very high. Ideas ranged from number of hours of sleepto SAT scores to the age of the player. This activity also seemedFrontiers in Psychology | Quantitative Psychology and Measurement September 2012 | Volume 3 | Article 354 | 155
Cummiskey et al.Testing assumptions with classroom dataTable 6 | Survey results for 115 students after completing the Tangrams lab.Survey questionStronglyagree (%)Agree(%)Neutral(%)Disagree(%)Stronglydisagree (%)The Tangrams lab was a good way of learning about hypothesis testing 43 38 8 7 3Students who do not major in science should not have to take statistics courses 5 10 23 37 24Statistics is essentially an accumulation of facts, rules, and formulas 10 34 30 19 6Creativity plays a role in research 30 47 12 7 4If an experiment shows that something does not work, the experiment was a failure 9 2 5 31 52The tangrams lab had a possible effect on my interest in statistics 17 38 32 13 1to truly engage students who were otherwise quiet throughout thesemester.Many students commented that they liked using “real” data forthe first time in their course. This comment came as a surprisebecause the instructors had used data from economics, sports, scientific,and military websites in lessons prior to this lab. However,to the students, if they are not involved in the collection of thedata, it is not real to them. Involving students in data collectionmakes them much more interested in the outcome of a statisticalprocess. In addition, messy data makes the decision process morereal to students.In many courses using the lab, students had yet to activelyexperience the context for the statistical procedures they werelearning. They had only seen textbook type questions that givethem the research question, the experiment, the data, and the statisticalprocedure to use. After completing the lab, many studentscommented that they saw how statistical procedures are actuallyused by people outside the statistics classroom. Survey results suggestthat students enjoyed the lab and felt like they had learnedfrom the experience. In this assessment, 81% of students eitheragreed or strongly agreed that the Tangrams lab was a good wayof learning about hypothesis testing, while only 10% disagreed.seventy-four percent either agreed or strongly agreed that theTangrams lab improved their understanding of using statistics inresearch. Complete results of the survey are displayed in Table 6.Although our students have laptop computers and are requiredto bring them to class, the Tangrams lab has been implementedin other classroom conditions. In large sections, where each studentdoes not have a laptop, students can play the game outsideof class in preparation for the next class period. If no computersare available in class, the guided labs that we have available aredetailed enough to allow students to do most of the computationalwork outside the classroom, where most students presumably haveaccess to a computer. The instructor can then use class time todiscuss results and interpretations of findings.CONCLUSIONOnline games and guided labs such as Tangrams are fun andeffective ways to incorporate a research-like experience into anintroductory course in data analysis or statistics. The labs leveragetheir natural curiosity and desire to explain the world around themso they can experience both the power and limitations of statisticalanalysis. They are an excellent way for instructors to transitionfrom textbook problems that focus on procedures to a deeperlearning experience that emphasizes the importance of properexperimental design and understanding assumptions. While playingthe role of a researcher, students are forced to make decisionsabout outliers and possibly erroneous data. They experience messydata that make model assumptions highly questionable. These labsgive students the context for understanding and discussing issuessurrounding data cleaning and model assumptions, topics that aregenerally overlooked in introductory statistics courses.ACKNOWLEDGMENTSSample materials and datasets are freely available at http://web.grinnell.edu/individuals/kuipers/stat2labs/. A project-based textbookthat incorporates some of these games is also available athttp://www.pearsonhighered.com/kuiper1einfo/index.html. Thesematerials were developed with partial support from the NationalScience Foundation, NSF DUE# 0510392, and NSF TUES DUE#1043814. The original games were developed by Grinnell Collegestudents under the direction of Henry Walker and Sam Rebelsky.John Jackson and William Kaczynski of the United States MilitaryAcademy provided significant input to the classroom materials.REFERENCESBarlett, T. (2012). Is psychology aboutto come undone. The Chronicleof Higher Education. Available at:http://chronicle.com/blogs/percolator/is-psychology-about-to-comeundone/29045(accessed April 17,2012).Box, G. E. P. (1953). Non-normalityand tests on variances. Biometrika40, 333.Brown, M. B., and Forsythe, A.B. (1974). Robust tests for equalityof variances, J. Am. Stat. Assoc. 69,364–367.Butler, J. L., and Baumeister, R. F.(1998). The trouble with friendlyfaces: skilled performance with asupportive audience. J. Pers. Soc.Psychol. 75, 1213–1230.Carling, K. (2000). Resistant outlierrules and the non-Gaussiancase. Comput. Stat. Data Anal. 33,249–258.DeVeaux, R., and Velleman, R. (2008).Math is music: statistics is literature.Amstat News 375, 54.Esar, E. (1949). “The dictionaryof humorous quotations,” inMathematics 436 – Finely Explained(2004). ed. R. H. Shutler, 3. Victoria:Trafford Publishing.Garfield, J., Aliaga, M., Cobb, G., Cuff,C., Gould, R., Lock, R., Moore,T., Rossman, A., Stephenson, B.,Utts, J., Velleman, P., and Witmer,J. (2007). Guidelines for Assessmentand Instruction in Statistics Education(GAISE): College Report.Alexandria: American StatisticalAssociation.Gever, J. (2012). Stats can tripup scientific data fabricators.MedPage Today. Availableat: http://www.medpagetoday.com/PublicHealthPolicy/Ethics/33655(accessed July 07, 2012).Hesterberg, T. (2008). “It’s timeto retire the ‘n>=30’ Rule,” inProceedings of the American StatisticalAssociation, StatisticalComputing Section (CD-ROM)(Alexandria: American StatisticalAssociation).Kruschke, J. (2011). Doing BayesianData Analysis. Amsterdam: AcademicPress.Levene, H. (1960). “Robust Testsfor Equality of Variances,” inContributions to Probability andwww.frontiersin.org September 2012 | Volume 3 | Article 354 | 156
Page 2 and 3:
FRONTIERS COPYRIGHTSTATEMENT© Copy
Page 4 and 5:
Table of Contents05 Is Data Cleanin
Page 7 and 8:
OsborneAssumptions and data cleanin
Page 9 and 10:
ORIGINAL RESEARCH ARTICLEpublished:
Page 12 and 13:
Hoekstra et al.Why assumptions are
Page 14 and 15:
Page 16 and 17:
Page 18 and 19:
Page 20 and 21:
García-PérezStatistical conclusio
Page 22 and 23:
Page 24 and 25:
Page 26 and 27:
Page 28 and 29:
Page 30 and 31:
Sheng and ShengEffect of non-normal
Page 32 and 33:
Page 34 and 35:
Page 36 and 37:
Page 38 and 39:
Page 40 and 41:
Page 42 and 43:
REVIEW ARTICLEpublished: 12 April 2
Page 44 and 45:
Nimon et al.The assumption of relia
Page 46 and 47:
Page 48 and 49:
Page 50 and 51:
Page 52 and 53:
Page 54 and 55:
Page 56 and 57:
TressoldiPower replication unreliab
Page 58 and 59:
TressoldiPower replication unreliab
Page 60 and 61:
Page 62 and 63:
FinchModern methods for the detecti
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
MINI REVIEW ARTICLEpublished: 28 Au
Page 74 and 75:
NimonStatistical assumptionsand Del
Page 76 and 77:
NimonStatistical assumptionsFor exa
Page 78 and 79:
Kraha et al.Interpreting multiple r
Page 80 and 81:
Page 82 and 83:
Page 84 and 85:
Page 86 and 87:
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
Page 94 and 95:
SmithsonComparing moderation of slo
Page 96 and 97:
Page 98 and 99:
Page 100 and 101:
Page 102 and 103:
REVIEW ARTICLEpublished: 01 March 2
Page 104 and 105:
Flora et al.Factor analysis assumpt
Page 106 and 107: Flora et al.Factor analysis assumpt
Page 124 and 125: Kasper and ÜnlüAssumptions of fac
Page 144 and 145: Lages and JaworskaHow predictable a
Page 152 and 153: Cummiskey et al.Testing assumptions
Page 154 and 155: Cummiskey et al.Testing assumptions
Page 158: Cummiskey et al.Testing assumptions
show all

Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?