Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

More documents

Recommendations

Info

MINI REVIEW ARTICLEpublished: 28 August 2012doi: 10.3389/fpsyg.2012.00322Statistical assumptions of substantive analyses across thegeneral linear model: a mini-reviewKim F. Nimon*Learning Technologies, University North Texas, Denton, TX, USAEdited by:Jason W. Osborne, Old DominionUniversity, USAReviewed by:Anne C. Black, Yale University Schoolof Medicine, USAMegan Welsh, University ofConnecticut, USACherng-Jyh Yen, Old DominionUniversity, USA*Correspondence:Kim F. Nimon, Learning Technologies,University North Texas, 3940 NorthElm Street, G150 Denton, 76207 TX,USA.e-mail: kim.nimon@gmail.comThe validity of inferences drawn from statistical test results depends on how well datameet associated assumptions.Yet, research (e.g., Hoekstra et al., 2012) indicates that suchassumptions are rarely reported in literature and that some researchers might be unfamiliarwith the techniques and remedies that are pertinent to the statistical tests they conduct.This article seeks to support researchers by concisely reviewing key statistical assumptionsassociated with substantive statistical tests across the general linear model. Additionally,the article reviews techniques to check for statistical assumptions and identifies remediesand problems if data do not meet the necessary assumptions.Keywords: assumptions, robustness, analyzing data, normality, homogeneityThe degree to which valid inferences may be drawn from theresults of inferential statistics depends upon the sampling techniqueand the characteristics of population data. This dependencystems from the fact that statistical analyses assume that sample(s)and population(s) meet certain conditions. These conditions arecalled statistical assumptions. If violations of statistical assumptionsare not appropriately addressed, results may be interpretedincorrectly. In particular, when statistical assumptions are violated,the probability of a test statistic may be inaccurate, distorting TypeI or Type II error rates.This article focuses on the assumptions associated with substantivestatistical analyses across the general linear model (GLM),as research indicates they are reported with more frequencyin educational and psychological research than analyses focusingon measurement (cf. Kieffer et al., 2001; Zientek et al.,2008). This review is organized around Table 1, which relateskey statistical assumptions to associated analyses and classifiesthem into the following categories: randomization, independence,measurement, normality, linearity, and variance. Note that theassumptions of independence, measurement, normality, linearity,and variance apply to population data and are tested byexamining sample data and using test statistics to draw inferencesabout the population(s) from which the sample(s) wereselected.RANDOMIZATIONA basic statistical assumption across the GLM is that sample dataare drawn randomly from the population. However, much socialscience research is based on unrepresentative samples (Thompson,2006) and many quantitative researchers select a samplethat suits the purpose of the study and that is convenient (Gallet al., 2007). When the assumption of random sampling is notmet, inferences to the population become difficult. In this case,researchers should describe the sample and population in sufficientdetail to justify that the sample was at least representative ofthe intended population (Wilkinson and APA Task Force on StatisticalInference, 1999). If such a justification is not made, readersare left to their own interpretation as to the generalizability of theresults.INDEPENDENCEAcross GLM analyses, it is assumed that observations are independentof each other. In quantitative research, data often do notmeet the independence assumption. The simplest case of nonindependentdata is paired sample or repeated measures data. Inthese cases, only pairs of observations (or sets of repeated data) canbe independent because the structure of the data is by design paired(or repeated). More complex data structures that do not meet theassumption of independence include nested data (e.g., employeeswithin teams and teams within departments) and cross-classifieddata (e.g., students within schools and neighborhoods).When data do not meet the assumption of independence, theaccuracy of the test statistics (e.g., t, F, χ 2 ) resulting from a GLManalysis depends on the test conducted. For data that is paired(e.g., pretest-posttest, parent-child), paired samples t test is anappropriate statistical analysis as long as the pairs of observationsare independent and all other statistical assumptions (see Table 1)are met. Similarly, for repeated measures data, repeated measuresANOVA is an appropriate statistical analysis as long as sets ofrepeated measures data are independent and all other statisticalassumptions (see Table 1) are met. For repeated measures and/ornon-repeated measures data that are nested or cross-classified,multilevel modeling (MLM) is an appropriate statistical analyticstrategy because it models non-independence. Statistical tests thatdo not model the nested or cross-classified structure of data willlead to a higher probability of rejecting the null hypotheses (i.e.,www.frontiersin.org August 2012 | Volume 3 | Article 322 | 71
NimonStatistical assumptionsTable 1 | Statistical assumptions associated with substantive analyses across the general linear model.Statistical analysis a Independence Measurement Normality Linearity VarianceLevel of variable(s) bDependentIndependentCHI-SQUARESingle sample Independent observations Nominal+ N/A2+ samples Independent observations Nominal+ Nominal+T -TESTSingle sample Independent observations Continuous N/A UnivariateDependent sample Independent paired observations Continuous N/A UnivariateIndependent sample Independent observations Continuous Dichotomous Univariate Homogeneity of varianceOVA-RELATED TESTSANOVA Independent observations Continuous Nominal Univariate Homogeneity of varianceANCOVA c Independent observations Continuous Nominal Univariate ̌ Homogeneity of varianceRM ANOVA Independent repeated observations Continuous Nominal (opt.) Multivariate ̌ SphericityMANOVA Independent observations Continuous Nominal Multivariate ̌ Homogeneity of covariance matrixMANCOVA c Independent observations Continuous Nominal Multivariate ̌ Homogeneity of covariance matrixREGRESSIONSimple linear Independent observations Continuous Continuous Bivariate ̌Multiple linear Independent observations Continuous Continuous Multivariate ̌ HomoscedasticityCanonical correlation Independent observations Continuous Continuous Multivariate ̌ HomoscedasticityaAcross all analyses, data are assumed to be randomly sampled from the population. b Data are assumed to be reliable. c ANCOVA and MANCOVA also assumeshomogeneity of regression and continuous covariate(s). Continuous refers to data that may be dichotomous, ordinal, interval, or ratio (cf. Tabachnick and Fidell, 2001).Type I error) than if an appropriate statistical analysis were performedor if the data did not violate the independence assumption(Osborne, 2000).Presuming that statistical assumptions can be met, MLM canbe used to conduct statistical analyses equivalent to those listedin Table 1 (see Garson, 2012 for a guide to a variety of MLManalyses). Because multilevel models are generalizations of multipleregression models (Kreft and de Leeuw, 1998), MLM analyseshave assumptions similar to analyses that do not model multileveldata. When violated, distortions in Type I and Type II error ratesare imminent (Onwuegbuzie and Daniel, 2003).MEASUREMENTRELIABILITYAnother basic assumption across the GLM is that population dataare measured without error. However, in psychological and educationalresearch, many variables are difficult to measure and yieldobserved data with imperfect reliabilities (Osborne and Waters,2002). Unreliable data stems from systematic and random errors ofmeasurement where systematic errors of measurement are “thosewhich consistently affect an individual’s score because of someparticular characteristic of the person or the test that has nothingto do with the construct being measured (Crocker and Algina,1986, p. 105) and random errors of measurement are those which“affect an individual’s score because of purely chance happenings”(Crocker and Algina, 1986, p. 106).Statistical analyses on unreliable data may cause effects to beunderestimated which increase the risk of Type II errors (Onwuegbuzieand Daniel, 2003). Alternatively in the presence of correlatederror, unreliable data may cause effects to be overestimated whichincrease the risk of Type I errors (Nimon et al., 2012).To satisfy the assumption of error-free data, researchers mayconduct and report analyses based on latent variables in lieu ofobserved variables. Such analyses are based on a technique calledstructural equation modeling (SEM). In SEM, latent variables areformed from item scores, the former of which become the unitof analyses (see Schumacker and Lomax, 2004 for an accessibleintroduction). Analyses based on latent-scale scores yield statisticsas if multiple-item scale scores had been measured withouterror. All of the analyses in Table 1 as well as MLM analyses can beconducted with SEM. The remaining statistical assumptions applywhen latent-scale scores are analyzed through SEM.Since SEM is a large sample technique (see Kline, 2005),researchers may alternatively choose to delete one or two items inorder to raise the reliability of an observed score. Although “extensiverevisions to prior scale dimensionality are questionable. . . oneor a few items may well be deleted” in order to increase reliability(Dillon and Bearden, 2001, p. 69). The process of item deletionshould be reported, accompanied by estimates of the reliability ofthe data with and without the deleted items (Nimon et al., 2012).MEASUREMENT LEVELTable 1 denotes measurement level as a statistical assumption.Whether level of measurement is considered a statistical assumptionis a point of debate in statistical literature. For example,proponents of Stevens (1946, 1951) argue that the dependent variablein parametric tests such as t tests and analysis-of-variancerelated tests should be scaled at the interval or ratio level (MaxwellFrontiers in Psychology | Quantitative Psychology and Measurement August 2012 | Volume 3 | Article 322 | 72
Page 2 and 3:
FRONTIERS COPYRIGHTSTATEMENT© Copy
Page 4 and 5:
Table of Contents05 Is Data Cleanin
Page 7 and 8:
OsborneAssumptions and data cleanin
Page 9 and 10:
ORIGINAL RESEARCH ARTICLEpublished:
Page 12 and 13:
Hoekstra et al.Why assumptions are
Page 14 and 15:
Page 16 and 17:
Page 18 and 19:
ORIGINAL RESEARCH ARTICLEpublished:
Page 20 and 21:
García-PérezStatistical conclusio
Page 22 and 23: García-PérezStatistical conclusio
Page 30 and 31: Sheng and ShengEffect of non-normal
Page 42 and 43: REVIEW ARTICLEpublished: 12 April 2
Page 44 and 45: Nimon et al.The assumption of relia
Page 56 and 57: TressoldiPower replication unreliab
Page 58 and 59: TressoldiPower replication unreliab
Page 60 and 61: ORIGINAL RESEARCH ARTICLEpublished:
Page 62 and 63: FinchModern methods for the detecti
Page 74 and 75: NimonStatistical assumptionsand Del
Page 76 and 77: NimonStatistical assumptionsFor exa
Page 78 and 79: Kraha et al.Interpreting multiple r
Page 94 and 95: SmithsonComparing moderation of slo
Page 102 and 103: REVIEW ARTICLEpublished: 01 March 2
Page 104 and 105: Flora et al.Factor analysis assumpt
Page 122 and 123:
Flora et al.Factor analysis assumpt
Page 124 and 125:
Kasper and ÜnlüAssumptions of fac
Page 126 and 127:
Page 128 and 129:
Page 130 and 131:
Page 132 and 133:
Page 134 and 135:
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Lages and JaworskaHow predictable a
Page 146 and 147:
Page 148 and 149:
Page 150 and 151:
Page 152 and 153:
Cummiskey et al.Testing assumptions
Page 154 and 155:
Page 156 and 157:
Page 158:
show all

Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

Create successful ePaper yourself

Delete template?

Save as template?