Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

Nimon et al.The assumption of reliable dataa table, as illustrated in the Appendix. Readers can also consultPajares and Graham (1999) as a guide for presenting data.WHAT DO WE DO IN THE PRESENCE OF UNRELIABLE DATA?Despite the best-laid plans and research designs, researchers will attimes still find data with poor reliability. In the real-world problemof conducting analyses on unreliable data, researchers are facedwith many options which may include: (a) omitting variables fromanalyses, (b) deleting items from scale scores, (c) conducting“whatif” reliability analyses, and (d) correcting effect sizes for reliability.OMITTING VARIABLES FROM ANALYSESYetkiner and Thompson (2010) suggested that researchers omitvariables (e.g., depression, anxiety) that exhibit poor reliabilityfrom their analyses. Alternatively, researchers may choose to conductSEM analyses in the presence of poor reliability wherebylatent variables are formed from item scores. The former becomethe units of analyses and yield statistics as if multiple-item scalescores had been measured without error. However, as noted byYetkiner and Thompson, reliability is important even when SEMmethods are used, as score reliability affects overall fit statistics.DELETING ITEMS FROM SCALE SCORESRather than omitting an entire variable (e.g., depression, anxiety)from an analysis, a researcher may choose to omit one or moreitems (e.g., BDI-1, BAI-2) that are negatively impacting the reliabilityof the observed score. Dillon and Bearden (2001) suggestedthat researchers consider deleting items when scores from publishedinstruments suffer from low reliability. Although “extensiverevisions to prior scale dimensionality are questionable ...one ora few items may well be deleted” in order to increase reliability(Dillon and Bearden, p. 69). Of course, the process of item deletionshould be documented in the methods section of the article.In addition, we suggest that researchers report the reliability of thescale with and without the deleted items in order to add to thebody of knowledge of the instrument and to facilitate the abilityto conduct RG studies.CONDUCTING “WHAT IF” RELIABILITY ANALYSESOnwuegbuzie et al. (2004) proposed a “what if reliability” analysisfor assessing the statistical significance of bivariate relationships.In their analysis, they suggested researchers use Spearman’s (1904)correction formula and determine the “minimum sample sizeneeded to obtain a statistically significant r based on observedreliability levels for x and y” (p. 236). They suggested, for example,that when r OX O Y= 0.30, r xx = 0.80, r yy = 0.80, r TX T Y, based onSpearman’s formula, yields 0.38 (0.30/ √ (0.80 × 0.80)) and “thatthis corrected correlation would be statistically significant with asample size as small as 28” (p. 235).Underlying the Onwuegbuzie et al. (2004) reliability analysis,presumably, is the assumption the error is uncorrelated in thepopulation and sample. However, even in the case that such anassumption in tenable, the problem of “what if reliability” analysisis that the statistical significance of correlation coefficients thathave been adjusted by Spearman’s formula cannot be tested forstatistical significance (Magnusson, 1967). As noted by Muchinsky(1996):Suppose an uncorrected validity coefficient of 0.29 is significantlydifferent than zero at p = 0.06. Upon application of thecorrection for attenuation (Spearman’s formula), the validitycoefficient is elevated to 0.36. The inference cannot be drawnthat the (corrected) validity coefficient is now significantlydifferent from zero at p < 0.05 (p. 71).As Spearman’s formula does not fully account for the measurementerror in an observed score correlation, correlations based onthe formula have a different sampling distribution than correlationsbased on reliable data (Charles, 2005). Only in the case whenthe full effect of measurement error on a sample observed scorecorrelation has been calculated (i.e., Eq. 4 or its equivalent) caninferences be drawn about the statistical significance of r TX T Y.CORRECTING EFFECT SIZES FOR RELIABILITYIn this article we presented empirical evidence that identified limitationsassociated with reporting correlations based on Spearman’s(1904) correction. Based on our review of the theoretical andempirical literature concerning Spearman’s correction, we offerresearchers the following suggestions.First,consider whether correlated errors exist in the population.If a research setting is consistent with correlated error (e.g.,tests areadministered on the same occasion, similar constructs, repeatedmeasures), SEM analyses may be more appropriate to conductwhere measurement error can be specifically modeled. However,as noted by Yetkiner and Thompson (2010), “score reliability estimatesdo affect our overall fit statistics, and so the quality of ourmeasurement error estimates is important even in SEM” (p. 9).Second, if Spearman’s correction is greater than 1.00, do nottruncate to unity. Rather consider the role that measurement andsampling error is playing in the corrected estimate. In some cases,the observed score correlation may be closer to the true scorecorrelation than a corrected correlation that has been truncatedto unity. Additionally, reporting the actual Spearman’s correctionprovides more information than a value that has been truncatedto unity.Third, examine the difference between the observed score correlationand Spearman’s correction. Several authors have suggestedthat a corrected correlation “very much higher than the originalcorrelation” (i.e., 0.85 vs. 0.45) is “probably inaccurate” (Zimmerman,2007, p. 938). A large difference between an observedcorrelation and corrected correlation “could be explained by correlatederrors in the population, or alternatively because error arecorrelated with true scores or with each other in an anomaloussample” (Zimmerman, 2007, p. 938).Fourth,if analyses based on Spearman’s correction are reported,at a minimum also report results based on observed score correlations.Additionally, explicitly report the level of correlation errorthat is assumed to exist in the population.CONCLUSIONIn the present article, we sought to help researchers understandthat (a) measurement error does not always attenuate observedscore correlations in the presence of correlated errors, (b) differentsources of measurement error are cumulative, and (c) reliabilityis a function of data, not instrumentation. We demonstrated thatreliability impacts the magnitude and statistical significance testsFrontiers in Psychology | Quantitative Psychology and Measurement April 2012 | Volume 3 | Article 102 | 50