Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

More documents

Recommendations

Info

Nimon et al.The assumption of reliable dataBoard, 2011; Public Agenda, 2011). We used data reported for 2011as observed data and the long-run average of SAT scores reportedsince the new form of the SAT was introduced as true scores,given the psychometric principle from Allen and Yen (1979) thatlong-run averages equal true score values (cf. Wetcher-Hendricks,2006). To compute error scores, we subtracted true scores fromobserved scores. The components of Eq. 6 applied to the SAT dataare presented in Table 2 and yield the observed score correlation(0.90), as follows:r OX O Y= 0.91 √ 0.71 × 0.65 + 0.84 √ 0.05 √ 0.06+ 0.62 √ 0.06 √ 0.71 + 0.68 √ 0.05 √ 0.650.90 = 0.62 + 0.04 + 0.12 + 0.12While the reliability of the SAT data served to attenuate thetrue score correlation between reading and writing scores (cf. firstterm in Eq. 7), the correlations between (a) reading error scoresand writing errors scores (cf. second term in Eq. 7), (b) readingerror scores and writing true scores (cf. third term in Eq. 7), and(c) writing errors scores and reading true scores (cf. forth termin Eq. 7), served to mitigate the effect of that attenuation. Alsonote that the observed score correlation (0.90) is more in-line withthe true score correlation (0.91) than what Spearman’s (1904) correctionformula yielded (i.e., 0.90/ √ 0.71 × 0.65 = 1.33). Giventhat Spearman’s correction produced a value in excess of 1.00, itTable 2 | Values for observed score correlation computation for SATand beck data.Component SAT Beckr(T X , T Y ) 0.91 0.69r(E X , E Y ) 0.84 0.76r(T X , E Y ) 0.62 0.47r(T Y , E X ) 0.68 −0.14r XX (SD 2 T X/SD 2 O X) 0.71 0.79r YY (SD 2 T Y/SD 2 O Y) 0.65 0.83e XX (SD 2 T X/SD 2 O X) 0.05 0.21e YY (SD 2 T Y/SD 2 O Y) 0.06 0.17(7)would be more accurate to report the observed score correlation,rather than follow conventional guidelines (e.g., Onwuegbuzieet al., 2004) and report 1.00.PSYCHOLOGY EXAMPLEWe applied Eq. 6 to average class BDI-II (Beck, 1996) and BAI(Beck, 1990) scoresfromNimon and Henson, 2010;seeTable 3).In their study, students responded to the BDI-II and BAI at twotimes within the same semester (i.e., time-1, time-2). FollowingWetcher-Hendricks’ (2006) example of using predicted scores astrue scores, we used scores for time-1 as observed data and thepredicted scores (regressing time-2 on time-1) as true scores. As inthe education example, we subtracted true scores from observedscores to compute error scores. The components of Eq. 6 appliedto Nimon and Henson’s (2010) data are presented in Table 2 andyield the observed score correlation of 0.81, as follows:r OX O Y= 0.69 √ 0.79 × 0.83 + 0.76 √ 0.21 √ 0.17+ 0.47 √ 0.17 √ 0.79 − 0.14 √ 0.21 √ 0.830.81 = 0.56 + 0.14 + 0.17 − 0.06In Nimon and Henson’s (2010) data, the true score correlation(0.69) is lower than the observed score correlation (0.81).In this case, the attenuating effect of unreliability in scores wasmitigated by other relationships involving error scores which,in the end, served to increase the observed correlation ratherthan attenuate it. As in the SAT data, Spearman’s (1904) correction(0.81/ √ 0.79 × 0.83 = 1.00) produced an over correctedcorrelation coefficient. The over-correction resulting from Spearman’scorrection was largely due to the formula not taking intoaccount the correlation between the error scores and the correlationbetween the true anxiety score and the error depressionscore.SUMMARYClassical test theory can be used to prove that ρ TX EX, ρ TY EY, ρ TX EY,and ρ TY EXall equal to 0 in a given population (Zimmerman, 2007).However, the tenets of CTT do not provide proof that ρ EX EY= 0.Furthermore, in the case of sample data, r TX E X, r TY E Y, r TX E Y, r TY E X,and r EX E Yare not necessarily zero.(8)Table 3 | Beck data observed, error, and true scores.Class Depression scores (BDI-II) Anxiety scores (BAI)Observed (O Y ) True (T Y ) Error (E Y ) Observed (O Y ) True (T Y ) Error (E Y )1 6.67 7.76 −1.10 7.34 9.08 −1.742 10.26 10.45 −0.19 11.04 11.26 −0.223 5.92 4.75 1.17 9.69 8.69 1.004 7.21 7.40 −0.19 7.00 6.98 0.025 6.85 7.28 −0.43 7.31 6.40 0.916 6.78 6.67 0.12 9.27 8.84 0.437 6.13 6.82 −0.70 6.60 7.70 −1.098 11.54 10.23 1.32 12.62 11.93 0.69M 7.67 7.67 0.00 8.86 8.86 0.00SD 2.06 1.88 0.85 2.17 1.94 0.98www.frontiersin.org April 2012 | Volume 3 | Article 102 | 45
Nimon et al.The assumption of reliable dataBecause r TX E X, r TY E Y, r TX E Y, r TY E X, and r EX E Ymay not be zeroin any given sample, researchers cannot assume that poor reliabilitywill always result in lower observed score correlations. As wehave demonstrated, observed score correlations may be less thanor greater than their true score counterparts and therefore less ormore accurate than correlations adjusted by Spearman’s (1904)formula.Just as reliability affects the magnitude of observed score correlations,it follows that statistical significance tests are also impactedby measurement error. While error that causes observed score correlationsto be greater than their true score counterparts increasesthe power of statistical significance tests,error that causes observedscore correlations to be less than their true score counterpartsdecreases the power of statistical significance tests, with all elsebeing constant. Consider the data from Nimon and Henson (2010)as an example. As computed by G∗Power3(Faul et al., 2007), withall other parameters held constant, the power of the observed scorecorrelation (r OX O Y= 0.81, 1 − β = 0.90) is greater than the powerof true score correlation (r TX T Y= 0.69, 1 − β = 0.62). In this case,error in the data served to decrease the Type II error rate ratherthan increase it.As we leave this section, it is important to note that the effectof reliability on observed score correlation decreases as reliabilityand sample size increase. Consider two research settings reviewedin Zimmerman (2007): In large n studies involving standardizedtests, “many educational and psychological tests have generallyaccepted reliabilities of 0.90 or 0.95, and studies with 500 or 1,000or more participants are not uncommon” (p. 937). In this researchsetting, the correction for and the effect of reliability on observedscore correlation may be accurately represented by Eqs 2 and 3,respectively, as long as there is not substantial correlated errorin the population. However, in studies involving a small numberof participants and new instrumentation, reliability may be 0.70,0.60, or lower. In this research setting, Eq. 2 may not accuratelycorrect and Eq. 3 may not accurately represent the effect of measurementerror on an observed score correlation. In general, if thecorrelation resulting from Eq. 2 is much greater than the observedscore correlation, it is probably inaccurate as it does not considerthe full effect of measurement error and error score correlationson the observed score correlation (cf. Eq. 4, Zimmerman, 2007).RELIABILITY: HOW DO WE ASSESS?Given that reliability affects the magnitude and statistical significanceof sample statistics, it is important for researchers toassess the reliability of their data. The technique to assess reliabilitydepends on the type of measurement error being considered.Under CTT, typical types of reliability assessed in educationaland psychological research are test–retest, parallel-form, interrater,and internal consistency. After we present the aforementionedtechniques to assess reliability, we conclude this section bycountering a common myth regarding their collective nature.TEST–RETESTReliability estimates that consider the consistency of scores acrosstime are referred to as test–retest reliability estimates. Test–retestreliability is assessed by having a set of individuals take the sameassessment at different points in time (e.g., week 1, week 2) andcorrelating the results between the two measurement occasions.For well-developed standardized achievement tests administeredreasonably close together, test–retest reliability estimates tend torange between 0.70 and 0.90 (Popham, 2000).PARALLEL-FORMReliability estimates that consider the consistency of scores acrossmultiple forms are referred to as parallel-form reliability estimates.Parallel-form reliability is assessed by having a set of individualstake different forms of an instrument (e.g., short and long; FormA and Form B) and correlating the results. For well-developedstandardized achievement tests, parallel-form reliability estimatestend to hover between 0.80 and 0.90 (Popham, 2000).INTER-RATERReliability estimates that consider the consistency of scores acrossraters are referred to as inter-rater reliability estimates. Inter-raterreliability is assessed by having two (or more) raters assess thesame set of individuals (or information) and analyzing the results.Inter-rater reliability may be found by computing consensus estimates,consistency estimates, or measurement estimates (Stemler,2004):ConsensusConsensus estimates of inter-rater reliability are based on theassumption that there should be exact agreement betweenraters. The most popular consensus estimate is simple percentagreement,which is calculated by dividing the number of casesthat received the same rating by the number of cases rated. Ingeneral, consensus estimates should be 70% or greater (Stemler,2004). Cohen’s kappa (κ; Cohen, 1960) is a derivation of simplepercent-agreement, which attempts to correct for the amount ofagreement that could be expected by chance:κ = p o − p c(9)1 − p cwhere p o is the observed agreement among raters and p c isthe hypothetical probability of chance agreement. Kappa valuesbetween 0.40 and 0.75 are considered moderate, and valuesbetween 0.75 and 1.00 are considered excellent (Fleiss, 1981).ConsistencyConsistency estimates of inter-rater reliability are based on theassumption that it is unnecessary for raters to yield the sameresponses as long as their responses are relatively consistent. Interraterreliability is typically assessed by correlating rater responses,where correlation coefficients of 0.70 or above are generallyconsidered acceptable (Barrett, 2001).MeasurementMeasurement estimates of inter-rater reliability are based on theassumption that all rater information (including discrepant ratings)should be used in creating a scale score. Principal componentsanalysis is a popular technique to compute the measurementestimate of inter-rater reliability (Harman, 1967). If the amountof shared variance in ratings that is accounted for by the first principalcomponent is greater than 60%, it is assumed that raters areassessing a common construct (Stemler, 2004).Frontiers in Psychology | Quantitative Psychology and Measurement April 2012 | Volume 3 | Article 102 | 46
Page 2 and 3: FRONTIERS COPYRIGHTSTATEMENT© Copy
Page 4 and 5: Table of Contents05 Is Data Cleanin
Page 7 and 8: OsborneAssumptions and data cleanin
Page 9 and 10: ORIGINAL RESEARCH ARTICLEpublished:
Page 12 and 13: Hoekstra et al.Why assumptions are
Page 20 and 21: García-PérezStatistical conclusio
Page 30 and 31: Sheng and ShengEffect of non-normal
Page 42 and 43: REVIEW ARTICLEpublished: 12 April 2
Page 44 and 45: Nimon et al.The assumption of relia
Page 56 and 57: TressoldiPower replication unreliab
Page 58 and 59: TressoldiPower replication unreliab
Page 62 and 63: FinchModern methods for the detecti
Page 72 and 73: MINI REVIEW ARTICLEpublished: 28 Au
Page 74 and 75: NimonStatistical assumptionsand Del
Page 76 and 77: NimonStatistical assumptionsFor exa
Page 78 and 79: Kraha et al.Interpreting multiple r
Page 94 and 95: SmithsonComparing moderation of slo
Page 96 and 97:
SmithsonComparing moderation of slo
Page 98 and 99:
Page 100 and 101:
Page 102 and 103:
REVIEW ARTICLEpublished: 01 March 2
Page 104 and 105:
Flora et al.Factor analysis assumpt
Page 106 and 107:
Page 108 and 109:
Page 110 and 111:
Page 112 and 113:
Page 114 and 115:
Page 116 and 117:
Page 118 and 119:
Page 120 and 121:
Page 122 and 123:
Page 124 and 125:
Kasper and ÜnlüAssumptions of fac
Page 126 and 127:
Page 128 and 129:
Page 130 and 131:
Page 132 and 133:
Page 134 and 135:
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Lages and JaworskaHow predictable a
Page 146 and 147:
Page 148 and 149:
Page 150 and 151:
Page 152 and 153:
Cummiskey et al.Testing assumptions
Page 154 and 155:
Page 156 and 157:
Page 158:
show all

Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

Create successful ePaper yourself

Delete template?

Save as template?