Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

More documents

Recommendations

Info

Kasper and ÜnlüAssumptions of factor analytic approaches• To what extent does the estimation accuracy for factorial structureof the classical factor analysis models depend on theskewness of the population latent ability distribution?• Are there specific aspects of the factorial structure or latent abilitydistribution with respect to which the classical factor analysismodels are more or less robust in estimation when true abilityvalues are skewed?• Given a skewed population ability distribution does the estimationaccuracy for factorial structure of the classical factoranalysis models depend on the extraction criterion applied fordetermining the number of factors from the data?• Can person ability scores estimated under classical factor analyticapproaches be representative of the true ability distributionor properties thereof when this distribution is skewed?Mattson (1997)’s method can be used for specifying the parametersettings for the simulation study (cf. Section 4.2). We brieflydescribe this method (for details, see Mattson, 1997). Assume thestandardized manifest variables are expressed as z = Aν, where ν isthe vector of latent variables and A is the matrix of model parameters.Moreover, assume that ν = Tω, where T is a lower triangularsquare matrix such that each component of ν is a linear combinationof at most two components of ω, E(vv ′ ) = ν = TT ′ , and ω isa vector of mutually independent standardized random variablesω i with finite central moments µ 1i , µ 2i , µ 3i , and µ 4i , of order upto four. ThenE(z) = AT E(ω) = 0andE(zz ′ )(= A ν A ′ ) = AT E(ωω ′ )T ′ A ′ = ATT ′ A ′ .then T and ω satisfy the required assumptions afore mentioned.Hence the skewness and kurtosis of any z i are given by, respectively,√β1i =β 2i =∑ k+pm=1 a3 im µ 3m[a ′ i a i] 3/2 and∑ k+pm=1 a4 im µ 4m + 6 ∑ k+pm=2∑ m−1o=1 a2 im a2 io[a ′ i a i] 2 .Mattson’s method is used to specify such settings for the simulationstudy as they may be observed in large scale assessmentdata. The next section describes this in detail.4.2. DESIGN OF THE SIMULATION STUDYThe number of manifest variables was fixed to p = 24 throughoutthe simulation study. For the number of factors, we usednumbers typically found in large scale assessment studies such asthe Progress in International Reading Literacy Study (PIRLS, e.g.,Mullis et al., 2006) or PISA (e.g., OECD, 2005). According to theassessment framework of PIRLS 2006 the number of dimensionsfor reading literacy was four, in PISA 2003 the scaling model hadseven dimensions. We decided to use a simple loading structurefor L, in the sense that every manifest variable was assumed to loadon only one factor (within-item unidimensionality) and that eachfactor was measured by the same number of manifest variables. Inreliance on PIRLS and PISA in our simulation study, the numbersof factors were assumed to be four or eight. We assumed that someof the factors were well explained by their indicators while otherswere not, with upper rows (variables) of the loading matrix generallyhaving higher factor loadings than lower rows (variables).Thus, the loading matrices employed in our study for the four andeight dimensional simulation models were, respectively,Or equivalently, E(z i z j ) = γ ′ i γ j, where γ i = (a ′ i T)′ and a ′ iis the i-th row of A. Under these conditions the third and fourthorder central moments of z i are given byE(z 3 i ) = ∑ mE(z 4 i ) = ∑ mγ 3im µ 3mandγ 4im µ 4m + 6 ∑ m2m−1∑o=1γ 2im γ 2io .Hence the univariate skewness √ β 1i and kurtosis β 2i of any z ican be calculated by√β1i =E ( zi3 )[ ( )]E z2 3/2and β 2i = E ( z 4 )i[ ( )]iE z2 2.iIn the simulation study, the exploratory factor analysis modelwith orthogonal factors (cov( f , f ) = I ) and error variablesassumed to be uncorrelated and unit normal (with standardizedmanifest variables) is used as the data generating model. LetA: = (L, I p ) be the concatenated matrix of dimension p × (k + p),where I p is the unit matrix of order p × p, and let v : = ( f ′ , e ′ ) ′be the concatenated vector of length k + p. Then we have z = Avfor the simulation factor model. Let T: = I (k+p)×(k+p) and ω: = ν,⎛⎞0.9 0 0 00.8 0 0 00.7 0 0 00.6 0 0 00.5 0 0 00.4 0 0 00 0.8 0 00 0.7 0 00 0.6 0 00 0.5 0 00 0.4 0 0L =0 0.3 0 00 0 0.6 00 0 0.6 00 0 0.5 00 0 0.4 00 0 0.4 00 0 0.3 00 0 0 0.60 0 0 0.50 0 0 0.50 0 0 0.4⎜⎟⎝ 0 0 0 0.3⎠0 0 0 0.3⎛⎞0.9 0 0 0 0 0 0 00.8 0 0 0 0 0 0 00.7 0 0 0 0 0 0 00 0.8 0 0 0 0 0 00 0.8 0 0 0 0 0 00 0.7 0 0 0 0 0 00 0 0.8 0 0 0 0 00 0 0.7 0 0 0 0 00 0 0.6 0 0 0 0 00 0 0 0.7 0 0 0 00 0 0 0.7 0 0 0 0and L =0 0 0 0.7 0 0 0 00 0 0 0 0.7 0 0 0.0 0 0 0 0.6 0 0 00 0 0 0 0.6 0 0 00 0 0 0 0 0.6 0 00 0 0 0 0 0.6 0 00 0 0 0 0 0.5 0 00 0 0 0 0 0 0.5 00 0 0 0 0 0 0.4 00 0 0 0 0 0 0.4 00 0 0 0 0 0 0 0.4⎜⎟⎝ 0 0 0 0 0 0 0 0.4⎠0 0 0 0 0 0 0 0.3Frontiers in Psychology | Quantitative Psychology and Measurement March 2013 | Volume 4 | Article 109 | 127
Kasper and ÜnlüAssumptions of factor analytic approachesWe decided to analyze released items of the PIRLS 2006 study(IEA, 2007) to have an empirical basis for the selection of skewnessvalues for ω(= ν). We used a data set of dichotomously scoredresponses of 7,899 German students to 125 test items. Figure 1displays the distribution of the PIRLS items’ (empirical) skewnessvalues. 6We decided to simulate under three conditions for the distributionsof ω. Under the first condition, ω m (m = 1, . . ., k) arenormal with µ 1m = 0, µ 2m = 1, µ 3m = 0, and µ 4m = 3. Under thesecond condition, ω m (m = 1, . . ., k) are slightly skewed withµ 1m = 0, µ 2m = 1, µ 3m = −0.20, and µ 4m = 3. Under the thirdcondition, ω m (m = 1, . . ., k) are strongly skewed with µ 1m = 0,µ 2m = 1, µ 3m = −2, and µ 4m = 9. The error terms were assumedto be unit normal, that is, we specified µ 1h = 0, µ 2h = 1, µ 3h = 0,and µ 4m = 3 for ω h (h = k + 1, . . ., k + p). Skewness and kurtosisof any z i under each of the three conditions were computedusing Mattson’s method (Section 4.1). The values are reported inTables 1 and 2 for the four and eight dimensional factor spaces,respectively.Under the slightly skewed distribution condition, the theoreticalvalues of skewness for the manifest variables range between−0.060 and −0.005, a condition that captured approximately 20%of the considered PIRLS test items. Under the strongly skewed distributioncondition, the theoretical values of skewness lie between−0.599 and −0.047, a condition that covered circa 30% of thePIRLS items (cf. Figure 1). Based on these theoretical skewnessand kurtosis statistics, we can see to what extent under these modelspecifications the distributions of the manifest variables deviatefrom the normal distribution.How to generate variates ω i (i = 1, . . ., k + p) such that theypossess predetermined moments µ 1i , µ 2i , µ 3i , and µ 4i ? To simulatevalues for ω i with predetermined moments, we used thegeneralized lambda distribution (Ramberg et al., 1979)ω i = λ 1 + uλ 3− (1 − u) λ 4λ 2,6 All figures of this paper were produced using the R statistical computing environment(R Development Core Team, 2011; www.r-project.org). The source files arefreely available from the authors.where u is uniform (0, 1), λ 1 is a location parameter, λ 2 a scaleparameter, and λ 3 and λ 4 are shape parameters. To realize thedesired distribution conditions for the simulation study (normal,slightly skewed, strongly skewed) using this general distributionits parameters λ 1 , λ 2 , λ 3 , and λ 4 had to be specified accordingly.Ramberg et al. (1979) tabulate the required values for the λ parametersfor different values of µ. In particular, for a (more orless) normal distribution with µ 1 = 0, µ 2 = 1, µ 3 = 0, and µ 4 = 3the corresponding values are λ 1 = 0, λ 2 = 0.197, λ 3 = 0.135, andλ 4 = 0.135. For a slightly skewed distribution with µ 1 = 0, µ 2 = 1,µ 3 = −0.20, and µ 4 = 3, the values are λ 1 = 0.237, λ 2 = 0.193,λ 3 = 0.167, and λ 4 = 0.107. For a strongly skewed distributionwith µ 1 = 0, µ 2 = 1, µ 3 = −2, and µ 4 = 9, the parameter valuesare given by λ 1 = 0.993,λ 2 = −0.108·10 −2 ,λ 3 = −0.108·10 −2 ,andλ 4 = −0.041·10 −3 .Remark. Indeed, various distributions are possible (see Mattson,1997); however, the generalized lambda distribution provesto be special. It performs very well in comparison to other distributions,when theoretical moments calculated according to theMattson formulae are compared to their corresponding empiricalmoments computed from data simulated under a factormodel (based on that distribution). For details, see Reinartz et al.(2002). These authors have also studied the effects of the useof different (pseudo) random number generators for realizingthe uniform distribution in such a comparison study. Out ofthree compared random number generators – RANUNI fromSAS, URAND from PRELIS, and RANDOM from Mathematica– the generator RANUNI performed relatively well or better.In this paper, we used the SAS program for our simulationstudy. 7Besides the number of factors and the distributions of thelatent variables, sample size was varied. In the small samplecase, every z i consisted of n = 200 observations, and in thelarge sample case z i contained n = 600 observations. Table 3summarizes the design of the simulation study. Overall there7 For the factor analyses in this paper, we used the SAS program and its PROC FAC-TOR implementation of the methods PCA, EFA, and PAA. More precisely, variationof the PROC FACTOR statements, run in their default settings, yields the performedprocedures PCA, EFA, and PAA (e.g., EFA if METHOD = ML).0.3density0.20.10.0−6−4−2−1.5−1−0.500.512skewness valueFIGURE 1 | Distribution of the skewness values for the 125 PIRLS test items.www.frontiersin.org March 2013 | Volume 4 | Article 109 | 128
Page 2 and 3:
FRONTIERS COPYRIGHTSTATEMENT© Copy
Page 4 and 5:
Table of Contents05 Is Data Cleanin
Page 7 and 8:
OsborneAssumptions and data cleanin
Page 9 and 10:
ORIGINAL RESEARCH ARTICLEpublished:
Page 12 and 13:
Hoekstra et al.Why assumptions are
Page 14 and 15:
Page 16 and 17:
Page 18 and 19:
Page 20 and 21:
García-PérezStatistical conclusio
Page 22 and 23:
Page 24 and 25:
Page 26 and 27:
Page 28 and 29:
Page 30 and 31:
Sheng and ShengEffect of non-normal
Page 32 and 33:
Page 34 and 35:
Page 36 and 37:
Page 38 and 39:
Page 40 and 41:
Page 42 and 43:
REVIEW ARTICLEpublished: 12 April 2
Page 44 and 45:
Nimon et al.The assumption of relia
Page 46 and 47:
Page 48 and 49:
Page 50 and 51:
Page 52 and 53:
Page 54 and 55:
Page 56 and 57:
TressoldiPower replication unreliab
Page 58 and 59:
TressoldiPower replication unreliab
Page 60 and 61:
Page 62 and 63:
FinchModern methods for the detecti
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
MINI REVIEW ARTICLEpublished: 28 Au
Page 74 and 75:
NimonStatistical assumptionsand Del
Page 76 and 77:
NimonStatistical assumptionsFor exa
Page 78 and 79: Kraha et al.Interpreting multiple r
Page 94 and 95: SmithsonComparing moderation of slo
Page 102 and 103: REVIEW ARTICLEpublished: 01 March 2
Page 104 and 105: Flora et al.Factor analysis assumpt
Page 124 and 125: Kasper and ÜnlüAssumptions of fac
Page 144 and 145: Lages and JaworskaHow predictable a
Page 152 and 153: Cummiskey et al.Testing assumptions
Page 158: Cummiskey et al.Testing assumptions
show all

Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

Create successful ePaper yourself

Delete template?

Save as template?