Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

Kasper and ÜnlüAssumptions of factor analytic approachesTable 5 | Number of extracted dimensions for the PIRLS 2006 testbooklet number 4, German sample.Table 7 | Loading matrix for four factors principal axis analysis of thePIRLS 2006 data for test booklet number 4, German sample.Extraction methodFactor modelItemFactorPCA a EFA b PAA c1 2 3 4Kaiser–Guttman criterion 6 6 6Scree test 1 1 1Parallel analysis method 1 4 4aPCA, principal component analysis; b EFA, exploratory factor analysis; c PAA,principal axis analysis.Table 6 | Loading matrix for four factors exploratory factor analysis ofthe PIRLS 2006 data for test booklet number 4, German sample.ItemFactor1 2 3 4R011A01C A,R 0.15 0.26 0.39 −0.05R011A02M A,R 0.14 0.28 0.34 0.19R011A03C A,R 0.16 0.24 0.09 0.03R011A04C A,I 0.39 0.19 0.10 0.06R011A05M A,R 0.22 0.08 0.19 0.21R011A06M A,R 0.20 0.03 0.14 0.06R011A07C A,R 0.50 0.20 0.22 0.15R011A08C A,R 0.35 0.04 0.38 −0.09R011A09C A,I 0.55 0.18 0.11 0.00R011A10M A,I 0.28 0.27 0.22 0.11R011A11C A,I 0.37 0.06 0.14 0.02R021E01M L,R 0.08 0.45 0.19 −0.06R021E02M L,R 0.02 0.49 0.09 0.24R021E03M L,R 0.14 0.34 −0.02 0.02R021E04M L,R 0.17 0.28 0.15 0.02R021E05C L,R 0.22 0.23 0.32 0.12R021E06M L,R 0.17 0.44 0.09 0.28R021E07C L,I 0.13 0.06 0.48 0.22R021E08M L,I 0.32 0.23 0.04 0.48R021E09C L,I 0.45 0.24 0.02 0.20R021E10C L,I 0.27 0.23 0.17 0.07R021E11M L,I 0.00 0.01 0.06 0.40R021E12C L,I 0.38 0.17 0.31 0.22Factor loadings greater or equal 0.30 are highlighted.A, Acquire and Use Information; L, Literary Experience; R, Retrieving andStraightforward Inferencing; I, Interpreting, Integrating, and Evaluating.7.2. OUTLOOKWe have discussed possible implications of the findings forcriterion-referenced tests and large scale educational assessment.The assumptions of the classical factor models have been seento be crucial in these application fields. We suggest, for instance,that the presented classical procedures should not be used, unlesswith special caution if at all, to examine the factorial structure ofdichotomously scored criterion-referenced tests. Instead, if modelviolations of the “sensitive” type are present, better suited or moresophisticated latent variable models can be used (see Skrondal andR011A01C A,R 0.15 0.26 0.40 −0.06R011A02M A,R 0.14 0.29 0.33 0.18R011A03C A,R 0.16 0.24 0.09 0.02R011A04C A,I 0.38 0.20 0.10 0.06R011A05M A,R 0.22 0.07 0.19 0.24R011A06M A,R 0.19 0.02 0.14 0.07R011A07C A,R 0.50 0.20 0.22 0.16R011A08C A,R 0.36 0.03 0.38 −0.08R011A09C A,I 0.54 0.19 0.12 0.00R011A10M A,I 0.28 0.27 0.22 0.11R011A11C A,I 0.38 0.07 0.13 0.02R021E01M L,R 0.07 0.45 0.19 −0.06R021E02M L,R 0.03 0.49 0.09 0.24R021E03M L,R 0.14 0.33 −0.02 0.02R021E04M L,R 0.17 0.26 0.16 0.04R021E05C L,R 0.21 0.23 0.32 0.12R021E06M L,R 0.17 0.44 0.08 0.27R021E07C L,I 0.13 0.06 0.47 0.23R021E08M L,I 0.32 0.24 0.05 0.46R021E09C L,I 0.45 0.24 0.02 0.19R021E10C L,I 0.27 0.24 0.17 0.06R021E11M L,I 0.00 0.02 0.05 0.40R021E12C L,I 0.38 0.17 0.30 0.22Factor loadings greater or equal 0.30 are highlighted.A, Acquire and Use Information; L, Literary Experience; R, Retrieving andStraightforward Inferencing; I: Interpreting, Integrating, and Evaluating.Rabe-Hesketh, 2004). Examples are item response theory parametricor non-parametric models for categorical response data (e.g.,van der Linden and Hambleton, 1997). Furthermore, we wouldlike to mention item response based factor analysis approaches byBock and Lieberman (1970) or Christoffersson (1975, 1977). Wemay also pay attention to tetrachoric or polychoric based structuralequation models by Muthén (1978, 1983, 1984) and Muthénand Christoffersson (1981).As with factor analysis a general problem (e.g., Maraun, 1996),we had to deal with the issue of rotational indeterminacy andof selecting a specific rotation. We have decided to use varimaxrotation, due to the fact that this rotation is most frequently usedin empirical educational studies (for better interpretability of thefactors). Future research may cover other rotations (e.g., quartimaxor equimax) or the evaluation of parameter estimation byexamining the communality estimates for each item (which arenot dependent on rotation, but are a function of the factor loadings).Moreover, the orthogonal factor model may not be realistic,as factors are correlated in general. However, in the current study,it may be unlikely that having non-zero population factor loadingsfor correlated dimensions would substantially affect the findings.In further research, we will have to study the case of the oblique(non-orthogonal) factor model.Frontiers in Psychology | Quantitative Psychology and Measurement March 2013 | Volume 4 | Article 109 | 139