Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

More documents

Recommendations

Info

Kasper and ÜnlüAssumptions of factor analytic approachesTable 5 | Number of extracted dimensions for the PIRLS 2006 testbooklet number 4, German sample.Table 7 | Loading matrix for four factors principal axis analysis of thePIRLS 2006 data for test booklet number 4, German sample.Extraction methodFactor modelItemFactorPCA a EFA b PAA c1 2 3 4Kaiser–Guttman criterion 6 6 6Scree test 1 1 1Parallel analysis method 1 4 4aPCA, principal component analysis; b EFA, exploratory factor analysis; c PAA,principal axis analysis.Table 6 | Loading matrix for four factors exploratory factor analysis ofthe PIRLS 2006 data for test booklet number 4, German sample.ItemFactor1 2 3 4R011A01C A,R 0.15 0.26 0.39 −0.05R011A02M A,R 0.14 0.28 0.34 0.19R011A03C A,R 0.16 0.24 0.09 0.03R011A04C A,I 0.39 0.19 0.10 0.06R011A05M A,R 0.22 0.08 0.19 0.21R011A06M A,R 0.20 0.03 0.14 0.06R011A07C A,R 0.50 0.20 0.22 0.15R011A08C A,R 0.35 0.04 0.38 −0.09R011A09C A,I 0.55 0.18 0.11 0.00R011A10M A,I 0.28 0.27 0.22 0.11R011A11C A,I 0.37 0.06 0.14 0.02R021E01M L,R 0.08 0.45 0.19 −0.06R021E02M L,R 0.02 0.49 0.09 0.24R021E03M L,R 0.14 0.34 −0.02 0.02R021E04M L,R 0.17 0.28 0.15 0.02R021E05C L,R 0.22 0.23 0.32 0.12R021E06M L,R 0.17 0.44 0.09 0.28R021E07C L,I 0.13 0.06 0.48 0.22R021E08M L,I 0.32 0.23 0.04 0.48R021E09C L,I 0.45 0.24 0.02 0.20R021E10C L,I 0.27 0.23 0.17 0.07R021E11M L,I 0.00 0.01 0.06 0.40R021E12C L,I 0.38 0.17 0.31 0.22Factor loadings greater or equal 0.30 are highlighted.A, Acquire and Use Information; L, Literary Experience; R, Retrieving andStraightforward Inferencing; I, Interpreting, Integrating, and Evaluating.7.2. OUTLOOKWe have discussed possible implications of the findings forcriterion-referenced tests and large scale educational assessment.The assumptions of the classical factor models have been seento be crucial in these application fields. We suggest, for instance,that the presented classical procedures should not be used, unlesswith special caution if at all, to examine the factorial structure ofdichotomously scored criterion-referenced tests. Instead, if modelviolations of the “sensitive” type are present, better suited or moresophisticated latent variable models can be used (see Skrondal andR011A01C A,R 0.15 0.26 0.40 −0.06R011A02M A,R 0.14 0.29 0.33 0.18R011A03C A,R 0.16 0.24 0.09 0.02R011A04C A,I 0.38 0.20 0.10 0.06R011A05M A,R 0.22 0.07 0.19 0.24R011A06M A,R 0.19 0.02 0.14 0.07R011A07C A,R 0.50 0.20 0.22 0.16R011A08C A,R 0.36 0.03 0.38 −0.08R011A09C A,I 0.54 0.19 0.12 0.00R011A10M A,I 0.28 0.27 0.22 0.11R011A11C A,I 0.38 0.07 0.13 0.02R021E01M L,R 0.07 0.45 0.19 −0.06R021E02M L,R 0.03 0.49 0.09 0.24R021E03M L,R 0.14 0.33 −0.02 0.02R021E04M L,R 0.17 0.26 0.16 0.04R021E05C L,R 0.21 0.23 0.32 0.12R021E06M L,R 0.17 0.44 0.08 0.27R021E07C L,I 0.13 0.06 0.47 0.23R021E08M L,I 0.32 0.24 0.05 0.46R021E09C L,I 0.45 0.24 0.02 0.19R021E10C L,I 0.27 0.24 0.17 0.06R021E11M L,I 0.00 0.02 0.05 0.40R021E12C L,I 0.38 0.17 0.30 0.22Factor loadings greater or equal 0.30 are highlighted.A, Acquire and Use Information; L, Literary Experience; R, Retrieving andStraightforward Inferencing; I: Interpreting, Integrating, and Evaluating.Rabe-Hesketh, 2004). Examples are item response theory parametricor non-parametric models for categorical response data (e.g.,van der Linden and Hambleton, 1997). Furthermore, we wouldlike to mention item response based factor analysis approaches byBock and Lieberman (1970) or Christoffersson (1975, 1977). Wemay also pay attention to tetrachoric or polychoric based structuralequation models by Muthén (1978, 1983, 1984) and Muthénand Christoffersson (1981).As with factor analysis a general problem (e.g., Maraun, 1996),we had to deal with the issue of rotational indeterminacy andof selecting a specific rotation. We have decided to use varimaxrotation, due to the fact that this rotation is most frequently usedin empirical educational studies (for better interpretability of thefactors). Future research may cover other rotations (e.g., quartimaxor equimax) or the evaluation of parameter estimation byexamining the communality estimates for each item (which arenot dependent on rotation, but are a function of the factor loadings).Moreover, the orthogonal factor model may not be realistic,as factors are correlated in general. However, in the current study,it may be unlikely that having non-zero population factor loadingsfor correlated dimensions would substantially affect the findings.In further research, we will have to study the case of the oblique(non-orthogonal) factor model.Frontiers in Psychology | Quantitative Psychology and Measurement March 2013 | Volume 4 | Article 109 | 139
Kasper and ÜnlüAssumptions of factor analytic approachesThe results of this paper provide implications for popularresearch practices in the empirical educational research field. Themethods that we have utilized are traditional and often appliedin practice (e.g., by educational scientists), for instance to determinethe factorial validity of criterion-referenced tests or to studylarge scale assessment measurement instruments. In addition, toconsider other, more sophisticated fit statistics can be interestingand valuable. For example, such model fit statistics as the rootmean square residual, comparative fit index, or the root meansquared error of approximation may be investigated. Albeit thesefit statistics are well-known and applied in the confirmatory factoranalysis (CFA) context, they could be produced for exploratoryfactor analysis (given that CFA and EFA are based on the samecommon factor model).We conclude with important research questions related tothe PISA study. In the context of PISA, principal componentanalysis is used, in the purely computational sense. Other distributional,inferential, or confirmatory factor models, especiallythose for the verification of the factorial validity of the PISAcontext questionnaires, have not been considered. Interestingquestions arise: are there other approaches to dimensionalityreduction that can perform at least as well as the principal componentanalysis method in PISA data (e.g., multidimensionalscaling; Borg and Groenen, 2005)? Is the 95% extraction rule inprincipal component analysis of PISA data an “optimal” criterion?How sensitive are PISA results if, for example, the parallelanalysis method is used as the extraction criterion? Answeringthese and other related questions is out of the scope of thepresent paper and can be pursued in more in-depth future analyses.Nonetheless, the important role of these problems in thePISA context is worth mentioning. The PISA procedure usesnot only manifest background information but also principalcomponent scores on complex constructs in order to assign literacyor plausible values to students. Future research is necessaryto investigate the effects and possible implications ofpotentially biased estimates of latent or complex backgroundinformation on students’ assigned literacy values, and especially,their competence levels, based on which the PISA rankings arereported.ACKNOWLEDGMENTSThe authors wish to thank Sabine Felbinger for her critical readingand helpful comments. In particular, we are deeply indebtedto Jason W. Osborne, Chief Editor, and four reviewers. Their criticaland valuable comments and suggestions have improved themanuscript greatly.REFERENCESAdams, R., Wilson, M., and Wang, W.(1997). The multidimensional randomcoefficients multinomial logitmodel. Appl. Psychol. Meas. 21, 1–23.Bartholomew, D., Knott, M., and Moustaki,I. (2011). Latent Variable Modelsand Factor Analysis: A UnifiedApproach. Chichester: John Wiley &Sons.Birnbaum, A. (1968). “Some latent traitmodels and their use in inferring anexaminee’s ability,”in Statistical Theoriesof Mental Test Scores, eds F. M.Lord and M. R. Novick (Reading,MA: Addison-Wesley), 397–479.Bock, D., and Lieberman, M. (1970).Fitting a response model for ndichotomously scored items. Psychometrika35, 179–197.Bolt, D. M. (2005). “Limited- andfull-information estimation of itemresponse theory models,” in ContemporaryPsychometrics, eds A.Maydeu-Olivares and J. J. McArdle(Mahwah,NJ: Lawrence ErlbaumAssociates), 27–72.Borg, I., and Groenen, P. J. F. (2005).Modern Multidimensional Scaling:Theory and Applications. Berlin:Springer.Browne, M. (1974). Generalized leastsquares estimators in the analysisof covariance structures. South Afr.Stat. J. 8, 1–24.Burt, C. (1909). Experimental tests ofgeneral intelligence. Br. J. Psychol. 3,94–177.Carroll, J. (1945). The effect of difficultyand chance success on correlationsbetween items or between tests.Psychometrika 10, 1–19.Cattell, R. (1966). The scree test forthe number of factors. MultivariateBehav. Res. 1, 245–276.Christoffersson,A. (1975). Factor analysisof dichotomized variables. Psychometrika40, 5–32.Christoffersson, A. (1977). Two-stepweighted least squares factor analysisof dichotomized variables. Psychometrika42, 433–438.Collins, L., Cliff, N., McCormick, D.,and Zatkin, J. (1986). Factor recoveryin binary data sets: a simulation.Multivariate Behav. Res. 21, 377–391.Cronbach, L., and Meehl, P. (1955).Construct validity in psychologicaltests. Psychol. Bull. 52, 281–302.Cudeck, R., and MacCallum, R. (eds).(2007). Factor Analysis at 100: HistoricalDevelopments and Future Directions.Mahwah, NJ: Lawrence ErlbaumAssociates.Dolan,C. (1994). Factor analysis of variableswith 2, 3, 5 and 7 response categories:a comparison of categoricalvariable estimators using simulateddata. Br. J. Math. Stat. Psychol. 47,309–326.Ferguson, E., and Cox, T. (1993).Exploratory factor analysis: a users’guide. Int. J. Sel. Assess. 1, 84–94.Ferguson, G. (1941). The factorial interpretationof test difficulty. Psychometrika6, 323–329.Flora, D., LaBrish, C., and Palmers,R. (2012). Old and new ideas fordata screening and assumption testingfor exploratory and confirmatoryfactor analysis. Front. Psychol.3:55. doi:10.3389/fpsyg.2012.00055Gorsuch, R. (1997). Exploratory factoranalysis: its role in item analysis. J.Pers. Assess. 68, 532–560.Green, S. (1983). Identifiability of spuriousfactors using linear factor analysiswith binary items. Appl. Psychol.Meas. 7, 139–147.Guttman, L. (1954). Some necessaryconditions for common factoranalysis. Psychometrika 19, 149–161.Harman, H. (1976). Modern FactorAnalysis. Chicago, IL: University ofChicago Press.Horn, J. (1965). A rationale and testfor the number of factors in factoranalysis. Psychometrika 30, 179–185.Hotelling, H. (1933a). Analysis of acomplex of statistical variables intoprincipal components. J. Educ. Psychol.24, 417–441.Hotelling, H. (1933b). Analysis of acomplex of statistical variables intoprincipal components. J. Educ. Psychol.24, 498–520.IEA. (2007). PIRLS 2006 Assessment.Boston, MA: TIMSS & PIRLS InternationalStudy Center.Jöreskog, K. G. (1966). Testing a simplestructure hypothesis in factoranalysis. Psychometrika 31, 165–178.Jöreskog, K. G. (1967). Some contributionsto maximum likelihood factoranalysis. Psychometrika 32, 443–482.Kaiser, H. (1958). The varimax criterionfor analytic rotation in factoranalysis. Psychometrika 23, 187–200.Kaiser, H., and Dickman, K. (1959).Analytic determination of commonfactors. Am. Psychol. 14, 425.Kasper, D. (2012). Klassische undDichotome Faktorenanalyse aufdem Prüfstand: Zum Einfluss derFähigkeits- und Aufgabenparameterauf die Schätzgenauigkeitverschiedener Faktorenmodelle[Classical and Dichotomous FactorAnalysis put to Test: On the Impact ofthe Ability and Item Parameters onthe Estimation Precision of VariousFactor Models]. Münster: Waxmann.Kelley, T. L. (1935). Essential Traits ofMental Life. Cambridge, MA: HarvardUniversity Press.Klauer, K. (1987). KriteriumsorientierteTests: Lehrbuch der Theorie undPraxis lehrzielorientierten Messens[Criterion-Referenced Tests: Textbookon Theory and Practice of TeachingGoal Oriented Measuring]. Göttingen:Hogrefe.Lienert, G., and Raatz, U. (1998). Testaufbauund Testanalyse [Test Constructionand Test Analysis]. Weinheim:Psychologie Verlags Union.MacCallum, R. (2009). “Factor analysis,”in The SAGE Handbook ofQuantitative Methods in Psychology,eds R. E. Millsap and A. Maydeu-Olivares (London: Sage Publications),123–147.www.frontiersin.org March 2013 | Volume 4 | Article 109 | 140
Page 2 and 3:
FRONTIERS COPYRIGHTSTATEMENT© Copy
Page 4 and 5:
Table of Contents05 Is Data Cleanin
Page 7 and 8:
OsborneAssumptions and data cleanin
Page 9 and 10:
ORIGINAL RESEARCH ARTICLEpublished:
Page 12 and 13:
Hoekstra et al.Why assumptions are
Page 14 and 15:
Page 16 and 17:
Page 18 and 19:
Page 20 and 21:
García-PérezStatistical conclusio
Page 22 and 23:
Page 24 and 25:
Page 26 and 27:
Page 28 and 29:
Page 30 and 31:
Sheng and ShengEffect of non-normal
Page 32 and 33:
Page 34 and 35:
Page 36 and 37:
Page 38 and 39:
Page 40 and 41:
Page 42 and 43:
REVIEW ARTICLEpublished: 12 April 2
Page 44 and 45:
Nimon et al.The assumption of relia
Page 46 and 47:
Page 48 and 49:
Page 50 and 51:
Page 52 and 53:
Page 54 and 55:
Page 56 and 57:
TressoldiPower replication unreliab
Page 58 and 59:
TressoldiPower replication unreliab
Page 60 and 61:
Page 62 and 63:
FinchModern methods for the detecti
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
MINI REVIEW ARTICLEpublished: 28 Au
Page 74 and 75:
NimonStatistical assumptionsand Del
Page 76 and 77:
NimonStatistical assumptionsFor exa
Page 78 and 79:
Kraha et al.Interpreting multiple r
Page 80 and 81:
Page 82 and 83:
Page 84 and 85:
Page 86 and 87:
Page 88 and 89:
Page 90 and 91: Kraha et al.Interpreting multiple r
Page 92 and 93: Kraha et al.Interpreting multiple r
Page 94 and 95: SmithsonComparing moderation of slo
Page 102 and 103: REVIEW ARTICLEpublished: 01 March 2
Page 104 and 105: Flora et al.Factor analysis assumpt
Page 124 and 125: Kasper and ÜnlüAssumptions of fac
Page 144 and 145: Lages and JaworskaHow predictable a
Page 152 and 153: Cummiskey et al.Testing assumptions
Page 158: Cummiskey et al.Testing assumptions
show all

Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?