Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

More documents

Recommendations

Info

Flora et al.Factor analysis assumptionsproduct-moment correlations were attenuated which led to negativelybiased factor loading estimates (especially with fewerresponse categories), whereas polychoric correlations remainedaccurate and produced accurate factor loadings. When theobserved variable set contained a mix of positively and negativelyskewed items (Cases 3 and 4), product-moment correlationswere strongly affected by the direction of skewness (especiallywith fewer response categories), which can lead to dramaticallymisleading factor analysis results. Unfortunately, strongly skeweditems can also be problematic for factor analyses of polychoric R inpart because they produce sparse observed frequency tables, whichleads to higher rates of improper solutions and inaccurate results(e.g., Flora and Curran, 2004; Forero et al., 2009). Yet this difficultyis alleviated with larger sample size and more item-responsecategories; if a proper solution is obtained, then the results of afactor analysis of polychoric R are more trustworthy than thoseobtained with product-moment R.GENERAL DISCUSSION AND CONCLUSIONFactor analysis is traditionally and most commonly an analysis ofthe correlations (or covariances) among a set of observed variables.A unifying theme of this paper is that if the correlationsbeing analyzed are misrepresentative or inappropriate summariesof the relationships among the variables, then the factor analysisis compromised. Thus, the process of data screening and assumptiontesting for factor analysis should begin with a focus on theadequacy of the correlation matrix among the observed variables.In particular, analysis of product-moment correlations or covariancesimplies that the bivariate association between two observedvariables can be adequately modeled with a straight line, whichin turn leads to the expression of the common factor model asa linear regression model. This crucial linearity assumption takesprecedence over any concerns about having normally distributedvariables, although normality is important for certain model fitstatistics and estimating parameter SEs. Other concerns aboutthe appropriateness of the correlation matrix involve collinearityand potential influence of unusual cases. Once one acceptsthe sufficiency of a given correlation matrix as a representationof the observed data, the actual formal assumptions of the commonfactor model are relatively mild. These assumptions are thatthe unique factors (i.e., the residuals, ε,inEq. 1) are uncorrelatedwith each other and are uncorrelated with the common factors(i.e., η in Eq. 1). Substantial violation of these assumptions typicallymanifests as poor model-data fit, and is otherwise difficult toassess with a priori data-screening procedures based on descriptivestatistics or graphs.After reviewing the common factor model, we gave separatepresentations of issues concerning factor analysis of continuousobserved variables and issues concerning factor analysis of categorical,item-level observed variables. For the former, we showedhow concepts from regression diagnostics apply to factor analysis,given that the common factor model is itself a multivariate multipleregression model with unobserved explanatory variables. Animportant point was that cases that appear as outliers in univariateor bivariate plots are not necessarily influential and conversely thatinfluential cases may not appear as outliers in univariate or bivariateplots (though they often do). If one can determine that unusualobservations are not a result of researcher or participant error,thenwe recommend the use of robust estimation procedures instead ofdeleting the unusual observation. Likewise, we also recommendthe use of robust procedures for calculating model fit statisticsand SEs when observed, continuous variables are non-normal.Next, a crucial message was that the linearity assumption isnecessarily violated when the common factor model is fitted toproduct-moment correlations among categorical, ordinally scaleditems, including the ubiquitous Likert-type items. At worst (e.g.,with a mix of positively and negatively skewed dichotomousitems), this assumption violation has severe consequences forfactor analysis results. At best (e.g., with symmetric items withfive response categories), this assumption violation still producesbiased factor-loading estimates. Alternatively, factor analysis ofpolychoric R among item-level variables explicitly specifies a nonlinearlink between the common factors and the observed variables,and as such is theoretically well-suited to the analysis ofitem-level variables. However, this method is also vulnerable tocertain data characteristics, particularly sparseness in the bivariatefrequency tables for item pairs, which occurs when stronglyskewed items are analyzed with a relatively small sample. Yet, factoranalysis of polychoric R among items generally produces superiorresults compared to those obtained with product-moment R,especially if there are five or fewer item-response categories.We have not yet directly addressed the role of sample size. Inshort,no simple rule-of-thumb regarding sample size is reasonablygeneralizable across factor analysis applications. Instead, adequatesample size depends on many features of the research, such as themajor substantive goals of the analysis, the number of observedvariables per factor, closeness to simple structure, and the strengthof the factor loadings (MacCallum et al., 1999, 2001). Beyond theseconsiderations, having a larger sample size can guard against someof the harmful consequences of unusual cases and assumptionviolation. For example, unusual cases are less likely to exert stronginfluence on model estimates as overall sample size increases. Conversely,removing unusual cases decreases the sample size, whichreduces the precision of parameter estimation and statistical powerfor hypothesis tests about model fit or parameter estimates. Yet,having a larger sample size does not protect against the negativeconsequences of treating categorical item-level variables as continuousby factor analyzing product-moment R. But we did illustratethat larger sample size produces better results for factor analysis ofpolychoric R among strongly skewed items, in part because largersample size reduces the occurrence of sparseness.In closing, we emphasize that factor analysis, whether EFA orCFA, is a method for modeling relationships among observedvariables. It is important for researchers to recognize that itis impossible for a statistical model to be perfect; assumptionswill always be violated to some extent in that no model canever exactly capture the intricacies of nature. Instead, researchersshould strive to find models that have an approximate fit todata such that the inevitable assumption violations are trivial,but the models can still provide useful results that help answerimportant substantive research questions (see MacCallum, 2003,and Rodgers, 2010, for discussions of this principle). We recommendextensive use of sensitivity analyses and cross-validation toaid in this endeavor. For example, researchers should comparewww.frontiersin.org March 2012 | Volume 3 | Article 55 | 119
Flora et al.Factor analysis assumptionsresults obtained from the same data using different estimationprocedures, such as comparing traditional ULS or ML estimationwith robust procedures with continuous variables or comparingfull-information factor analysis results with limited-informationresults with item-level variables. Additionally, as even CFA analysesmay become exploratory through model modification, it isimportant to cross-validate models across independent data sets.Because different modeling procedures place different demands ondata, comparing results obtained with different methods and samplescan help researchers gain a fuller, richer understanding of theusefulness of their statistical models given the natural complexityof real data.REFERENCESAsparouhov, T., and Muthén, B.(2009). Exploratory structural equationmodeling. Struct. Equ. Modeling16, 397–438.Babakus, E., Ferguson, C. E., andJoreskög, K. G. (1987). The sensitivityof confirmatory maximum likelihoodfactor analysis to violationsof measurement scale and distributionalassumptions. J. Mark. Res. 37,72–141.Bartholomew, D. J. (2007). “Three facesof factor analysis,” in Factor Analysisat 100: Historical Developmentsand Future Directions, eds R. Cudeckand R. C. MacCallum (Mahwah,NJ: Lawrence Erlbaum Associates),9–21.Belsley, D. A., Kuh, E., and Welsch, R. E.(1980). Regression Diagnostics: IdentifyingInfluential Data and Sourcesof Collinearity. New York: Wiley.Bentler, P. M. (2004). EQS 6 StructuralEquations Program Manual. Encino,CA: Multivariate Software, Inc.Bentler, P. M. (2007). Can scientificallyuseful hypotheses be tested with correlations?Am. Psychol. 62, 772–782.Bentler, P. M., and Savalei, V. (2010).“Analysis of correlation structures:current status and open problems,”in Statistics in the Social Sciences:Current Methodological Developments,eds S. Kolenikov, D. Steinley,and L. Thombs, (Hoboken, NJ:Wiley), 1–36.Bernstein, I. H., and Teng, G. (1989).Factoring items and factoring scalesare different: spurious evidence formultidimensionality due to itemcategorization. Psychol. Bull. 105,467–477.Bock, R. D., Gibbons, R., and Muraki,E. (1988). Full-information item factoranalysis. Appl. Psychol. Meas. 12,261–280.Bollen, K. A. (1987). Outliers andimproper solutions: a confirmatoryfactor analysis example. Sociol.Methods Res. 15, 375–384.Bollen, K. A. (1989). Structural Equationswith Latent Variables. NewYork: Wiley.Bollen, K. A., and Arminger, G.(1991). Observational residuals infactor analysis and structural equationmodels. Sociol. Methodol. 21,235–262.Briggs, N. E., and MacCallum, R. C.(2003). Recovery of weak commonfactors by maximum likelihood andordinary least squares estimation.Multivariate Behav. Res. 38, 25–56.Browne, M. W., Cudeck, R., Tateneni,K., and Mels, G. (2010). CEFA:Comprehensive Exploratory FactorAnalysis, Version 3. 03 [ComputerSoftware, and Manual]. Availableat: http://faculty.psy.ohio-state.edu/browne/Cadigan, N. G. (1995). Local influencein structural equation models.Struct. Equ. Modeling 2, 13–30.Cudeck, R. (1989). Analysis of correlationmatrices using covariancestructure models. Psychol. Bull. 105,317–327.Cudeck, R., and O’Dell, L. L. (1994).Applications of standard error estimatesin unrestricted factor analysis:significance tests for factor loadingsand correlations. Psychol. Bull. 115,475–487.Dolan, C. V. (1994). Factor analysis ofvariables with 2, 3, 5 and 7 responsecategories: a comparison of categoricalvariable estimators using simulateddata. Br. J. Math. Stat. Psychol.47, 309–326.Ferguson, G. A. (1941). The factorialinterpretation of test difficulty.Psychometrika 6, 323–329.Finney, S. J., and DiStefano, C. (2006).“Non-normal and categorical datain structural equation modeling,”in Structural Equation Modeling: ASecond Course, eds G. R. Hancockand R. O. Mueller (Greenwich,CT: Information Age Publishing),269–314.Flora, D. B., and Curran, P. J. (2004).An empirical evaluation of alternativemethods of estimation forconfirmatory factor analysis withordinal data. Psychol. Methods 9,466–491.Forero, C. G., and Maydeu-Olivares, A.(2009). Estimation of IRT gradedmodels for rating data: limited vs.full information methods. Psychol.Methods 14, 275–299.Forero, C. G., Maydeu-Olivares, A.,and Gallardo-Pujol, D. (2009). Factoranalysis with ordinal indicators:a Monte Carlo study comparingDWLS and ULS estimation. Struct.Equ. Modeling 16, 625–641.Fox, J. (1991). Regression Diagnostics: AnIntroduction. Thousand Oaks, CA:SAGE Publications.Fox, J. (2008). Applied Regression Analysisand Generalized Linear Models,2nd Edn. Thousand Oaks, CA: SAGEPublications.Gorsuch, R. L. (1983). Factor Analysis,2nd Edn. Hillsdale, NJ: LawrenceErlbaum Associates.Green, S. B., Akey, T. M., Fleming, K. K.,Hershberger, S. L., and Marquis, J. G.(1997). Effect of the number of scalepoints on chi-square fit indices inconfirmatory factor analysis. Struct.Equ. Modeling 4, 108–120.Harman, H. H. (1960). Modern FactorAnalysis. Chicago, IL: University ofChicago Press.Jöreskog, K. G. (1969). A generalapproach to confirmatory maximumlikelihood factor analysis. Psychometrika34, 183–202.Lawley, D. N., and Maxwell, A. E.(1963). Factor Analysis as a StatisticalMethod. London: Butterworth.Lee, S.-Y., and Wang, S. J. (1996). Sensitivityanalysis of structural equationmodels. Psychometrika 61, 93–108.MacCallum, R. C. (2003). Workingwith imperfect models. MultivariateBehav. Res. 38, 113–139.MacCallum,R. C. (2009).“Factor analysis,”inThe SAGE Handbook of QuantitativeMethods in Psychology, eds R.E. Millsap and A. Maydeu-Olivares(Thousand Oaks, CA: SAGE Publications),123–147.MacCallum, R. C., Widaman, K. F.,Preacher, K., and Hong, S. (2001).Sample size in factor analysis: therole of model error. MultivariateBehav. Res. 36, 611–637.MacCallum, R. C., Widaman, K. F.,Zhang, S., and Hong, S. (1999). Samplesize in factor analysis. Psychol.Methods 4, 84–99.Mavridis, D., and Moustaki, I. (2008).Detecting outliers in factor analysisusing the forward search algorithm.Multivariate Behav. Res. 43,453–475.Muthén, B. (1978). Contributions tofactor analysis of dichotomous variables.Psychometrika 43, 551–560.Muthén, B., and Kaplan, D. (1985).A comparison of some methodologiesfor the factor-analysis of nonnormalLikert variables. Br. J. Math.Stat. Psychol. 38, 171–180.Muthén, B., and Kaplan, D. (1992).A comparison of some methodologiesfor the factor-analysis of nonnormalLikert variables: a note onthe size of the model. Br. J. Math.Stat. Psychol. 45, 19–30.Muthén, L., and Muthén, B. (2010).Mplus User’s Guide, 6th Edn. LosAngeles, CA: Muthén & Muthén.Nunnally, J.C., and Bernstein, I. H.(1994). Psychometric Theory, 3rdEdn. New York: McGraw-Hill.Olsson, U. (1979). Maximum likelihoodestimation of the polychoric correlationcoefficient. Psychometrika 44,443–460.Pek, J., and MacCallum, R. C. (2011).Sensitivity analysis in structuralequation models: cases and theirinfluence. Multivariate Behav. Res.46, 202–228.Pison, G., Rousseeuw, P. J., Filzmoser, P.,and Croux, C. (2003). Robust factoranalysis. J. Multivar. Anal. 84,145–172.Poon, W.-Y., and Wong, Y.-K. (2004). Aforward search procedure for identifyinginfluential observations in theestimation of a covariance matrix.Struct. Equ. Modeling 11, 357–374.Ramsey, J. B. (1969). Tests for specificationerrors in classical linear leastsquares regression analysis. J. R. Stat.Soc. Series B 31, 350–371.Rodgers, J. L. (2010). The epistemologyof mathematical and statisticalmodeling: a quiet methodologicalrevolution. Am. Psychol. 65, 1–12.Satorra, A., and Bentler, P. M. (1994).“Corrections to test statistics andstandard errors in covariance structureanalysis,” in Latent VariablesAnalysis: Applications for DevelopmentalResearch, eds A. von Eye andC. C. Clogg (Thousand Oaks, CA:SAGE Publications), 399–419.Savalei, V. (2011). What to do aboutzero frequency cells when estimatingpolychoric correlations. Struct. Equ.Modeling 18, 253–273.Spearman, C. (1904). General intelligenceobjectively determined andmeasured. Am.J.Psychol.5,201–293.Thurstone, L. L. (1947). Multiple FactorAnalysis. Chicago: University ofChicago Press.Weisberg, S. (1985). Applied LinearRegression, 2nd Edn. New York:Wiley.Frontiers in Psychology | Quantitative Psychology and Measurement March 2012 | Volume 3 | Article 55 |120
Page 2 and 3:
FRONTIERS COPYRIGHTSTATEMENT© Copy
Page 4 and 5:
Table of Contents05 Is Data Cleanin
Page 7 and 8:
OsborneAssumptions and data cleanin
Page 9 and 10:
ORIGINAL RESEARCH ARTICLEpublished:
Page 12 and 13:
Hoekstra et al.Why assumptions are
Page 14 and 15:
Page 16 and 17:
Page 18 and 19:
Page 20 and 21:
García-PérezStatistical conclusio
Page 22 and 23:
Page 24 and 25:
Page 26 and 27:
Page 28 and 29:
Page 30 and 31:
Sheng and ShengEffect of non-normal
Page 32 and 33:
Page 34 and 35:
Page 36 and 37:
Page 38 and 39:
Page 40 and 41:
Page 42 and 43:
REVIEW ARTICLEpublished: 12 April 2
Page 44 and 45:
Nimon et al.The assumption of relia
Page 46 and 47:
Page 48 and 49:
Page 50 and 51:
Page 52 and 53:
Page 54 and 55:
Page 56 and 57:
TressoldiPower replication unreliab
Page 58 and 59:
TressoldiPower replication unreliab
Page 60 and 61:
Page 62 and 63:
FinchModern methods for the detecti
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71: FinchModern methods for the detecti
Page 72 and 73: MINI REVIEW ARTICLEpublished: 28 Au
Page 74 and 75: NimonStatistical assumptionsand Del
Page 76 and 77: NimonStatistical assumptionsFor exa
Page 78 and 79: Kraha et al.Interpreting multiple r
Page 94 and 95: SmithsonComparing moderation of slo
Page 102 and 103: REVIEW ARTICLEpublished: 01 March 2
Page 104 and 105: Flora et al.Factor analysis assumpt
Page 124 and 125: Kasper and ÜnlüAssumptions of fac
Page 144 and 145: Lages and JaworskaHow predictable a
Page 152 and 153: Cummiskey et al.Testing assumptions
Page 158: Cummiskey et al.Testing assumptions
show all

Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?