Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

Flora et al.Factor analysis assumptionsproduct-moment correlations were attenuated which led to negativelybiased factor loading estimates (especially with fewerresponse categories), whereas polychoric correlations remainedaccurate and produced accurate factor loadings. When theobserved variable set contained a mix of positively and negativelyskewed items (Cases 3 and 4), product-moment correlationswere strongly affected by the direction of skewness (especiallywith fewer response categories), which can lead to dramaticallymisleading factor analysis results. Unfortunately, strongly skeweditems can also be problematic for factor analyses of polychoric R inpart because they produce sparse observed frequency tables, whichleads to higher rates of improper solutions and inaccurate results(e.g., Flora and Curran, 2004; Forero et al., 2009). Yet this difficultyis alleviated with larger sample size and more item-responsecategories; if a proper solution is obtained, then the results of afactor analysis of polychoric R are more trustworthy than thoseobtained with product-moment R.GENERAL DISCUSSION AND CONCLUSIONFactor analysis is traditionally and most commonly an analysis ofthe correlations (or covariances) among a set of observed variables.A unifying theme of this paper is that if the correlationsbeing analyzed are misrepresentative or inappropriate summariesof the relationships among the variables, then the factor analysisis compromised. Thus, the process of data screening and assumptiontesting for factor analysis should begin with a focus on theadequacy of the correlation matrix among the observed variables.In particular, analysis of product-moment correlations or covariancesimplies that the bivariate association between two observedvariables can be adequately modeled with a straight line, whichin turn leads to the expression of the common factor model asa linear regression model. This crucial linearity assumption takesprecedence over any concerns about having normally distributedvariables, although normality is important for certain model fitstatistics and estimating parameter SEs. Other concerns aboutthe appropriateness of the correlation matrix involve collinearityand potential influence of unusual cases. Once one acceptsthe sufficiency of a given correlation matrix as a representationof the observed data, the actual formal assumptions of the commonfactor model are relatively mild. These assumptions are thatthe unique factors (i.e., the residuals, ε,inEq. 1) are uncorrelatedwith each other and are uncorrelated with the common factors(i.e., η in Eq. 1). Substantial violation of these assumptions typicallymanifests as poor model-data fit, and is otherwise difficult toassess with a priori data-screening procedures based on descriptivestatistics or graphs.After reviewing the common factor model, we gave separatepresentations of issues concerning factor analysis of continuousobserved variables and issues concerning factor analysis of categorical,item-level observed variables. For the former, we showedhow concepts from regression diagnostics apply to factor analysis,given that the common factor model is itself a multivariate multipleregression model with unobserved explanatory variables. Animportant point was that cases that appear as outliers in univariateor bivariate plots are not necessarily influential and conversely thatinfluential cases may not appear as outliers in univariate or bivariateplots (though they often do). If one can determine that unusualobservations are not a result of researcher or participant error,thenwe recommend the use of robust estimation procedures instead ofdeleting the unusual observation. Likewise, we also recommendthe use of robust procedures for calculating model fit statisticsand SEs when observed, continuous variables are non-normal.Next, a crucial message was that the linearity assumption isnecessarily violated when the common factor model is fitted toproduct-moment correlations among categorical, ordinally scaleditems, including the ubiquitous Likert-type items. At worst (e.g.,with a mix of positively and negatively skewed dichotomousitems), this assumption violation has severe consequences forfactor analysis results. At best (e.g., with symmetric items withfive response categories), this assumption violation still producesbiased factor-loading estimates. Alternatively, factor analysis ofpolychoric R among item-level variables explicitly specifies a nonlinearlink between the common factors and the observed variables,and as such is theoretically well-suited to the analysis ofitem-level variables. However, this method is also vulnerable tocertain data characteristics, particularly sparseness in the bivariatefrequency tables for item pairs, which occurs when stronglyskewed items are analyzed with a relatively small sample. Yet, factoranalysis of polychoric R among items generally produces superiorresults compared to those obtained with product-moment R,especially if there are five or fewer item-response categories.We have not yet directly addressed the role of sample size. Inshort,no simple rule-of-thumb regarding sample size is reasonablygeneralizable across factor analysis applications. Instead, adequatesample size depends on many features of the research, such as themajor substantive goals of the analysis, the number of observedvariables per factor, closeness to simple structure, and the strengthof the factor loadings (MacCallum et al., 1999, 2001). Beyond theseconsiderations, having a larger sample size can guard against someof the harmful consequences of unusual cases and assumptionviolation. For example, unusual cases are less likely to exert stronginfluence on model estimates as overall sample size increases. Conversely,removing unusual cases decreases the sample size, whichreduces the precision of parameter estimation and statistical powerfor hypothesis tests about model fit or parameter estimates. Yet,having a larger sample size does not protect against the negativeconsequences of treating categorical item-level variables as continuousby factor analyzing product-moment R. But we did illustratethat larger sample size produces better results for factor analysis ofpolychoric R among strongly skewed items, in part because largersample size reduces the occurrence of sparseness.In closing, we emphasize that factor analysis, whether EFA orCFA, is a method for modeling relationships among observedvariables. It is important for researchers to recognize that itis impossible for a statistical model to be perfect; assumptionswill always be violated to some extent in that no model canever exactly capture the intricacies of nature. Instead, researchersshould strive to find models that have an approximate fit todata such that the inevitable assumption violations are trivial,but the models can still provide useful results that help answerimportant substantive research questions (see MacCallum, 2003,and Rodgers, 2010, for discussions of this principle). We recommendextensive use of sensitivity analyses and cross-validation toaid in this endeavor. For example, researchers should comparewww.frontiersin.org March 2012 | Volume 3 | Article 55 | 119