Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

Kasper and ÜnlüAssumptions of factor analytic approachesaccuracies of the factor models can be compared. This is known as“rotational indeterminacy”in the factor analysis approach (e.g., seeMaraun, 1996). For more information, the reader is also referredto Footnote 8 and Section 7.Second, the criterion used to determine the number of factorsextracted from the data must be distinguished as well. In practice,not all k or p but instead ˆk < k or p factors with the ˆk largest eigenvaluesare extracted. Various procedures are available to determineˆk. Commonly used criteria in educational research are the Kaiser-Guttman criterion (Guttman, 1954; Kaiser and Dickman, 1959),the scree test (Cattell, 1966), and the method of parallel analysis(Horn, 1965).3. ASSUMPTIONS ASSOCIATED WITH THE CLASSICALFACTOR MODELSThe three models described in the previous section in particularassume interval-scaled data and full rank covariance or correlationmatrices for the manifest variables. Typically in the exploratoryfactor analysis model, the manifest variables y or the standardizedvariables z are assumed to be normally distributed. For the PCAand PAA models, we additionally want to presuppose – for computationalreasons – that the variances of the manifest variables aresubstantially large. The EFA and PAA models assume uncorrelatedfactor terms and uncorrelated error terms (which can be relaxedin the framework of structural equation models; e.g., Jöreskog,1966), uncorrelatedness between the error and latent ability variables,and expected values of zero for the errors as well as latentability variables.The question now arises whether the assumptions are criticalwhen it comes to educational tests or survey data? 43.1. CRITERION-REFERENCED TESTS AND PISA QUESTIONNAIREDATAFrom the perspective of applying these models to data of criterionreferencedtests, the last three of the above mentioned assumptionsare less problematic. For a criterion-referenced test, it is importantthat all items of the test are valid for the investigated content. Assuch, the usual way of excluding items from the analysis whenthe covariance or correlation matrices are not of full rank does notwork for criterion-referenced tests, because this can reduce contentvalidity of a test. A similar argument applies to the assumption of4 Note that, at the latent level, there is no formal assumption that the latent factors(what we synonymously also want to call “person abilities”) are normally distributed.At the manifest level, maximum likelihood estimation (EFA) assumes that theobserved variables are normal; ULS and GLS (EFA), PAA (EFA), and PCA do not.The latter two methods only require a non-singular correlation matrix (e.g., seeMacCallum, 2009). However, in applications, for example in empirical educationalresearch, one often assumes that the latent ability values follow a normal distributionin the population of reference. Moreover, Mattson (1997)’s method describedin Section 4.1 states that there is a connection between the manifest and latent distributionsin factor analysis. Hence the question is what implications one can expect ifthis “implicit assumption” may not be justified. Related to the study and evaluationof the underlying assumptions associated with these classical factor models, thispaper, amongst others, shows that the data re-expression method PCA performssurprisingly well if compared to the results obtained based on the latent variableapproaches EFA and PAA. Moreover, ML and PAA estimation procedures for EFAare compared with one another, for different degrees of violating the normalityassumption at the manifest or latent levels.substantially large variances of the manifest variables. As Klauer(1987) suggested and Sturzbecher et al. (2008) have shown forthe driving license test in Germany, the variances of the manifestvariables of criterion-referenced tests are seldom high, and in generalthe data obtained from those tests may lead to extracting toofew dimensions. However, for the analysis of criterion-referencedtests, the assumption of interval-scaled data and the assumptionof normality of the manifest test and latent ability scores are evenmore problematic. Data from criterion-referenced tests are rarelyinterval-scaled – instead the items of criterion-referenced tests areoften dichotomous (Klauer,1987). For criterion-referenced tests,itis plausible to have skewed (non-symmetric) test and ability scoredistributions, because criterion-referenced tests are constructedto assess whether a desired and excessive teaching goal has beenachieved or not. In other words, the tested population is explicitlyand intensively trained regarding the evaluated ability, and so it israther likely that most people will have high values on the manifesttest score as well as latent ability (e.g., see the German drivinglicense test; Sturzbecher et al., 2008).The assumption of interval-scaled data and the normalityassumption for the manifest test and latent ability scores mayalso be crucial for the scaling of cognitive data in PISA (OECD,2012; Chap. 9 therein). In PISA, the generated students’ scores areplausible values. These are randomly drawn realizations basicallyfrom a multivariate normal distribution (as the prior) of latentability values (person ability is modeled as a random effect, alatent variable), in correspondence to a fitted item response theorymodel (Adams et al., 1997) giving the estimated parameters of thenormal distribution. The mean of the multivariate normal distributionis expressed as linear regression of various direct manifestregressors (e.g., administered test booklet, gender) and indirect“latent” or complex regressors obtained by aggregating over manifestand latent context or background variables (e.g., indicatorsfor economic, social, and cultural status) in a principal componentanalysis. The component scores used in the scaling model as theindirect “latent” regressors are extracted, in the purely computationalsense, to account for approximately 95% of the total variancein all the original variables. The background variables may be categoricalor dummy-coded and may not be measured at an intervalscale (nor be normally distributed). So as we said before, if onecan assume that the distribution of latent background informationrevealed through questionnaires is skewed, we can expect thatthe empirical distribution of background information computedby principal component analysis is likely to be biased compared tothe true distribution. This is suggested by the results of our simulationstudy. The bias of the empirical distribution in turn mayresult in biasing the regression expression for the mean. Therefore,special caution has to be taken regarding possible violationsof those assumptions, and a minimum of related sensitivity analysesare required and necessary in order to control for their potentialeffects.3.2. HISTORICAL REMARKSThe primary aim is to review results of previous studies focusingon the impact of violations of model assumptions. As to ourknowledge, such studies did not systematically vary the distributionsof the factors (in the case of continuous data as well) andFrontiers in Psychology | Quantitative Psychology and Measurement March 2013 | Volume 4 | Article 109 | 125