Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

More documents

Recommendations

Info

Kasper and ÜnlüAssumptions of factor analytic approachesTable 1 | Theoretical values of skewness and kurtosis for z i (fourfactors).Table 2 | Theoretical values of skewness and kurtosis for z i (eightfactors).z iLatent variablez iLatent variableNormal aSlightlyskewed bStronglyskewed cNormal aSlightlyskewed bStronglyskewed c√β1i β 2i√β1i β 2i√β1i β 2i√β1i β 2i√β1i β 2i√β1i β 2iz 1 0 3 −0.060 3 −0.599 4.202z 2 0 3 −0.049 3 −0.488 3.914z 3 0 3 −0.038 3 −0.377 3.649z 4 0 3 −0.027 3 −0.272 3.420z 5 0 3 −0.018 3 −0.179 3.240z 6 0 3 −0.010 3 −0.102 3.114z 7 0 3 −0.049 3 −0.488 3.914z 8 0 3 −0.038 3 −0.377 3.649z 9 0 3 −0.027 3 −0.272 3.420z 10 0 3 −0.018 3 −0.179 3.240z 11 0 3 −0.010 3 −0.102 3.114z 12 0 3 −0.005 3 −0.047 3.041z 13 0 3 −0.027 3 −0.272 3.420z 14 0 3 −0.027 3 −0.272 3.420z 15 0 3 −0.018 3 −0.179 3.240z 16 0 3 −0.010 3 −0.102 3.114z 17 0 3 −0.010 3 −0.102 3.114z 18 0 3 −0.005 3 −0.047 3.041z 19 0 3 −0.027 3 −0.272 3.420z 20 0 3 −0.018 3 −0.179 3.240z 21 0 3 −0.018 3 −0.179 3.240z 22 0 3 −0.010 3 −0.102 3.114z 23 0 3 −0.005 3 −0.047 3.041z 24 0 3 −0.005 3 −0.047 3.041aµ 1m = 0, µ 2m = 1, µ 3m = 0, and µ 4m = 3.bµ 1m = 0, µ 2m = 1, µ 3m = −0.20, and µ 4m = 3.cµ 1m = 0, µ 2m = 1, µ 3m = −2, and µ 4m = 9.z 1 0 3 −0.060 3 −0.599 4.202z 2 0 3 −0.049 3 −0.488 3.914z 3 0 3 −0.038 3 −0.377 3.649z 4 0 3 −0.049 3 −0.488 3.914z 5 0 3 −0.049 3 −0.488 3.914z 6 0 3 −0.038 3 −0.377 3.649z 7 0 3 −0.049 3 −0.488 3.914z 8 0 3 −0.038 3 −0.377 3.649z 9 0 3 −0.027 3 −0.272 3.420z 10 0 3 −0.038 3 −0.377 3.649z 11 0 3 −0.038 3 −0.377 3.649z 12 0 3 −0.038 3 −0.377 3.649z 13 0 3 −0.038 3 −0.377 3.649z 14 0 3 −0.027 3 −0.272 3.420z 15 0 3 −0.027 3 −0.272 3.420z 16 0 3 −0.027 3 −0.272 3.420z 17 0 3 −0.027 3 −0.272 3.420z 18 0 3 −0.018 3 −0.179 3.240z 19 0 3 −0.018 3 −0.179 3.240z 20 0 3 −0.010 3 −0.102 3.114z 21 0 3 −0.010 3 −0.102 3.114z 22 0 3 −0.010 3 −0.102 3.114z 23 0 3 −0.010 3 −0.102 3.114z 24 0 3 −0.005 3 −0.047 3.041aµ 1m = 0, µ 2m = 1, µ 3m = 0, and µ 4m = 3.bµ 1m = 0, µ 2m = 1, µ 3m = −0.20, and µ 4m = 3.cµ 1m = 0, µ 2m = 1, µ 3m = −2, and µ 4m = 9.are 12 conditions and for every condition 100 data sets weresimulated.Each of the generated 1,200 data sets were analyzed using allof the models of principal component analysis, exploratory factoranalysis (ML estimation), and principal axis analysis altogetherwith a varimax rotation (Kaiser, 1958). 8 For any data set undereach model, the factors, and hence, the numbers of retained factorswere determined by applying the following three extractioncriteria or approaches: the Kaiser-Guttman criterion, the scree test,and the parallel analysis procedure. 9Table 3 | Summary of the simulation design and number of generateddata sets.SamplesizeNumber offactorsLatent variable distributionNormal Slightly skewed Strongly skewed200 4 100 100 1008 100 100 100600 4 100 100 1008 100 100 1008 Because of rotational indeterminacy in the factor analysis approach (e.g., Maraun,1996), the results are as much an evaluation of varimax rotation as they are anevaluation of the manipulated variables in the study. For more information, seeSection 7.9 The Kaiser-Guttman criterion is a poor way to determine the number of factors.However, due to the fact that none of the existing studies has investigated the estimationaccuracy of this criterion when the latent ability distribution is skewed, we havedecided to include the Kaiser-Guttman criterion in our study. This criterion may4.3. EVALUATION CRITERIAThe criteria for evaluating the performance of the classical factormodels are the number of extracted factors (as compared totrue dimensionality), the skewness of the estimated latent abilityalso be viewed as a “worst performing” baseline criterion, which other extractionmethods need to outperform, as best as possible.Frontiers in Psychology | Quantitative Psychology and Measurement March 2013 | Volume 4 | Article 109 | 129
Kasper and ÜnlüAssumptions of factor analytic approachesdistribution, and the discrepancy between the estimated and thetrue loading matrix. The latter two criteria are computed usingthe true number of factors. Furthermore, Shapiro-Wilk tests forassessing normality of the ability estimates are presented anddistributions of the estimated and true factor scores are compared.For the skewness criterion, under a factor model and a simulationcondition, for any data set the factor scores on a factor werecomputed and their empirical skewness was the value for this dataset that was used and plotted. For the discrepancy criterion, undera factor model and a simulation condition, for any data set i = 1,. . ., 100 a discrepancy measure D i was calculated,D i =∑ px=1∑ ky=1| ˆl i;xy −l xy |,kpwhere ˆl i;xy and l xy represent the entries of the estimated (varimaxrotated, for data set i = 1, . . ., 100) and true loading matrices,respectively. It gives the averaged sum of the absolute differencesbetween the estimated and true factor loadings. We also report theaverage and variance (or standard deviation) of these discrepancymeasures, over all simulated data sets,D = 1 ∑100D i and s 2 1 ∑100=(D i − D) 2 .100100 − 1i=1i=1In addition to calculating estimated factor score skewness values,we also tested for univariate normality of the estimated factorscores. We used the Shapiro-Wilk test statistic W (Shapiro andWilk, 1965). In comparison to other univariate normality tests,the Shapiro-Wilk test seems to have relatively high power (Seier,2002). In our study, under a factor model and a simulation condition,for any data set the Shapiro-Wilk test statistic’s p-value wascalculated for the estimated factor scores on a factor and the distributionof the p-values obtained from 100 simulated data setswas plotted.5. RESULTSWe present the results of our simulation study.5.1. NUMBER OF EXTRACTED FACTORSFigure 2 shows the relative frequencies of the numbers of extractedfactors for sample size n = 200 and k = 4 as true number of factors.If the Kaiser-Guttman criterion is used, the number of extractedfactors is overestimated (for PCA) or tends to be underestimated(for EFA and PAA). With the scree test, four dimensions whereextracted in the majority of cases, but variation of the numbersof extracted factors over the different data sets is high. High variationin this case can be explained by the ambiguous and hencedifficult to interpret eigenvalue graphics that one needs to visuallyinspect for the scree test. Applying the parallel analysis method,variation of the numbers of extracted factors can be reduced andthe true number of factors is estimated very well (e.g., for PCA).There does not seem to be a relationship between the number ofextracted factors and the underlying distribution (normal, slightlyskewed, strongly skewed) of the latent ability values.When sample size is increased to n = 600, variation of the estimatednumbers of factors decreases substantially under manyconditions (see Figure 3). Compared to small sample sizes, thescree test and the parallel analysis method perform very well. TheKaiser-Guttman criterion still leads to a biased estimation of thetrue number of factors. Once again, there seems to be no relationshipbetween the distribution of the latent ability values and thenumber of extracted factors.Figure 4, for a sample size of n = 200, shows the case whenthere are k = 8 factors underlying the data. The Kaiser-Guttmancriterion again leads to overestimation or underestimation of thetrue number of factors. The extraction results for the scree testhave very high variation, and estimation of the true number offactors is least biased when the parallel analysis method is used.Increasing sample size from n = 200 to 600 results in a significantreduction of variation (Figure 5). However, the true numberof factors can be estimated without bias only when the parallelanalysis method is used as extraction criterion. A possible relationshipbetween the distribution of the latent ability values andthe number of extracted factors once again does not seem to beapparent.To sum up, we suppose that the“number of factors extracted” isrelatively robust against the extent the latent ability values may beskewed. Another observation is that the parallel analysis methodseems to outperform the scree test and the Kaiser-Guttman criterionwhen it comes to detecting the number of underlyingfactors.5.2. SKEWNESS OF THE ESTIMATED LATENT ABILITY DISTRIBUTIONFigure 6A shows the distributions of the estimated factor scoreskewness values, for n = 200, k = 4, and µ 3m = 0. The majority ofthe skewness values lies in close vicinity of 0. In other words, for atrue normal latent ability distribution with skewness µ 3 = 0, underthe classical factor models the estimated latent ability scores mostlikely seem to have skewness values of approximately 0. An impactof the factor model used for the analysis of the data on the skewnessof the estimated latent ability values cannot be seen under this simulationcondition. However, the standard deviations of the skewnessvalues clearly decrease from the first to the fourth factor. Inother words,the true skewness of the latent ability distribution maybe more precisely estimated for the fourth factor than for the first.When true latent ability values are slightly negative skewed,µ 3 = −0.20, in our simulation study this skewness may onlybe properly estimated for the first and second extracted factors(Figure 6B). The estimated latent ability values of the third andfourth extracted factors more give skewness values of approximately0. The true value of skewness for these factors hence maylikely to be overestimated.If true latent ability values are strongly negative skewed,µ 3 = −2, unbiased estimation of true skewness may not be possible(Figure 6C). Even in the case of the first and second factors,the estimation is biased now. True skewness of the latent abilitydistribution may be overestimated regardless of the used factormodel or factor position.To sum up, under the classical factor models, the concept of“skewness of the estimated latent ability distribution” seems to besensitive with respect to the extent the latent ability values may beskewed. It seems that, the more the true latent ability values areskewed, the greater is overestimation of true skewness. In otherwww.frontiersin.org March 2013 | Volume 4 | Article 109 | 130
Page 2 and 3:
FRONTIERS COPYRIGHTSTATEMENT© Copy
Page 4 and 5:
Table of Contents05 Is Data Cleanin
Page 7 and 8:
OsborneAssumptions and data cleanin
Page 9 and 10:
ORIGINAL RESEARCH ARTICLEpublished:
Page 12 and 13:
Hoekstra et al.Why assumptions are
Page 14 and 15:
Page 16 and 17:
Page 18 and 19:
Page 20 and 21:
García-PérezStatistical conclusio
Page 22 and 23:
Page 24 and 25:
Page 26 and 27:
Page 28 and 29:
Page 30 and 31:
Sheng and ShengEffect of non-normal
Page 32 and 33:
Page 34 and 35:
Page 36 and 37:
Page 38 and 39:
Page 40 and 41:
Page 42 and 43:
REVIEW ARTICLEpublished: 12 April 2
Page 44 and 45:
Nimon et al.The assumption of relia
Page 46 and 47:
Page 48 and 49:
Page 50 and 51:
Page 52 and 53:
Page 54 and 55:
Page 56 and 57:
TressoldiPower replication unreliab
Page 58 and 59:
TressoldiPower replication unreliab
Page 60 and 61:
Page 62 and 63:
FinchModern methods for the detecti
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
MINI REVIEW ARTICLEpublished: 28 Au
Page 74 and 75:
NimonStatistical assumptionsand Del
Page 76 and 77:
NimonStatistical assumptionsFor exa
Page 78 and 79:
Kraha et al.Interpreting multiple r
Page 80 and 81: Kraha et al.Interpreting multiple r
Page 94 and 95: SmithsonComparing moderation of slo
Page 102 and 103: REVIEW ARTICLEpublished: 01 March 2
Page 104 and 105: Flora et al.Factor analysis assumpt
Page 124 and 125: Kasper and ÜnlüAssumptions of fac
Page 144 and 145: Lages and JaworskaHow predictable a
Page 152 and 153: Cummiskey et al.Testing assumptions
Page 158: Cummiskey et al.Testing assumptions
show all

Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

Create successful ePaper yourself

Delete template?

Save as template?