Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

More documents

Recommendations

Info

Kraha et al.Interpreting multiple regressionbe considered the most important variable (RIW = 0.241), followedby X3 (RIW = 0.045) and X2 (RIW = 0.015). The RIWsoffer an additional representation of the individual effect ofeach predictor while simultaneously considering the combinationof predictors as well (Johnson, 2000). The sum of theweights (0.241 + 0.045 + 0.015 = 0.301) is equal to R 2 . PredictorX1 can be interpreted as the most important variablerelative to other predictors (Johnson, 2001). The interpretationis consistent with a full DA, because both the individualpredictor contribution with the outcome variable (r X 1·Y ),and the potential multicollinearity (r X 1·X2 and r X 1·X3) withother predictors are accounted for. While the RIWs may differslightly compared to general dominance weights (e.g., 0.015and 0.016, respectively, for X2), the conclusions are the consistentwith those from a full DA. This method rank ordersthe variables with X1 as the most important, followed by X3and X2. The suppression role of X2, however, is not identifiedby this method, which helps explain its rank as third in thisprocess.DISCUSSIONPredictor variables are more commonly correlated than not inmost practical situations, leaving researchers with the necessity toaddressing such multicollinearity when they interpret MR results.Historically, views about the impact of multicollinearity on regressionresults have ranged from challenging to highly problematic.At the extreme, avoidance of multicollinearity is sometimes evenconsidered a prerequisite assumption for conducting the analysis.These perspectives notwithstanding, the current article haspresented a set of tools that can be employed to effectively interpretthe roles various predictors have in explaining variance in acriterion variable.To be sure, traditional reliance on standardized or unstandardizedweights will often lead to poor or inaccurate interpretationswhen multicollinearity or suppression is present in the data. Ifresearchers choose to rely solely on the null hypothesis statisticalsignificance test of these weights, then the risk of interpretive erroris noteworthy. This is primarily because the weights are heavilyaffected by multicollinearity, as are their SE which directly impactthe magnitude of the corresponding p values. It is this realitythat has led many to suggest great caution when predictors arecorrelated.Advances in the literature and supporting software technologyfor their application have made the issue of multicollinearity muchless critical. Although predictor correlation can certainly complicateinterpretation, use of the methods discussed here allowfor a much broader and more accurate understanding of theMR results regarding which predictors explain how much variancein the criterion, both uniquely and in unison with otherpredictors.In data situations with a small number of predictors or very lowlevels of multicollinearity, the interpretation method used mightnot be as important as results will most often be very similar.However, when the data situation becomes more complicated (asis often the case in real-world data, or when suppression exists asexampled here), more care is needed to fully understand the natureand role of predictors.CAUSE AND EFFECT, THEORY, AND GENERALIZATIONAlthough current methods are helpful, it is very important thatresearchers remain aware that MR is ultimately a correlationalbasedanalysis, as are all analyses in the general linear model.Therefore, variable correlations should not be construed as evidencefor cause and effect relationships. The ability to claim causeand effect are predominately issues of research design rather thanstatistical analysis.Researchers must also consider the critical role of theory whentrying to make sense of their data. Statistics are mere tools tohelp understand data, and the issue of predictor importance inany given model must invoke the consideration of the theoreticalexpectations about variable relationships. In different contextsand theories, some relationships may be deemed more or lessrelevant.Finally, the pervasive impact of sampling error cannot beignored in any analytical approach. Sampling error limits the generalizabilityof our findings and can cause any of the methodsdescribed here to be more unique to our particular sample thanto future samples or the population of interest. We should notassume too easily that the predicted relationships we observe willnecessarily appear in future studies. Replication continues to be akey hallmark of good science.INTERPRETATION METHODSThe seven approaches discussed here can help researchers betterunderstand their MR models, but each has its own strengths andlimitations. In practice, these methods should be used to informeach other to yield a better representation of the data. Below wesummarize the key utility provided by each approach.Pearson r correlation coefficientPearson r is commonly employed in research. However, as illustratedin the heuristic example, r does not take into accountthe multicollinearity between variables and they do not allowdetection of suppressor effects.Beta weights and structure coefficientsInterpretations of both β weights and structure coefficients providea complementary comparison of predictor contribution to theregression equation and the variance explained in the effect. Betaweights alone should not be utilized to determine the contributionpredictor variables make to a model because a variable mightbe denied predictive credit in the presence of multicollinearity.Courville and Thompson, 2001; see also Henson, 2002) advocatedfor the interpretation of (a) both β weights and structurecoefficients or (b) both β weights and correlation coefficients.When taken together, β and structure coefficients can illuminatethe impact of multicollinearity, reflect more clearly the ability ofpredictors to explain variance in the criterion, and identify suppressoreffects. However, they do not necessarily provide detailedinformation about the nature of unique and commonly explainedvariance, nor about the magnitude of the suppression.All possible subsets regressionAll possible subsets regression is exploratory and comes withincreasing interpretive difficulty as predictors are added toFrontiers in Psychology | Quantitative Psychology and Measurement March 2012 | Volume 3 | Article 44 | 83
Kraha et al.Interpreting multiple regressionthe model. Nevertheless, these variance portions serve as thefoundation for unique and common variance partitioning andfull DA.Commonality analysis, dominance analysis, and relative importanceweightsCommonality analysis decomposes the regression effect intounique and common components and is very useful for identifyingthe magnitude and loci of multicollinearity and suppression.DA explores predictor contribution in a variety of situations andprovides consistent conclusions with RIWs. Both general dominanceand RIWs provide alternative techniques to decomposingthe variance in the regression effect and have the desirable featurethat there is only one coefficient per independent variableto interpret. However, the existence of suppression is not readilyunderstood by examining general dominance weights or RIWs.Nor do the indices yield information regarding the magnitude andloci of multicollinearity.CONCLUSIONThe real world can be complex – and correlated.We hope the methodssummarized here are useful for researchers using regression toconfront this multicollinear reality. For both multicollinearity andsuppression, multiple pieces of information should be consultedto understand the results. As such, these data situations shouldnot be shunned, but simply handled with appropriate interpretiveframeworks. Nevertheless, the methods are not a panacea, andrequire appropriate use and diligent interpretation. As correctlystated by Wilkinson and the APA Task Force on Statistical Inference(1999),“Good theories and intelligent interpretation advancea discipline more than rigid methodological orthodoxy...Statisticalmethods should guide and discipline our thinking but shouldnot determine it” (p. 604).REFERENCESAiken, L. S., West, S. G., andMillsap, R. E. (2008). Doctorialtraining in statistics, measurement,and methodology in psychology:replication and extension ofAiken, West, Sechrest, and Reno’s(1990) survey of PhD programs inNorth America. Am. Psychol. 63,32–50.Azen, R., and Budescu, D. V. (2003).The dominance analysis approachto comparing predictors in multipleregression. Psychol. Methods 8,129–148.Braun, M. T., and Oswald, F. L. (2011).Exploratory regression analysis: atool for selecting models anddetermining predictor importance.Behav. Res. Methods 43, 331–339.Budescu, D. V. (1993). Dominanceanalysis: a new approach to the problemof relative importance of predictorsin multiple regression. Psychol.Bull. 114, 542–551.Budescu, D. V., and Azen, R. (2004).Beyond global measures of relativeimportance: some insights fromdominance analysis. Organ. Res.Methods 7, 341–350.Capraro, R. M., and Capraro, M.M. (2001). Commonality analysis:understanding variance contributionsto overall canonical correlationeffects of attitude toward mathematicson geometry achievement.Mult. Lin. Regression Viewpoints 27,16–23.Courville, T., and Thompson, B. (2001).Use of structure coefficients in publishedmultiple regression articles: isnot enough. Educ. Psychol. Meas. 61,229–248.Darlington, R. B. (1968). Multipleregression in psychological researchand practice. Psychol. Bull. 69,161–182.Henson, R. K. (2002). The logic andinterpretation of structure coefficientsin multivariate general linearmodel analyses. Paper Presentedat the Annual Meeting of the AmericanEducational Research Association,New Orleans.Henson, R. K., Hull, D. M., andWilliams, C. (2010). Methodologyin our education research culture:toward a stronger collective quantitativeproficiency. Educ. Res. 39,229–240.International Business Machines Corp.(2010). Can SPSS Help me Generatea File of Raw Data with a SpecifiedCorrelation Structure? Available at:https://www-304.ibm.com/support/docview.wss?uid=swg21480900Johnson, J. W. (2000). A heuristicmethod for estimating the relativeweight of predictor variables in multipleregression. Multivariate Behav.Res. 35, 1–19.Johnson, J. W. (2001). “Determiningthe relative importance of predictorsin multiple regression: practicalapplications of relative weights,” inAdvances in Psychology Research,Vol.V, eds F. Columbus and F. Columbus(Hauppauge, NY: Nova SciencePublishers), 231–251.Johnson, J. W. (2004). Factors affectingrelative weights: the influenceof sampling and measurement error.Organ. Res. Methods 7, 283–299.LeBreton, J. M., and Tonidandel, S.(2008). Multivariate relative importance:relative weight analysis tomultivariate criterion spaces. J. Appl.Psychol. 93, 329–345.Lindeman, R. H., Merenda, P. F., andGold, R. Z. (1980). Introduction toBivariate and Multivariate Analysis.Glenview, IL: Scott Foresman.Lorenzo-Seva, U., and Ferrando, P. J.(2011). FIRE: an SPSS program forvariable selection in multiple linearregression via the relative importanceof predictors. Behav. Res.Methods 43, 1–7.Lorenzo-Seva, U., Ferrando, P. J., andChico, E. (2010). Two SPSS programsfor interpreting multipleregression results. Behav. Res. Methods42, 29–35.Lumley, T. (2009). Leaps: RegressionSubset Selection. R Package Version2.9. Available at: http://CRAN.Rproject.org/package=leapsMadden, J. M., and Bottenberg, R. A.(1963). Use of an all possible combinationsolution of certain multipleregression problems. J. Appl. Psychol.47, 365–366.Morris, J. D. (1976). A computer programto accomplish commonalityanalysis. Educ. Psychol. Meas. 36,721–723.Nimon, K. (2010). Regression commonalityanalysis: demonstration ofan SPSS solution. Mult. Lin. RegressionViewpoints 36, 10–17.Nimon, K., Henson, R., and Gates,M. (2010). Revisiting interpretationof canonical correlation analysis:a tutorial and demonstrationof canonical commonality analysis.Multivariate Behav. Res. 45,702–724.Nimon, K., Lewis, M., Kane, R., andHaynes, R. M. (2008). An R packageto compute commonality coefficientsin the multiple regressioncase: an introduction to the packageand a practical example. Behav. Res.Methods 40, 457–466.Nimon, K., and Reio, T. (2011). Regressioncommonality analysis: a techniquefor quantitative theory building.Hum. Resour. Dev. Rev. 10,329–340.Nimon, K., and Roberts, J. K. (2009).Yhat: Interpreting Regression effects.R Package Version 1.0-3. Available at:http://CRAN.R-project.org/package=yhatNunnally, J.C., and Bernstein, I. H.(1994). Psychometric Theory, 3rdEdn. New York: McGraw-Hill.Osborne, J., and Waters, E. (2002). Fourassumptions of multiple regressionthat researchers should always test.Practical Assessment, Research &Evaluation, 8(2). Available at: http://PAREonline.net/getvn.asp?v=8&n=2[accessed December 12, 2011]Pedhazur, E. J. (1997). Multiple Regressionin Behavioral Research: Explanationand Prediction, 3rd Edn. FortWorth, TX: Harcourt Brace.Rowell, R. K. (1991). Partitioning predictedvariance into constituentparts: how to conduct commonalityanalysis. Paper Presented atthe Annual Meeting of the SouthwestEducational Research Association,San Antonio.Rowell, R. K. (1996). “Partitioningpredicted variance into constituentparts: how to conduct commonalityanalysis,” in Advances in Socialscience Methodology, Vol. 4, ed. B.Thompson (Greenwich, CT: JAIPress), 33–44.Schneider, W. J. (2008). Playing statisticalouija board with commonalityanalysis: good questions, wrongassumptions. Appl. Neuropsychol. 15,44–53.Stevens, J. P. (2009). Applied MultivariateStatistics for the Social Sciences,4th Edn. New York: Routledge.Thompson, B. (2006). Foundations ofBehavioral Statistics: An Insight-Based Approach. New York: GuilfordPress.Thompson, B., and Borrello, G. M.(1985). The importance of structurecoefficients in regression research.Educ. Psychol. Meas. 45, 203–209.www.frontiersin.org March 2012 | Volume 3 | Article 44 | 84
Page 2 and 3:
FRONTIERS COPYRIGHTSTATEMENT© Copy
Page 4 and 5:
Table of Contents05 Is Data Cleanin
Page 7 and 8:
OsborneAssumptions and data cleanin
Page 9 and 10:
ORIGINAL RESEARCH ARTICLEpublished:
Page 12 and 13:
Hoekstra et al.Why assumptions are
Page 14 and 15:
Page 16 and 17:
Page 18 and 19:
ORIGINAL RESEARCH ARTICLEpublished:
Page 20 and 21:
García-PérezStatistical conclusio
Page 22 and 23:
Page 24 and 25:
Page 26 and 27:
Page 28 and 29:
Page 30 and 31:
Sheng and ShengEffect of non-normal
Page 32 and 33:
Sheng and ShengEffect of non-normal
Page 34 and 35: Sheng and ShengEffect of non-normal
Page 42 and 43: REVIEW ARTICLEpublished: 12 April 2
Page 44 and 45: Nimon et al.The assumption of relia
Page 56 and 57: TressoldiPower replication unreliab
Page 58 and 59: TressoldiPower replication unreliab
Page 60 and 61: ORIGINAL RESEARCH ARTICLEpublished:
Page 62 and 63: FinchModern methods for the detecti
Page 72 and 73: MINI REVIEW ARTICLEpublished: 28 Au
Page 74 and 75: NimonStatistical assumptionsand Del
Page 76 and 77: NimonStatistical assumptionsFor exa
Page 78 and 79: Kraha et al.Interpreting multiple r
Page 94 and 95: SmithsonComparing moderation of slo
Page 102 and 103: REVIEW ARTICLEpublished: 01 March 2
Page 104 and 105: Flora et al.Factor analysis assumpt
Page 124 and 125: Kasper and ÜnlüAssumptions of fac
Page 134 and 135:
Kasper and ÜnlüAssumptions of fac
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Lages and JaworskaHow predictable a
Page 146 and 147:
Page 148 and 149:
Page 150 and 151:
Page 152 and 153:
Cummiskey et al.Testing assumptions
Page 154 and 155:
Page 156 and 157:
Page 158:
show all

Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?