Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

García-PérezStatistical conclusion validitymeasures from signal detection theory analyses). The differencebetween regression and structural relations is briefly mentioned inpassing in many elementary books on regression,the issue of fittingstructural relations (sometimes referred to as Deming’s regressionor the errors-in-variables regression model) is addressed in detail inmost intermediate and advance books on regression (e.g., Fuller,1987; Draper and Smith, 1998) and hands-on tutorials have beenpublished (e.g., Cheng and Van Ness, 1994; Dunn and Roberts,1999; Dunn, 2007). But this type of analysis is not in the toolboxof the average researcher in psychology 1 . In contrast, recourse tothis type analysis is quite common in the biomedical sciences.Use of this commendable method may generalize whenresearchers realize that estimates of the slope β and the intercept αof a structural relation can be easily computed throughˆβ =√ ( 2Sy 2 − λS2 x + Sy x) 2 − λS2 + 4λS 2 xy2S xy, (1)ˆα = Ȳ − ˆβ ¯X, (2)where ¯X, Ȳ , Sx 2, S2 y , and S xy are the sample means, variances, andcovariance of X and Y, and λ = σε 2 y/σε 2 xis the ratio of the variancesof measurement errors in Y and in X. When X and Y arethe same variable measured at different times or under differentconditions (as in Maylor and Rabbitt, 1993; Baddeley and Wilson,2002), λ = 1 can safely be assumed (for an actual application, seeSmith et al., 2004). In other cases, a rough estimate can be used,as the estimates of α and β have been shown to be robust exceptunder extreme departures of the guesstimated λ from its true value(Ketellapper, 1983).For illustration, consider Yeshurun et al. (2008) comparison ofsignal detection theory estimates of d ′ in each of the intervals of atwo alternative forced-choice task, which they pronounced differentas revealed by a regression analysis through the origin. Notethat this is the context in which Isaac (1970) had illustrated theinappropriateness of regression. The data are shown in Figure 1,and Yeshurun et al. rejected equality of d 1 ′ and d′ 2 because theregression slope through the origin (red line, whose slope is 0.908)differed significantly from unity: The 95% confidence interval forthe slope ranged between 0.844 and 0.973. Using Eqs 1 and 2,the estimated structural relation is instead given by the blue linein Figure 1. The difference seems minor by eye, but the slope ofthe structural relation is 0.963, which is not significantly differentfrom unity (p = 0.738, two-tailed; see Isaac, 1970, p. 215). Thisoutcome, which reverses a conclusion raised upon inadequate dataanalyses, is representative of other cases in which the null hypothesisH 0 : β = 1 was rejected. The reason is dual: (1) the slope ofa structural relation is estimated with severe bias through regression(Riggs et al., 1978; Kalantar et al., 1995; Hawkins, 2002) and1 SPSS includes a regression procedure called “two-stage least squares” which onlyimplements the method described by Mandansky (1959) as “use of instrumentalvariables” to estimate the slope of the relation between X and Y. Use of this methodrequires extra variables with specific characteristics (variables which may simply notbe available for the problem at hand) and differs meaningfully from the simpler andmore generally applicable method to be discussed nextdˆ’ 25432100 1 2 3 4 5dˆ’ 1FIGURE 1 | Replot of data from Yeshurun et al. (2008, their Figure 8)with their fitted regression line through the origin (red line) and afitted structural relation (blue line). The identity line is shown withdashed trace for comparison. For additional analyses bearing on the SCV ofthe original study, see García-Pérez and Alcalá-Quintana (2011).(2) regression-based statistical tests of H 0 : β = 1 render empiricalType-I error rates that are much higher than the nominal ratewhen both variables are measured with error (García-Pérez andAlcalá-Quintana, 2011).In sum, SCV will improve if structural relations instead ofregression equations were fitted when both variables are measuredwith error.CONCLUSIONType-I and Type-II errors are essential components of the statisticaldecision theory underlying NHST and, therefore, data cannever be expected to answer a research question unequivocally.This paper has promoted a view of SCV that de-emphasizes considerationof these unavoidable errors and considers instead twoalternative issues: (1) whether statistical tests are used that matchthe research design, goals of the study, and formal characteristicsof the data and (2) whether they are applied in conditions underwhich the resultant Type-I and Type-II error rates match thosethat are declared as limiting the validity of the conclusion. Someexamples of common threats to SCV in these respects have beendiscussed and simple and feasible solutions have been proposed.For reasons of space, another threat to SCV has not been coveredin this paper, namely, the problems arising from multiple testing(i.e., in concurrent tests of more than one hypothesis). Multipletesting is commonplace in brain mapping studies and some implicationson SCV have been discussed, e.g., by Bennett et al. (2009),Vul et al. (2009a,b), and Vecchiato et al. (2010).All the discussion in this paper has assumed the frequentistapproach to data analysis. In closing, and before commenting onFrontiers in Psychology | Quantitative Psychology and Measurement August 2012 | Volume 3 | Article 325 | 22