Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

More documents

Recommendations

Info

Lages and JaworskaHow predictable are voluntary decisions?STUDY 1: SPONTANEOUS MOTOR DECISIONSIn this replication study we investigated response bias and responsedependency of binary decisions in a spontaneous motor task. Weclosely replicated the study by Soon et al. (2008) in terms of stimuli,task, and instructions (Soon et al., 2008) but without monitoringfMRI brain activity since we are mainly interested in behavioralcharacteristics.METHODSSubjects were instructed to relax while fixating on the center of ascreen where a stream of random letters was presented in 500 msintervals. At some point, when participants felt the urge to do so,they immediately pressed one of two buttons with their left or rightindex finger. Simultaneously, they were asked to remember the letterthat appeared on the screen at the time when they believed theirdecision to press the button was made. Shortly afterward, the lettersfrom three preceding trials and an asterisk were presented onscreen randomly arranged in a two-by-two matrix. The participantswere asked to select the remembered letter in order to reportthe approximate time point when their decision was formed. Ifthe participant chose the asterisk it indicated that the rememberedletter was not among the three preceding intervals and thevoluntary decision occurred more than 1.5 s ago. Subjects wereasked to avoid any form of preplanning for choice of movementor time of execution.PARTICIPANTSAll participants (N = 20, age 17–25, 14 female) were students atGlasgow University. They were naïve as to the aim of the study,right-handed,and with normal or corrected-to-normal visual acuity.The study was conducted according to the Declaration ofHelsinki ethics guidelines. Informed written consent was obtainedfrom each participant before the study.RESULTSFollowing Soon et al. (2008) we computed for each participant thefrequency of a left or right response. If we assume that the spontaneousdecision task produces independent responses then theprocess can be modeled by a binomial distribution where probabilityfor a left and right response may vary from participant toparticipant.P[t(x) = s|θ] =( ns)θ s (1 − θ) n−sThe observed data t(x) is simply the sum of s left (right) responses,n is the total number of responses, and θ is a parameter that reflectsthe unknown probability of responding Left (Right) with θ ∈ [0,1].The hypothesis of a balanced response corresponds to a responserate of θ = 0.5. Rather than trying to affirm this null hypothesiswe can test whether the observed number of left (right) responsesdeviates significantly from the null hypothesis by computing thecorresponding p-value (two-sided).p (two - sided) =n−s∑x i =0t (x i ) +n∑t (x i )x i =sWe found that response frequencies of 4 out of 20 participants (2out of 20 if adjusted for multiple tests according to Sidak–Dunn)significantly deviated from a binomial distribution with equalprobabilities (p < 0.05, two-sided). Soon et al. (2008) excluded24 out of 36 participants who exceeded a response criterion that isequivalent to a binomial test with p < 0.11 (two-sided). Bode et al.(2011) applied a similar response criterion but did not documentselection of participants. They reported exclusion of a single participantfrom their sample of N = 12 due to relatively unbalanceddecisions and long trial durations; responses from the remaining11 subjects were included in their analyses. In the present study 8out of 20 participants did not meet Soon et al.’s response criterion(for details see Table A1 in Appendix).Selection of participants is a thorny issue. While the intentionmay have been to select participants who made truly spontaneousand therefore independent decisions they selected participantswho generated approximately balanced responses. This assumptionis fallible since subjects’ response probabilities are unlikely tobe perfectly balanced and the null hypothesis of θ = 0.5 can bedifficult to affirm.Excluding 2/3 of the subjects reduces generalizability of resultsand imposing the assumption of no response bias on the remainingsubjects seems inappropriate because these participants can stillhave true response probabilities θ that are systematically differentfrom 0.5.To give an example of how a moderate response bias mayaffect prediction accuracy of a trained classifier, consider a participantwho generates 12 left and 20 right responses in 32 trials.Although this satisfies the response criterion mentioned above, aclassifier trained on this data is susceptible to response bias. If theclassifier learns to match the individual response bias predictionaccuracy may exceed the chance level of 50%. (If, for example, theclassifier trivially predicts the more frequent response then thisstrategy leads to 62.5% rather than 50% correct predictions in ourexample.)To alleviate the problem of response bias Soon et al. (2008)and Bode et al. (2011) not only selected among participants butalso designated equal numbers of left (L) and right (R) responsesfrom the experimental trials before entering the data into theirclassification analysis. It is unclear how they sampled trials buteven if they selected trials randomly the voxel activities beforeeach decision are drawn from an experiment with unbalanced Land R responses. As a consequence the problem does not dissipatewith trial selection. After selecting an equal number of Land R responses from the original data set this subsample stillhas an unbalanced number of L and R responses in the precedingtrials so that the distribution of all possible pairs of successiveresponses in trial t − 1 and trial t (LL, LR, RR, RL) is not uniform.Since there are more Right responses in the original dataset we are more likely to sample more RR “stay” trials and lessLR “switch” trials as well as more RL “switch” trials compared toLL “stay” trials. The exact transition probabilities for these eventsdepend on the individual response pattern. Switching and stayingbetween successive responses creates a confounding variable thatmay introduce spurious correlations between voxel activities fromprevious responses and the predicted responses. This confoundmay be picked up when training a linear support vector machineFrontiers in Psychology | Quantitative Psychology and Measurement March 2012 | Volume 3 | Article 56 | 143
Lages and JaworskaHow predictable are voluntary decisions?(SVM) classifier to predict current responses from voxel activitiesin previous trials.Similar to Soon et al. (2008) and Bode et al. (2011) we first computedthe length and frequency of the same consecutive responses(L and R runs) for each participant and fitted the pooled and averageddata by an exponential function. However, here we fitted thepooled data with the (single-parameter) exponential probabilitydistribution functionf (x) = λe −λx , for x 0(see Figure 1A). We found reasonable agreement with the exponentialdistribution (R = 0.89) as an approximation of the geometricdistribution. The estimated parameter λ =−0.805 is equivalentto a response rate of θ = 1 − e −λ = 0.553, slightly elevated fromθ = 0.5.Although the exponential distribution suggests an independentand memoryless process such a goodness-of-fit does not qualify asevidence for independence and stationarity in the individual data.Averaging or pooling of data across blocks and participants canhide systematic trends and patterns in individual data.To avoid these sampling issues we applied the Wald–Wolfowitzor Runs test (MatLab, MathWorks Inc.) to each individualsequence of 32 responses. This basic non-parametric test is basedon the number of runs above and below the median and does notrely on the assumption that binary responses have equal probabilities(Kvam and Vidakovic, 2007). Our results indicate that 3out of the 12 selected participants in our replication study showstatistically significant (p < 0.05) departures from stationarity (2out of 12 adjusted for multiple tests). Similarly, approximatingthe binomial by a normal distribution with unknown parameters(Lilliefors, 1967), the Lilliefors test detected 4 out of 20 (4 out ofthe selected 12) statistically significant departures from normalityin our replication study (3 out of 20 and 1 out of 12 adjustedfor multiple tests). These violations of stationarity and normalitypoint to response dependencies between trials in at least some ofthe participants (for details see Table A1 in Appendix).CLASSIFICATION ANALYSISIn analogy to Soon et al. (2008) and Bode et al. (2011) we also performeda multivariate pattern classification. To assess how muchdiscriminative information is contained in the pattern of previousresponses rather than voxel activities, we included up to twopreceding responses to predict the following response within anindividual data set. Thereto, we entered the largest balanced setof left/right response trials and the unbalanced responses fromthe preceding trials into the analysis and assigned every 9 out of10 responses to a training data set. This set was used to traina linear SVM classifier (MatLab, MathWorks Inc.). The classifierestimated a decision boundary separating the two classes(Jäkel et al., 2007). The learned decision boundary was appliedto classify the remaining sets and to establish a predictive accuracy.This was repeated 10 times, each time using a differentsample of learning and test sets, resulting in a 10-fold crossvalidation. The whole procedure was bootstrapped a 100 timesto obtain a mean prediction accuracy and a measure of variabilityfor each individual data set (see Figure 1B; Table A1 inAppendix).One participant (Subject 11) had to be excluded from theclassification analysis because responses were too unbalanced totrain the classifier. For most other participants the classifier performedbetter when it received only one rather than two precedingresponses to predict the subsequent response and thereforewe report only classification results based on a single precedingresponse.If we select N = 12 participants according to the response criterionemployed by Soon et al. (2008) prediction accuracy for aresponse based on its preceding response reaches 61.6% whichis significantly higher than 50% (t(11) = 2.6, CI = [0.52–0.72],p = 0.013). If we include all participants except Subject 11 thenFIGURE 1|Study1(N = 12): Left/Right motor decision task. (A)Histogram for proportion of length of response runs pooled acrossparticipants. Superimposed in Red is the best fitting exponential distributionfunction. (B) Prediction accuracies of linear SVM trained on preceding andcurrent responses for individual data sets (black circles, error bars =±1SDbootstrapped) and group average (red circle, error bar =±1 SD).www.frontiersin.org March 2012 | Volume 3 | Article 56 | 144
Page 2 and 3:
FRONTIERS COPYRIGHTSTATEMENT© Copy
Page 4 and 5:
Table of Contents05 Is Data Cleanin
Page 7 and 8:
OsborneAssumptions and data cleanin
Page 9 and 10:
ORIGINAL RESEARCH ARTICLEpublished:
Page 12 and 13:
Hoekstra et al.Why assumptions are
Page 14 and 15:
Page 16 and 17:
Page 18 and 19:
Page 20 and 21:
García-PérezStatistical conclusio
Page 22 and 23:
Page 24 and 25:
Page 26 and 27:
Page 28 and 29:
Page 30 and 31:
Sheng and ShengEffect of non-normal
Page 32 and 33:
Page 34 and 35:
Page 36 and 37:
Page 38 and 39:
Page 40 and 41:
Page 42 and 43:
REVIEW ARTICLEpublished: 12 April 2
Page 44 and 45:
Nimon et al.The assumption of relia
Page 46 and 47:
Page 48 and 49:
Page 50 and 51:
Page 52 and 53:
Page 54 and 55:
Page 56 and 57:
TressoldiPower replication unreliab
Page 58 and 59:
TressoldiPower replication unreliab
Page 60 and 61:
Page 62 and 63:
FinchModern methods for the detecti
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
MINI REVIEW ARTICLEpublished: 28 Au
Page 74 and 75:
NimonStatistical assumptionsand Del
Page 76 and 77:
NimonStatistical assumptionsFor exa
Page 78 and 79:
Kraha et al.Interpreting multiple r
Page 80 and 81:
Page 82 and 83:
Page 84 and 85:
Page 86 and 87:
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
Page 94 and 95: SmithsonComparing moderation of slo
Page 102 and 103: REVIEW ARTICLEpublished: 01 March 2
Page 104 and 105: Flora et al.Factor analysis assumpt
Page 124 and 125: Kasper and ÜnlüAssumptions of fac
Page 146 and 147: Lages and JaworskaHow predictable a
Page 152 and 153: Cummiskey et al.Testing assumptions
Page 158: Cummiskey et al.Testing assumptions
show all

Sweating the Small Stuff: Does data cleaning and testing ... - Frontiers

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?