A comparison of bootstrap methods and an adjusted bootstrap ...

Table 1. Simulation study with sample of size 20 and 800 genes. Two simulation setups are considered. Case 1: Nogenes are differentially expressed between the two classes. Case 2: Gene expression levels of class 0 patients follownormal distribution with mean 0; for class 1 patients, the 2% differentially expressed genes follow normal mixtures,half with mean µ = 0.5 and half with mean µ1 2= 1.5 . The “true” prediction error e is the misclassification ratenwhen a prediction rule built on the sample is tested on 1000 independent data with the same structure.Classifier Prediction Error Estimation Case 1: No differential genesCase 2: 2% differential genesMethod Est. STD Bias MSE Est. STD Bias MSEDLDA “True” Error ( e n )0.500 0.016 0.184 0.067Resubstitution 0.009 0.020 -0.491 0.242 0.006 0.017 -0.177 0.036Ordinary Bootstrap 0.196 0.022 -0.304 0.093 0.130 0.036 -0.054 0.006Bootstrap Cross Validation 0.205 0.024 -0.295 0.088 0.139 0.037 -0.045 0.006.632 Bootstrap 0.344 0.039 -0.157 0.026 0.229 0.064 0.045 0.007LOOCV 0.527 0.206 0.026 0.043 0.206 0.152 0.022 0.019Out-Of-Bag Estimation 0.590 0.156 0.090 0.033 0.243 0.153 0.059 0.022Leave-One-Out Bootstrap 0.538 0.059 0.038 0.005 0.359 0.098 0.175 0.038.632+ Bootstrap 0.516 0.054 0.015 0.003 0.318 0.111 0.134 0.027RLOOB:l=1 0.539 0.058 0.039 0.005 0.358 0.098 0.175 0.038RLOOB:l=2 0.537 0.098 0.036 0.011 0.278 0.121 0.095 0.020RLOOB:l=10 0.532 0.160 0.032 0.027 0.217 0.136 0.034 0.015Adjusted Bootstrap 0.534 0.128 0.033 0.018 0.237 0.133 0.053 0.0161NN “True” Error ( e n )0.501 0.016 0.211 0.071Resubstitution 0 0 -0.501 0.251 0 0 -0.211 0.050Ordinary Bootstrap 0.194 0.021 -0.306 0.094 0.127 0.038 -0.085 0.010Bootstrap Cross Validation 0.205 0.022 -0.296 0.088 0.136 0.040 -0.075 0.009.632 Bootstrap 0.342 0.036 -0.159 0.027 0.223 0.067 0.012 0.004LOOCV 0.529 0.184 0.029 0.035 0.241 0.166 0.030 0.020Out-of-Bag Estimation 0.600 0.161 0.100 0.036 0.243 0.161 0.031 0.018Leave-One-Out Bootstrap 0.541 0.058 0.040 0.005 0.354 0.106 0.142 0.027.632+ Bootstrap 0.518 0.051 0.018 0.003 0.312 0.118 0.100 0.018RLOOB:l=1 0.541 0.055 0.041 0.005 0.354 0.105 0.143 0.027RLOOB:l=2 0.536 0.090 0.036 0.010 0.289 0.127 0.077 0.015RLOOB:l=10 0.532 0.140 0.032 0.021 0.247 0.144 0.036 0.014Adjusted Bootstrap 0.533 0.114 0.032 0.014 0.258 0.139 0.046 0.014CART “True” Error ( e n )0.500 0.016 0.290 0.089Resubstitution 0 0 -0.500 0.250 0 0 -0.290 0.092Ordinary Bootstrap 0.188 0.020 -0.312 0.098 0.129 0.041 -0.162 0.033Bootstrap Cross Validation 0.197 0.022 -0.303 0.092 0.135 0.042 -0.156 0.031.632 Bootstrap 0.331 0.034 -0.169 0.030 0.227 0.073 -0.064 0.013LOOCV 0.528 0.225 0.029 0.516 0.326 0.194 0.035 0.038Out-of-Bag Estimation 0.574 0.180 0.074 0.038 0.253 0.150 -0.038 0.022Leave-One-Out Bootstrap 0.524 0.054 0.024 0.004 0.359 0.115 0.068 0.018.632+ Bootstrap 0.505 0.056 0.005 0.003 0.320 0.126 0.030 0.016RLOOB:l=1 0.523 0.053 0.023 0.004 0.358 0.114 0.067 0.018RLOOB:l=2 0.521 0.099 0.021 0.010 0.350 0.136 0.060 0.023RLOOB:l=10 0.520 0.173 0.020 0.030 0.331 0.170 0.041 0.032Adjusted Bootstrap 0.520 0.135 0.020 0.019 0.343 0.155 0.053 0.028DLDA: diagonal linear discriminant analysis; 1NN: one nearest neighbor; CART: classification and regression tree.RLOOB: repeated leave-one-out bootstrap20