A comparison of bootstrap methods and an adjusted bootstrap ...

More documents

Recommendations

Info

signals ( µ = 1.5 i.e. µ1= µ2in the simulation model). With these larger sample sizes(even with n=100), we find that when there are no real differences between the twoclasses, the out-of-bag estimation (OOB) and the leave-one-out cross-validation (LOOCV)still give larger variability compared to other methods; the resubstitution, the ordinarybootstrap, the bootstrap cross-validation and the .632 bootstrap still have substantialdownward bias (Case 1, Tables 2 and 3). When there are strong differences between theclasses, the LOOCV and OOB give smaller variability, and the resubstitution, theordinary bootstrap, the bootstrap cross-validation and the .632 bootstrap give smallerdownward bias (Case 2, Tables 2 and 3) as sample size n increases in comparison to Case2, Table 1. The .632+ bootstrap and the OOB perform better as sample size increases, butthey sometimes suffer from downward bias when n=40 and 100 (Case 2, Tables 2 and 3)and this is illustrated in Figure 3 using the results for n=40 with CART classifier.The adjusted bootstrap is robust in the sense that it remains conservative (has nodownward bias) under all circumstances considered in the simulation with varying signallevels, classifiers and sample sizes. It does not suffer from extremely large upward bias orvariability in comparison to other methods for small to moderate sized samples (Tables 1and 2), and it performs no worse than the competitors such as the .632+ bootstrap, theOOB and the LOOCV for larger sample sizes (Table 3).Insert Tables 2-3 and Figure 3 about here.Additional simulations are conducted for varying signals and n/p ratios (Tables A1, A2 insupplement). We found the comparative conclusion does not depend on the n/p ratios(with n
eported in the supplementary material for this paper available athttp://linus.nci.nih.gov/~brb/TechReport.htm.4.2 Lymphoma DataRosenwald et al. [17] conducted a microarray study among patients with large-B-celllymphoma and identified the germinal-center B-cell-like subgroup which had the highestfive-year survival rate after chemotherapy. The study measured 7399 genes on 240patients. In the following analysis, we define the classes of outcome by the lymphomasubgroups, the germinal-center B-cell-like as class 1, the activated B-cell-like and type 3as class 0. This expression data provides only moderate signals to distinguish the classes(Wright et al. [18]). To assess the performance of the methods described in Sections 2and 3, we repeatedly draw a sample of size n from the 240 patients, find the estimates forprediction error and use the remaining patients as an independent test set to calculate the“true” prediction error. This procedure is repeated R=1000 times. The number ofbootstrap replications is B=50 for the methods reviewed in Section 2 andrepeated leave-one-out bootstrap.B1= 20for theWe present the results for n=14 and n=20 in Table 4 and only DLDA classifier is appliedin this study. Feature selection is performed prior to each application of DLDA bychoosing the 10 genes having the largest absolute-value t -statistics. With this moderatesignal example, the methods behave in conformity with the trend we observe from theprevious simulation study. The ordinary bootstrap and the bootstrap cross-validation areless competitive because they underestimate the true prediction error; the out-of-bagestimation is a less stable variant of the leave-one-out bootstrap while the .632 bootstrapperforms well for both sample sizes. The leave-one-out cross-validation has the largestvariability. The leave-one-out bootstrap and the .632+ bootstrap overestimate the truth byabout 64% and 51% when n=14 and by 55% and 33% when n=20. With the adjustedbootstrap, the ratios of overestimation drop to 20% and 14% for n=14 and n=20.5. DISCUSSION15
Page 4 and 5: n> p situation, the .632+ bootstrap
Page 6 and 7: Ordinary Bootstrap method [4] has t
Page 8 and 9: ecomes e , this results in systema
Page 10 and 11: form e = am −α+b with j=1, …,
Page 12 and 13: In Table 1, we report the simulatio
Page 16 and 17: As CART is not a satisfactory class
Page 18 and 19: ootstrap method and demonstrated th
Page 20 and 21: Table 1. Simulation study with samp
Page 22 and 23: Table 3. Simulation study with samp
Page 24 and 25: BIASSTDMSECase 10.150.10.050-0.05-0
Page 26: BIASSTDMSECase 10.10.050-0.05-0.1-0

A comparison of bootstrap methods and an adjusted bootstrap ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?