statisticalrethinkin..

194 6. MODEL SELECTION, COMPARISON, AND AVERAGING(a)(b)Density0.0 0.2 0.4 0.6N = 1Density0.00 0.02 0.04 0.06N = 100 2 5 10 15-20 -10 0 10 20Dtest - DtrainDtest - Dtrain(c)(d)deviance48 52 56 602.1 4N = 205.3 7.59.9deviance25 30 352.1 4.4N = 106.29.7141 2 3 4 5parameters1 2 3 4 5parametersFIGURE 6.6. Simulations of the AIC gambit. (a) e distribution of D test −D train , from 10-thousand simulations. Mean difference marked by the verticaldashed line. (b) Like before, but now with N = 10. (c) D train and D testas a function of model complexity. Each point is the mean of 10-thousandsimulations, each with sample size N = 20. e filled points are training,and the empty points are testing. e solid black line displays AIC for eachmodel. (d) Like before, but now with only N = 10. AICc now displayed bythe dashed line.Panel (d) displays the same analysis, but now with half the sample size, N = 10. Nowthe more complex models with four and five parameters overfit more than AIC expects themto, as indicated by the open points lying above the solid black line. e dashed line at thetop shows AICc, which recall uses the penalty 2k/(1 − (k + 1)/N). In this example, AICcanticipates the accelerating overfitting as k approaches N. But it is also rather conservative,overestimating D test by sometimes a large amount. Being conservative isn’t always the bestapproach, but peer review tends to favor conservative analyses, so using AICc in this kind ofsituation is conventional.However, notice that in this case both AIC and AICc nominate the same model as best,the simplest model with one parameter. is model is also the model with the best D test . isis true even though the correct model is the model with three parameters. is situation is