statisticalrethinkin..

206 6. MODEL SELECTION, COMPARISON, AND AVERAGINGe Akaike weight formula might look rather odd, but really all it is doing is putting AICcon a probability scale. So each weight will be a number from 0 to 1, and the weights togetheralways sum to 1. Now larger values are better.But what do these probabilities mean? ere actually isn’t a consensus about that. Buthere’s Akaike’s interpretation, which is common. 89A model’s weight is an estimate of the probability that the model will make thebest predictions on new data, conditional on the set of models considered.Here’s the heuristic explanation. First, regard AICc as the expected deviance of a model onfuture data. at is to say that AICc gives us an estimate of E(D test ). Akaike weights convertthese deviance values, which are log-likelihoods, to plain likelihoods and then standardizethem all. is is just like Bayes’ theorem uses a sum in the denominator to standardize theproduct of the likelihood and prior. erefore the Akaike weights are analogous to posteriorprobabilities of models, conditional on expected future data, and assuming equal priors. 90 Soyou can usefully read each weight as an estimated probability that each model will performbest on future data. In simulation at least, interpreting weights in this way turns out to beappropriate. 91In this analysis, the best model has 94% of the model weight. at’s pretty good. Notehowever that these weights are conditional upon the set of models considered. If you addanother model, or take one away, all of the weights will change. is is still a small worldanalysis, incapable of considering models that we have yet to imagine or analyze. Still, it’sclear that if we restrict ourselves to these four simple linear regressions, we benefit a lot fromusing both of the predictor variables.Notice as well that either predictor alone is actually expected to do worse than the modelwithout either predictor, m6.11. e expected difference is small, but since neither predictoralone improves the deviance very much, the penalty term knocks down the two models withonly one predictor.Overthinking: Calculating Akaike weights. e formula for Akaike weights looks very odd at first.Here’s the quick run down of its pieces. Deviance, and therefore AICc, is on a log-probability scale.So exponentiating restores it as regular probability. e factor of −0.5 just cancels out the −2 usedin computing deviance. e delta-AICc (δAICc) values are used instead of raw AICc values, becausethis makes the numerical calculations inside the computer more reliable. Specifically, exponentiatinga very negative number will sometimes round to zero. Try exp(-1000) for example. Rescalingto differences doesn’t change the weights, but does change accuracy. So always do it. Finally, eachexponentiated value is summed up in the denominator, to ensure that all of the individual weightssum to 1, like good probabilities.Rethinking: How a big a difference in AIC is “significant”? Newcomers to information criteriaoen ask whether a difference between AIC values is “significant.” For example, models m6.14 andm6.11 differ by 6.42 units of deviance. Is it possible to say whether this difference is big enoughto conclude that m6.14 is significantly better? In general, it is not possible to provide a principledthreshold of difference that makes one model “significantly” better than another, whatever that means.e same is actually true of ordinary significance testing—the 5% convention is just a convention. Wecould invent some convention for AIC, but it too would just a be a convention. Moreover, we knowthe models will not make the same predictions—they are different models. So “significance” in thiscontext must have a very different definition than usual.