statisticalrethinkin..

174 6. MODEL SELECTION, COMPARISON, AND AVERAGINGFIGURE 6.1. Ptolemaic (le) and Copernican (right) models of the solarsystem. Both models use epicycles (circles on circles), and both modelsproduce exactly the same predictions. However, the Copernican model requiresfewer circles.So instead of Ockham’s razor, think of Ulysses’ compass. Ulysses was the hero of Homer’sOdyssey. During his voyage, Ulysses had to navigate a narrow straight between the manyheadedbeast Scylla—who attacked from a cliff face and gobbled up sailors—and the seamonster Charybdis—who pulled boats and men down to a watery grave. Passing too closeto either meant disaster. In the context of scientific models, you can think of these monstersas representing two fundamental kinds of statistical error:(1) e many-headed beast of OVERFITTING, which leads to poor prediction by learningtoo much from the data(2) e whirlpool of UNDERFITTING, which leads to poor prediction by learning toolittle from the dataOur job is to carefully navigate between these monsters. ere are two common familiesof approaches. e first is to use a REGULARIZING PRIOR to tell the model not to gettoo excited by the data. e second is to use some scoring device, like INFORMATION CRI-TERIA, to explicitly model the prediction task. is chapter focuses on information criteria,because they are harder to explain and entail a number of new concepts. But both families ofapproaches are routinely used in the natural and social sciences, and frequently a procedurefrom one family can be shown to be equivalent to a procedure from the other. Furthermore,they can be maybe should be used in combination.is chapter informally develops and demonstrates two popular information criteria,AIC (Akaike information criterion) and DIC (Deviance information criterion). AIC andDIC provide an estimate of a model’s forecasting error. To build up to AIC and DIC, thischapter first discusses overfitting and underfitting in more detail. en there are two distinctsteps in justifying AIC and its relatives. First, a measure of model performance known asDEVIANCE is chosen for comparing models. Information theory justifies this choice. Second,