Ivancevic_Applied-Diff-Geom

328 Applied Differential Geometry: A Modern IntroductionAIC and BICAIC and BIC for a given model are defined as follows:AIC = −2 ln f(y|w ∗ ) + 2k,BIC = −2 ln f(y|w ∗ ) + k ln n,where w ∗ is a MLE estimate, k is the number of parameters and n is thesample size. For normally distributed errors with constant variance, thefirst term of both criteria, -2 ln f(y|w ∗ ), is reduced to (n·ln(SSE(w ∗ )) + c o )where c o is a constant that does not depend upon the model. In eachcriterion, the first term represents a lack of fit measure, the second termrepresents a complexity measure, and together they represent a lack ofgeneralizability measure. A lower value of the criterion means better generalizability.Therefore, the model that minimizes a given criterion shouldbe chosen.Complexity in AIC and BIC is a function of only the number of parameters.Functional form, another important dimension of model complexity,is not considered. For this reason, these methods are not recommended forcomparing models with the same number of parameters but different functionalforms. The other two selection methods, CV and MDL, describednext, are sensitive to functional form as well as the number of parameters.Cross–ValidationIn CV, a model’s generalizability is estimated without defining an explicitmeasure of complexity. Instead, models with more complexity thannecessary to capture the regularity in the data are penalized through aresampling procedure, which is performed as follows: The observed datasample is divided into two sub–samples, calibration and validation. Thecalibration sample is then used to find the best–fitting values of a model’sparameters by MLE or LSE. These values, denoted by wcal ∗ , are then fixedand fitted, without any further tuning of the parameters, to the validationsample, denoted by y val . The resulting fit to y val by wcal ∗ is calledas the model’s CV index and is taken as the model’s generalizability estimate.If desired, this single–division–based CV index may be replaced bythe average CV index calculated from multiple divisions of calibration andvalidation samples. The latter is a more accurate estimate of the model’sgeneralizability, though it is also more computationally demanding.