Ivancevic_Applied-Diff-Geom

326 Applied Differential Geometry: A Modern IntroductionComplexityNot only should a model describe the data in hand well, but it shouldalso do so in the least complex (i.e., simplest) way. Intuitively, complexityhas to do with a model’s inherent flexibility that enables it to fit a widerange of data patterns. There seem to be at least two dimensions of modelcomplexity, the number of parameters and the model’s functional form. Thelatter refers to the way the parameters are combined in the model equation.The more parameters a model has, the more complex it is. Importantly also,two models with the same number of parameters but different functionalforms can differ significantly in their complexity. For example, it seemsunlikely that two one–parameter models, y = x + w and y = e wx areequally complex. The latter is probably much better at fitting data thanthe former.It turns out that one can devise a quantitative measure of model complexitythat takes into account both dimensions of complexity and at thesame time is theoretically justified as well as intuitive. One example is thegeometric complexity (GC) of a model [Pitt et. al. (2002)] defined as:GC = k 2 ln n ∫2π + ln dw √ det I(w), (3.159)orGC = parametric complexity + functional complexity,where k is the number of parameters, n is the sample size, I(w) is theFisher information matrix (or, covariance matrix) defined asI ij (w) = −E [ ∂ 2 ln f(y|w)/∂w i ∂w j], i, j = 1, ..., k, (3.160)orI ij = −Expect.Value(Hessian(loglik (w))).Functional form effects of complexity are reflected in the second termof GC through I(w). How do we interpret geometric complexity? Themeaning of geometric complexity is related to the number of ‘different’(i.e., distinguishable) probability distributions that a model can accountfor. The more distinguishable distributions that the model can describe byfinely tuning its parameter values, the more complex it is ([Myung et. al.(2000a)]). For example, when geometric complexity is calculated for thefollowing two-parameter psychophysical models, Stevens’ law (y = w 1 x w2 )and Fechner’s logarithmic law (y = w 1 ln (x + w 2 )), the former turns outto be more complex than the latter [Pitt et. al. (2002)].