Ivancevic_Applied-Diff-Geom

Applied Manifold Geometry 327GeneralizabilityThe fourth quantitative criterion for model evaluation is generalizability.This criterion is defined as a model’s ability to fit not only the observed dataat hand, but also new, as yet unseen data samples from the same probabilitydistribution. In other words, model evaluation should not be focused solelyon how well a model fits observed data, but how well it fits data generatedby the cognitive process underlying the data. This goal will be achievedbest when generalizability is considered.To summarize, these four quantitative criteria work together to assistin model evaluation and guide (even constrain) model development andselection. The model must be sufficiently complex, but not too complex, tocapture the regularity in the data. Both a good fit to the data and goodgeneralizability will ensure an appropriate degree of complexity, so that themodel captures the regularity in the data. In addition, because of its broadfocus, generalizability will constrain the power of the model, thus makingit falsifiable. Although all four criteria are inter-related, generalizabilitymay be the most important. By making it the guiding principle in modelevaluation and selection, one cannot go wrong.3.11.4.4 Selection Among Different ModelsSince a model’s generalizability is not directly observable, it must be estimatedusing observed data. The measure developed for this purpose tradesoff a model’s fit to the data with its complexity, the aim being to select themodel that is complex enough to capture the regularity in the data, but notoverly complex to capture the ever-present random variation. Looked at inthis way, generalizability embodies the principle of Occam’s razor (or principleof parsimony, i.e., the requirement of maximal simplicity of cognitivemodels).Model Selection MethodsNow, we describe specific measures of generalizability. Four representativegeneralizability criteria are introduced. They are the Akaike InformationCriterion (AIC), the Bayesian Information Criterion (BIC), crossvalidation(CV), and minimum description length (MDL) (see a special Journalof Mathematical Psychology issue on model selection, in particular [Myunget. al. (2000b)]). In all four methods, the maximized log–likelihood is usedas a goodness–of–fit measure, but they differ in how model complexity isconceptualized and measured.