Statistical Methods in Medical Research 4ed

16.5 Model assessment and model choice 567
encountered when attempting to use the marginal distribution of y for model
checking. Various methods for overcoming this difficulty have been discussed
(see, for example, O'Hagan, 1994, Chapter 7), but all involve some degree of
arbitrariness.
Suppose the result of the model comparisons is to decide that model M1
should be used to describe the data. It would then be natural to base inferences
on the posterior distribution corresponding to that model. However, in the
notation of the present discussion, this amounts to basing inferences on
p…u1jy, M1†. This approach takes the model to be fixed and ignores the uncertainty
encountered in the choice of the model. This is a Bayesian analogue of the
problem that was illustrated in Fig. 12.8. There, two estimates of an asymptote
were found from two non-linear regression equations. The two estimates differed
by much more than their standard errors under each model. The standard errors
had taken no account of the uncertainty in choosing the model and the high
precisions ascribed to each estimate arose because in each case the underlying
model was taken as correct. In Bayesian terms this amounts to basing inferences
on p…u1jy, M1†.
The more formal framework provided by the Bayesian approach appears to
allow a resolution of this problem. Suppose that m models are available for the
data and each contains a parameter, t, about which inferences must be made. If
appropriate priors for the parameters in each model are specified, then the
marginal posterior distribution for t under model i can be determined as
p…tjy, M1† and the posterior distribution of t, unconditional on any model, can
be estimated by
p…tjy† ˆ Pm
p…tjy, Mj†p…Mjjy†: …16:20†
jˆ1
This technique is sometimes referred to as Bayesian model averaging. In a formal
sense it overcomes the problem of overstating the precision of estimators because
uncertainty in model selection has been overlooked. It also has implications for
model choice, in so far as the use of (16.20) for inference avoids the need to
decide on which model to use. Draper (1995) discusses many of these issues.
The procedure has a number of drawbacks. It can be difficult to decide on an
appropriate family of models, M1, ..., Mm, on which to base the model averaging.
On a purely technical level, if m is large (and basing the family of models
on all main-effects regressions for 10 covariates would give m ˆ 2 10 ), the computation
of (16.20) can be demanding. However, more fundamentally, the utility
of divorcing an inference from the underlying model will also be questionable in
many circumstances. Although superficially attractive, the practical utility of
many formal techniques for assessing model uncertainty has yet to find universal
acceptance; see, for example, the contributions of Cox and Tukey to the discussion
of Draper (1995).