View PDF Version - RePub - Erasmus Universiteit Rotterdam

Chapter 1
16
Prediction models with correction for overfi tting
To develop an individual baseline prediction to a dichotomous response of treatment
statistical logistic regression techniques are used. The aim of the fi nal model is to identify
which patient characteristics or disease specifi c factors are independently or in combination
with each other associated with response. As a result the prediction of response for
the individual patient can be assessed.
The design of a good model is a laborious process of comparing different strategies and
combinations of covariates using statistical measurements of model fi t and performance
measurements, such as the Akaike’s Information Criteria (AIC) and the Area Under the
receiver operating Curve (AUC), in combination with sound statistical knowledge and
logical sense, build on knowledge from previous studies and clinical experience. 25-26
When a well-fi tted and stable prediction model has fi nally been achieved the fi tted coeffi
cients needs to be corrected for overfi tting. Because the model has been designed on
a fi xed dataset it is known that the model in general will tend to overfi t when applied to a
new individual or dataset. As a result the predictive performance will therefore be worse.
The best option to solve this issue is to validate the model in an independent and similar
dataset. When this option is not available bootstrapping26 is an established method to
study the degree of overfi tting. Penalized likelihood estimation26 can also be used to fi t
a penalty-score to correct for overfi tting.
The fi nal step is to present the prediction model and the method will depend on the
audience and the user. The model in general is quite complex, and mathematical formulas
can present diffi culties in interpretation to some users. Instead nomograms or
graphical presentations can be used along with medical decision trees. Another option
might be the design of a website which generates a specifi c predicted probability of
response after entering the required patient specifi c variables.
Dynamic prediction models
During treatment patients will be monitored regularly. Dynamic update of an individual’s
prediction of response to treatment based on new information is not routinely implemented
in prediction models, but can be of great importance for the individual subject
and the further choice of treatment.
Two different statistical approaches are considered: (1) directly modelling the prediction
of the outcome variable with the use of logistic regression techniques and (2) indirectly
classifying individuals into an outcome category over time using Bayes’ theorem.
For the direct approach (1) either the observed marker value or the subject specifi c
pattern of the marker are used as predictors. A generalized estimating equations (GEE)