2DkcTXceO

A.S. Whittemore 399represented, for example, by participants in one of the cross-sectional studiesconducted by the National Health Interview Survey (NHANES). Methods areneeded to use these distributions to estimate the model performance measureswe would see if the model were applied to subjects whose covariates reflectthose of the general population.35.5.2 Evaluating risk models with case-control dataData from case-control studies nested within a cohort are not useful for evaluatingmodel calibration, which concerns the agreement between a model’sassigned risks and the actual probabilities of adverse outcome occurrencewithin a future risk period. Sampling only the outcome-positive and outcomenegativesubjects (ignoring the time at risk contributed by censored subjects)can lead to severe bias in calibration measures, due to overestimation of outcomeprobabilities (Whittemore and Halpern, 2013). However under certainassumptions, unbiased (though inefficient) estimates of discrimination measurescan be obtained from nested case-control studies. The critical assumptionis that the censoring be uninformative; i.e., that subjects censored at agiven follow-up time are a random sample of all cohort members alive andoutcome-free at that time (Heagerty et al., 2000). This assumption is reasonablefor the type of censoring encountered in most cohort studies. There isaneedtoevaluatetheefficiencylossinestimateddiscriminationmeasuresassociated with excluding censored subjects.However when interest centers on special populations, such as those at highrisk of the outcome, it may not be feasible to find case-control data nestedwithin a cohort to evaluate model discrimination. For example, we may wantto use breast cancer cases and cancer-free control women ascertained in a highriskcancer clinic to determine and compare discrimination of several modelsfor ten-year breast cancer risk. Care is needed in applying the risk models tonon-nested case-control data such as these, and interpreting the results. Tomimic the models’ prospective setting, two steps are needed: 1) the modelsmust assign outcome risks conditional on the absence of death during the riskperiod; and 2) subjects’ covariates must be assessed at a date ten years beforeoutcome assessment (diagnosis date for cases, date of interview for controls).In principle, the data can then be used to estimate ten-year breast cancerprobabilities ignoring the competing risk of death. In practice, the rules forascertaining cases and controls need careful consideration to avoid potentialselection bias (Wacholder et al., 1992).35.5.3 Designing and evaluating models formultiple outcomesValidating risk models that focus on a single adverse outcome (such as developingbreast cancer within ten years) involves estimating a woman’s ten-yearbreast cancer probability in the presence of co-morbidities causing her death