2DkcTXceO

A.S. Whittemore 397the observed flattening of the estimated outcome probability curves in theright tails is an artifact of the NN method. Such flattening reflects the clumpingof sparse subjects in the tails into the same neighborhood to estimate asingle common outcome probability. New methods are needed to address theseissues.Model discrimination is commonly assessed using the concordance statisticor C-statistic, also called the area under the receiver-operating-characteristiccurve (Hanley and McNeil, 1982; Pepe, 2003). This statistic estimates theprobability that the risk assigned to a randomly sampled individual who developsthe outcome exceeds that of a randomly sampled individual who doesnot. The C-statistic has several limitations. Like all summary statistics, it failsto indicate subgroups for whom a model discriminates poorly, or subgroupsfor which one model discriminates better than another. In addition, patientsand health professionals have difficulty interpreting it. A more informativemeasure is the Case Risk Percentile (CRP), defined for each outcome-positivesubject (case) as the percentile of his/her assigned risk in the distribution ofassigned risks of all outcome-negative subjects. The CRP equals 1 SPV, whereSPV denotes her standardized placement value (Pepe and Cai, 2004; Pepe andLongton, 2005). The CRP can be useful for comparing the discrimination oftwo risk models.For example, Figure 35.3 shows the distribution of CRPs for 81 breastcancer cases in the NY-BCFR data, based on the BCRAT & IBIS models. Eachpoint in the figure corresponds to a subject who developed breast cancer within10 years of baseline. Each of the 49 points above the diagonal represents a casewhose IBIS CRP exceeds her BCRAT CRP (i.e., IBIS better discriminates herrisk from that of non-cases than does BCRAT), and the 32 points below theline represent cases for whom BCRAT discriminates better than IBIS. (Notethat CRPs can be computed for any assigned risk, not just those of cases.)Amodel’sC-statistic is just the mean of its CRPs, averaged over all cases.Importantly, covariates associated with having a CRP above or below thediagonal line can indicate which subgroups are better served by one modelthan the other. The CRPs are individualized measures of model sensitivity.Research is needed to develop alternatives to the C-statistic that are moreuseful for evaluating model discrimination. Further discussion of this issue canbe found in Pepe et al. (2010) and Pepe and Janes (2008).35.5 Can we improve how we use epidemiological datafor risk model assessment?We need better methods to accommodate the inherent limitations of epidemiologicaldata for assessing risk model performance. For example, the subjectsin large longitudinal cohort studies are highly selected, so that findings may