Fundamentals of Clinical Research for Radiologists ROC Analysis

ROC Analysismost lenient decision threshold whereby alltest results are positive for disease. An empiricROC curve has h − 1 additional coordinates,where h is the number of unique testresults in the sample. In Table 1 there are 200test results, one for each of the 200 patients inthe sample, but there are only five unique results:normal, benign, probably benign, suspicious,and malignant. Thus, h = 5, and thereare four coordinates plotted in Figure 1 correspondingto the four decision thresholds describedin Table 1.The line connecting the (0, 0) and (1, 1)coordinates is called the “chance diagonal”and represents the ROC curve of a diagnostictest with no ability to distinguish patientswith versus those without disease. An ROCcurve that lies above the chance diagonal,such as the ROC curve for our fictitiousmammography example, has some diagnosticability. The further away an ROC curve isfrom the chance diagonal, and therefore, thecloser to the upper left-hand corner, the betterdiscriminating power and diagnostic accuracythe test has.In characterizing the accuracy of a diagnostic(or screening) test, the ROC curve ofthe test provides much more informationabout how the test performs than just a singleestimate of the test’s sensitivity and specificity[1, 2]. Given a test’s ROC curve, a cliniciancan examine the trade-offs in sensitivityversus specificity for various decision thresholds.Based on the relative costs of false-positiveand false-negative errors and the pretestprobability of disease, the clinician canchoose the optimal decision threshold foreach patient. This idea is discussed in moredetail in a later section of this article. Often,TABLE 1patient management is more complex than isallowed with a decision threshold that classifiesthe test results into positive or negative.For example, in mammography suspiciousand malignant findings are usually followedup with biopsy, probably benign findings usuallyresult in a follow-up mammogram in 3–6months, and normal and benign findings areconsidered negative.When comparing two or more diagnostictests, ROC curves are often the only validmethod of comparison. Figure 2 illustratestwo scenarios in which an investigator,comparing two diagnostic tests, could bemisled by relying on only a single sensitivity–specificity pair. Consider Figure 2A. Supposea more expensive or risky test (represented byROC curve Y) was reported to have thefollowing accuracy: sensitivity = 0.40,specificity = 0.90 (labeled as coordinate 1 inFig. 2A); a less expensive or less risky test(represented by ROC curve X) was reportedto have the following accuracy: sensitivity =0.80, specificity = 0.65 (labeled as coordinate2 in Fig. 2A). If the investigator is looking forthe test with better specificity, then he or shemay choose the more expensive, risky test,not realizing that a simple change in thedecision threshold of the less expensive,cheaper test could provide the desiredspecificity at an even higher sensitivity(coordinate 3 in Fig. 2A).Now consider Figure 2B. The ROC curvefor test Z is superior to that of test X for a narrowrange of FPRs (0.0–0.08); otherwise, diagnostictest X has superior accuracy. Acomparison of the tests’ sensitivities at lowFPRs would be misleading unless the diagnostictests are useful only at these low FPRs.Construction of Receiver Operating Characteristic Curve Based onFictitious Mammography DataMammography Results Pathology/Follow-Up Results Decision Rules 1–4(BI-RADs Score)Not Malignant Malignant FPR SensitivityNormal 65 5 (1) 35/100 95/100Benign 10 15 (2) 25/100 80/100Probably benign 15 10 (3) 10/100 70/100Suspicious 7 60 (4) 3/100 10/100Malignant 3 10Total 100 100Note.—Decision rule 1 classifies normal mammography findings as negative; all others are positive. Decision rule 2classifies normal and benign mammography findings as negative; all others are positive. Decision rule 3 classifies normal,benign, and probably benign findings as negative; all others are positive. Decision rule 4 classifies normal, benign,probably benign, and suspicious findings as negative; malignant is the only finding classified as positive. BI-RADS =Breast Imaging Reporting and Data System, FPR = false-positive rate.To compare two or more diagnostic tests, itis convenient to summarize the tests’ accuracieswith a single summary measure. Severalsuch summary measures are used in the literature.One is Youden’s index, defined as sensitivity+ specificity − 1 [2]. Note, however,that Youden’s index is affected by the choiceof the decision threshold used to define sensitivityand specificity. Thus, different decisionthresholds yield different values of theYouden’s index for the same diagnostic test.Another summary measure commonlyused is the probability of a correct diagnosis,often referred to simply as “accuracy” in theliterature. It can be shown that the probabilityof a correct diagnosis is equivalent toprobability (correct diagnosis) = PREV s ×sensitivity + (1 – PREV s ) × specificity, (1)where PREV s is the prevalence of disease inthe sample. That is, this summary measure ofaccuracy is affected not only by the choice ofthe decision threshold but also by the prevalenceof disease in the study sample [2]. Thus,even slight changes in the prevalence of diseasein the population of patients being testedcan lead to different values of “accuracy” forthe same test.Summary measures of accuracy derivedfrom the ROC curve describe the inherent accuracyof a diagnostic test because they are notaffected by the choice of the decision thresholdand they are not affected by the prevalence ofdisease in the study sample. Thus, these summarymeasures are preferable to Youden’s indexand the probability of a correct diagnosis[2]. The most popular summary measure of accuracyis the area under the ROC curve, oftendenoted as “AUC” for area under curve. Itranges in value from 0.5 (chance) to 1.0 (perfectdiscrimination or accuracy). The chancediagonal in Figure 1 has an AUC of 0.5. In Figure2A the areas under both ROC curves arethe same, 0.841. There are three interpretationsfor the AUC: the average sensitivity over allfalse-positive rates; the average specificityover all sensitivities [3]; and the probabilitythat, when presented with a randomly chosenpatient with disease and a randomly chosen patientwithout disease, the results of the diagnostictest will rank the patient with disease ashaving higher suspicion for disease than thepatient without disease [4].The AUC is often too global a summarymeasure. Instead, for a particular clinical application,a decision threshold is chosen sothat the diagnostic test will have a low FPRAJR:184, February 2005 365