Fundamentals of Clinical Research for Radiologists ROC Analysis

More documents

Recommendations

Info

ROC Analysisbased on increasing the values of the attenuationcoefficient. Other diagnostic tests must beinterpreted by a trained observer, often a radiologist,and so the interpretation is subjective.Two general scales are often used in radiologyfor observers to assign a value to their subjectiveinterpretation of an image. One scale is the5-point rank scale: 1 = definitely normal, 2 =probably normal, 3 = possibly abnormal orequivocal, 4 = probably abnormal, and 5 = definitelyabnormal.The other popular scale is the 0–100% confidencescale, where 0% implies that the observeris completely confident in the absenceof the disease of interest, and 100% impliesthat the observer is completely confident inthe presence of the disease of interest. Thetwo scales have strengths and weaknesses [2,8], but both are reasonably well suited to radiologyresearch. In mammography a ratingscale already exists, the BI-RADS score,which can be used to form decision thresholdsfrom least to most suspicion for the presenceof breast cancer.When the diagnostic test requires a subjectiveinterpretation by a trained reviewer, thereviewer becomes part of the diagnostic process[9]. Thus, to properly characterize the accuracyof the diagnostic test, we must includemultiple reviewers in the study. This is the socalledMRMC, multiple-reader multiplecase,ROC study. Much has been writtenabout the design and analysis of MRMC studies[10–20]. We mention here only the basicdesign of MRMC studies, and in a later subsectionwe describe their statistical analysis.The usual design for the MRMC study is afactorial design, in which every reviewer interpretsthe image (or images if there is morethan one test) of every patient. Thus, if thereare R reviewers, C patients, and I diagnostictests, then each reviewer interprets C × I images,and the study involves R × C × I total interpretations.The accuracy of each reviewerwith each diagnostic test is characterized byan ROC curve, so R × I ROC curves are constructed.Constructing pooled or consensusROC curves is not the goal of these studies.Rather, the primary goals are to document thevariability in diagnostic test accuracy betweenreviewers and report the average, ortypical, accuracy of reviewers. In order for theresults of the study to be generalizeable to therelevant patient and reviewer populations,representative samples from both populationsare needed for the study. Often expert reviewerstake part in studies of diagnostic test accuracy,but the accuracy for a nonexpert may beFig. 4.—Three receiver operating characteristic (ROC)curves with same binormal parameter A (i.e., A = 1.0)but different values for parameter B of 3.0 (3σ 1 = σ o ),1.0 (σ 1 = σ o ), and 0.33 (σ 1 = 3σ o ).When B = 3.0, ROCcurve dips below chance diagonal; this is called an improperROC curve [2].considerably less. An excellent illustration ofthe issues involved in sampling reviewers foran MRMC study can be found in the study byBeam et al. [21].Sensitivity1.00.80.60.40.20.00.0B = 3.3B = 1.00.2B = 3.00.40.6False-Positive Rate0.81.0Examples of ROC Studies in RadiologyThe radiology literature, and the clinicallaboratory and more general medical literature,contain many excellent examples of howROC curves are used to characterize the accuracyof a diagnostic test and to compare accuraciesof diagnostic tests. We briefly describehere three recent examples of ROC curves beingused in the radiology literature.Kim et al. [22] conducted a prospectivestudy to determine if rectal distention usingwarm water improves the accuracy of MRIfor preoperative staging of rectal cancer. AfterMRI, the patients underwent surgical resection,considered the gold standardregarding the invasion of adjacent structuresand regional lymph node involvement. Fourobservers, unaware of the pathology results,independently scored the MR images using 4-and 5-point rating scales. Using statisticalmethods for MRMC studies [13], the authorsdetermined that typical reviewers’ accuracyfor determining outer wall penetration is improvedwith rectum distention, but that revieweraccuracy for determining regionallymph node involvement is not affected.Osada et al. [23] used ROC analysis to assessthe ability of MRI to predict fetal pulmonaryhypoplasia. They imaged 87fetuses, measuring both lung volume andsignal intensity. An ROC curve based onlung volume showed that lung volume hassome ability to discriminate between fetuseswho will have good versus those who willhave poor respiratory outcome after birth.An ROC curve based on the combined informationfrom lung volume and signal intensity,however, has superior accuracy. Formore information on the optimal way tocombine measures or test results, see the articleby Pepe and Thompson [24].In a third study, Zheng et al. [25] assessedhow the accuracy of a mammographic computer-aideddetection (CAD) scheme wasaffected by restricting the maximum numberof regions that could be identified aspositive. Using a sample of 300 cases with amalignant mass and 200 normals, the investigatorsapplied their CAD system, each timereducing the maximum number of positiveregions that the CAD system could identifyfrom seven to one. A special ROC techniquecalled “free-response receiver operatingcharacteristic curves” (FROC) was used.The horizontal axis of the FROC curve differsfrom the traditional ROC curve in thatit gives the average number of false-positivesper image. Zheng et al. concluded thatlimiting the maximum number of positiveregions that the CAD could identify improvesthe overall accuracy of CAD in mammography.For more information on FROCcurves and related methods, I refer you toother articles [26–29].Statistical Methods for ROC AnalysisFitting Smooth ROC CurvesIn Figure 1 we saw the empiric ROC curvefor the test results in Table 1. The curve wasconstructed with line segments connectingthe observed points on the ROC curve. EmpiricROC curves often have a jagged appearance,as seen in Figure 1, and often lie slightlybelow the “true,” smooth, ROC curve—thatis, the test’s ROC curve if it were constructedwith an infinite number of points (not just thefour points in Fig. 1) and an infinitely largesample size. A smooth curve gives us a betteridea of the relationship between the diagnos-AJR:184, February 2005 367
Obuchowskitic test and the disease. In this subsection wedescribe some methods for constructingsmooth ROC curves.The most popular method of fitting asmooth ROC curve is to assume that the testresults (e.g., the BI-RADS scores in Table 1)come from two unobserved distributions, onedistribution for the patients with disease andone for the patients without the disease. Usuallyit is assumed that these two distributionscan be transformed to normal distributions,referred to as the binormal assumption. It isthe unobserved, underlying distributions thatwe assume can be transformed to follow abinormal distribution, and not the observedtest results. Figure 3 illustrates the hypothesizedunobserved binormal distribution estimatedfor the observed BI-RADS results inTable 1. Note how the distributions for thediseased and nondiseased patients overlap.Let the unobserved binormal variables forthe nondiseased and diseased patients havemeans µ 0 and µ 1 , and variances σ 0 [2] and σ 1[2], respectively. Then it can be shown [30]that the ROC curve is completed described bytwo parameters:A = (µ 1 – µ 0 ) / σ 1 (2)B = σ 0 / σ 1 . (3)(See Appendix 1 for a formula that linksparameters A and B to the ROC curve.) Figure4 illustrates three ROC curves. ParameterA was set to be constant at 1.0 and parameterB varies as follows: 0.33 (the underlying distributionof the diseased patients is threetimes more variable than that of the nondiseasedpatients), 1.0 (the two distributionshave the same SD), and 3.0 (the underlyingdistribution of the nondiseased patients isthree times more variable than that of the diseasedpatients). As one can see, the curvesdiffer dramatically with changes in parameterB. Parameter A, on the other hand, determineshow far the curve is above the chance diagonal(where A = 0); for a constant B parameter,the greater the value of A, the higher the ROCcurve lies (i.e., greater accuracy).Parameters A and B can be estimated fromdata such as in Table 1 using maximum likelihoodmethods [30, 31]. For the data in Table1, the maximum likelihood estimates (MLEs)of parameters A and B are 2.27 and 1.70, respectively;the smooth ROC curve is given inFigure 1. Fortunately, some useful software[32] has been written to perform the necessarycalculations of A and B, along with estimationof the area under the smooth curve(see next subsection), its SE and confidenceinterval (CI), and CIs for the ROC curve itself(see Appendix 1).Dorfman and Alf [30] suggested a statisticaltest to evaluate whether the binormal assumptionwas reasonable for a given data set. Others[33, 34] have shown through empiric investigationand simulation studies that many differentunderlying distributions are well approximatedby the binormal assumption.When the diagnostic test results are themselvesa continuous measurement (e.g., CTTABLE 2attenuation values, or measured lesion diameter),it may not be necessary to assume theexistence of an unobserved, underlying distribution.Sometimes continuous-scale testresults themselves follow a binormal distribution,but caution should be taken that thefit is good (see the article by Goddard andHinberg [35] for a discussion of the resultingbias when the distribution is not truly binormalyet the binormal distribution is assumed).Zou et al. [36] suggest using a Box-Cox transformation to transform data toEstimating Area Under Empirical Receiver Operating CharacteristicCurveTest ResultsScore xScore No. of PairsNondiseasedDiseasedNo. of PairsNormal Normal 1/2 65 x 5 = 325 162.5Normal Benign 1 65 x 15 = 975 975Normal Probably benign 1 65 x 10 = 650 650Normal Suspicious 1 65 x 60 = 3,900 3,900Normal Malignant 1 65 x 10 = 650 650Benign Normal 0 10 x 5 = 50 0Benign Benign 1/2 10 x 15 = 150 75Benign Probably benign 1 10 x 10 = 100 100Benign Suspicious 1 10 x 60 = 600 600Benign Malignant 1 10 x 10 = 100 100Probably benign Normal 0 15 x 5 = 75 0Probably benign Benign 0 15 x 15 = 225 0Probably benign Probably benign 1/2 15 x 10 = 150 75Probably benign Suspicious 1 15 x 60 = 900 900Probably benign Malignant 1 15 x 10 = 150 150Suspicious Normal 0 7 x 5 = 35 0Suspicious Benign 0 7 x 15 = 105 0Suspicious Probably benign 0 7 x 10 = 70 0Suspicious Suspicious 1/2 7 x 60 = 420 210Suspicious Malignant 1 7 x 10 = 70 70Malignant Normal 0 3 x 5 = 15 0Malignant Benign 0 3 x 15 = 45 0Malignant Probably benign 0 3 x 10 = 30 0Malignant Suspicious 0 3 x 60 = 180 0Malignant Malignant 1/2 3 x 10 = 30 15Total 10,000 pairs 8,632.5368 AJR:184, February 2005
Page 2 and 3: ROC Analysismost lenient decision t
Page 6 and 7: ROC AnalysisTABLE 3 Fictitious Data
Page 8 and 9: ROC Analysiswhere A is the paramete

Fundamentals of Clinical Research for Radiologists ROC Analysis

Create successful ePaper yourself

Delete template?

Save as template?