Deep-Learning-with-PyTorch

420 CHAPTER 14 End-to-end nodule analysis, and where to go next
No one true way to measure false positives: Precision vs. false positive rate
The FPR here and the precision from chapter 12 are rates (between 0 and 1) that
measure things that are not quite opposites. As we discussed, precision is TP /
(TP + FP) and measures how many of the samples predicted to be positive will actually
be positive. The FPR is FP / (FP + TN) and measures how many of the actually
negative samples are predicted to be positive. For heavily imbalanced datasets (like
the nodule versus non-nodule classification), our model might achieve a very good
FPR (which is closely related to the cross-entropy criterion as a loss) while the precision—and
thus the F1 score—is still very poor. A low FPR means we’re weeding out
a lot of what we’re not interested in, but if we are looking for that proverbial needle,
we still have mostly hay.
For our nodule data, that’s from 3.25 mm (the smallest nodule) to 22.78 mm (the
largest). If we pick a threshold value somewhere between those two values, we can
then compute FPR(threshold) and TPR(threshold). If we set the FPR value to X and
TPR to Y, we can plot a point that represents that threshold; and if we instead plot the
FPR versus TPR for every possible threshold, we get a diagram called the receiver operating
characteristic (ROC) shown in figure 14.6. The shaded area is the area under the
(ROC) curve, or AUC. It is between 0 and 1, and higher is better. 5
1.0
roc diameter baseline, auc=0.901
true positive rate
0.8
0.6
0.4
0.2
5.42 Mm
threshold
10.55 Mm
Threshold
0.0
0.0 0.2
0.4 0.6
false positive rate
0.8 1.0
Figure 14.6
Receiver operating characteristic (ROC) curve for our baseline
5
Note that random predictions on a balanced dataset would result in an AUC of 0.5, so that gives us a floor for
how good our classifier must be.