Deep-Learning-with-PyTorch

Why isn’t the model learning to detect nodules?
315
11.10 Why isn’t the model learning to detect nodules?
Our model is clearly learning something—the loss trend lines are consistent as epochs
increase, and the results are repeatable. There is a disconnect, however, between what
the model is learning and what we want it to learn. What’s going on? Let’s use a quick
metaphor to illustrate the problem.
Imagine that a professor gives students a final exam consisting of 100 True/False
questions. The students have access to previous versions of this professor’s tests going
back 30 years, and every time there are only one or two questions with a True answer.
The other 98 or 99 are False, every time.
Assuming that the grades aren’t on a curve and instead have a typical scale of 90%
correct or better being an A, and so on, it is trivial to get an A+: just mark every question
as False! Let’s imagine that this year, there is only one True answer. A student like
the one on the left in figure 11.13 who mindlessly marked every answer as False would
get a 99% on the final but wouldn’t really demonstrate that they had learned anything
(beyond how to cram from old tests, of course). That’s basically what our model is
doing right now.
1. F
6. F
2. F 7. F
3. F 8. F
4. F 9. F
5. F 10. F
1. F 6. F
2. F 7. F
3. T 8. F
4. F9
9. T
5. F 10. F
Figure 11.13 A professor
giving two students the same
grade, despite different levels
of knowledge. Question 9 is
the only question with an
answer of True.
Contrast that with a student like the one on the right who also got 99% of the questions
correct, but did so by answering two questions with True. Intuition tells us that
the student on the right in figure 11.13 probably has a much better grasp of the material
than the all-False student. Finding the one True question while only getting one
answer wrong is pretty difficult! Unfortunately, neither our students’ grades nor our
model’s grading scheme reflect this gut feeling.
We have a similar situation, where 99.7% of the answers to “Is this candidate a nodule?”
are “Nope.” Our model is taking the easy way out and answering False on every
question.
Still, if we look back at our model’s numbers more closely, the loss on the training
and validation sets is decreasing! The fact that we’re getting any traction at all on the
cancer-detection problem should give us hope. It will be the work of the next chapter
to realize this potential. We’ll start chapter 12 by introducing some new, relevant