Deep-Learning-with-PyTorch

434 CHAPTER 14 End-to-end nodule analysis, and where to go next
The general message is that there are subtle ways for bias to creep into our models.
We should use extra care to control information leakage at every step of the way and
verify its absence using independent data as much as possible. The price to pay for taking
shortcuts is failing egregiously at a later stage, at the worst possible time: when
we’re closer to production.
14.7 What next? Additional sources of inspiration (and data)
Further improvements will be difficult to measure at this point. Our classification validation
set contains 154 nodules, and our nodule classification model is typically getting
at least 150 of them right, with most of the variance coming from epoch-by-epoch
training changes. Even if we were to make a significant improvement to our model, we
don’t have enough fidelity in our validation set to tell whether that change is an
improvement for certain! This is also very pronounced in the benign versus malignant
classification, where the validation loss zigzags a lot. If we reduced our validation
stride from 10 to 5, the size of our validation set would double, at the cost of one-ninth
of our training data. That might be worth it if we wanted to try other improvements.
Of course, we would also need to address the question of a test set, which would take
away from our already limited training data.
We would also want to take a good look at the cases where the network does not
perform as well as we’d like, to see if we can identify any pattern. But beyond that, let’s
talk briefly about some general ways we could improve our project. In a way, this section
is like section 8.5 in chapter 8. We will endeavor to fill you with ideas to try; don’t
worry if you don’t understand each in detail. 10
14.7.1 Preventing overfitting: Better regularization
Reflecting on what we did throughout part 2, in each of the three problems—the classifiers
in chapter 11 and section 14.5, as well as the segmentation in chapter 13—we
had overfitting models. Overfitting in the first case was catastrophic; we dealt with it
by balancing the data and augmentation in chapter 12. This balancing of the data to
prevent overfitting has also been the main motivation to train the U-Net on crops
around nodules and candidates rather than full slices. For the remaining overfitting,
we bailed out, stopping training early when the overfitting started to affect our validation
results. This means preventing or reducing overfitting would be a great way to
improve our results.
This pattern—get a model that overfits, and then work to reduce that overfitting—
can really can be seen as a recipe. 11 So this two-step process should be used when we
want to improve on the state we have achieved now.
10 At least one of the authors would love to write an entire book on the topics touched on in this section.
11 See also Andrej Karparthy’s blog post “A Recipe for Training Neural Networks” at https://karpathy.github
.io/2019/04/25/recipe for a more elaborate recipe.