Deep-Learning-with-PyTorch

Revisiting the problem of overfitting
345
tag: loss/neg
Both loSses
are trending
down.
Figure 12.19
Our negative loss showing no signs of overfitting
12.5 Revisiting the problem of overfitting
We touched on the concept of overfitting in chapter 5, and now it’s time to take a
closer look at how to address this common situation. Our goal with training a model is
to teach it to recognize the general properties of the classes we are interested in, as
expressed in our dataset. Those general properties are present in some or all samples
of the class and can be generalized and used to predict samples that haven’t been
trained on. When the model starts to learn specific properties of the training set, overfitting
occurs, and the model starts to lose the ability to generalize. In case that’s a bit
too abstract, let’s use another analogy.
12.5.1 An overfit face-to-age prediction model
Let’s pretend we have a model that takes an image of a human face as input and outputs
a predicted age in years. A good model would pick up on age signifiers like wrinkles,
gray hair, hairstyle, clothing choices, and similar, and use those to build a general
model of what different ages look like. When presented with a new picture, it would
consider things like “conservative haircut” and “reading glasses” and “wrinkles” to
conclude “around 65 years old.”
An overfit model, by contrast, instead remembers specific people by remembering
identifying details. “That haircut and those glasses mean it’s Frank. He’s 62.8 years
old”; “Oh, that scar means it’s Harry. He’s 39.3”; and so on. When shown a new person,
the model won’t recognize the person and will have absolutely no idea what age
to predict.
Even worse, if shown a picture of Frank Jr. (the spittin’ image of his dad, at least
when he’s wearing his glasses!), the model will say, “I think that’s Frank. He’s 62.8
years old.” Never mind that Junior is 25 years younger!