Deep-Learning-with-PyTorch

132 CHAPTER 5 The mechanics of learning
behaves well away from or in between the data points. After all, that’s what we’re asking
the optimizer to do: minimize the loss at the data points. Sure enough, if we had independent
data points that we didn’t use to evaluate our loss or descend along its negative
gradient, we would soon find out that evaluating the loss at those independent
data points would yield higher-than-expected loss. We have already mentioned this
phenomenon, called overfitting.
The first action we can take to combat overfitting is recognizing that it might happen.
In order to do so, as Kepler figured out in 1600, we must take a few data points
out of our dataset (the validation set) and only fit our model on the remaining data
points (the training set), as shown in figure 5.12. Then, while we’re fitting the model,
we can evaluate the loss once on the training set and once on the validation set. When
we’re trying to decide if we’ve done a good job of fitting our model to the data, we
must look at both!
dataproducing
proceSs
training
set
validation
set
trained
model
performance
parameter
optimization
(training)
model
Figure 5.12 Conceptual representation of a dataproducing
process and the collection and use of
training data and independent validation data
EVALUATING THE TRAINING LOSS
The training loss will tell us if our model can fit the training set at all—in other words,
if our model has enough capacity to process the relevant information in the data. If
our mysterious thermometer somehow managed to measure temperatures using a logarithmic
scale, our poor linear model would not have had a chance to fit those measurements
and provide us with a sensible conversion to Celsius. In that case, our
training loss (the loss we were printing in the training loop) would stop decreasing
well before approaching zero.