Deep-Learning-with-PyTorch

136 CHAPTER 5 The mechanics of learning
Epoch 1000, Training loss 3.5940, Validation loss 5.3110
Epoch 1500, Training loss 3.0942, Validation loss 4.1611
Epoch 2000, Training loss 3.0238, Validation loss 3.7693
Epoch 2500, Training loss 3.0139, Validation loss 3.6279
Epoch 3000, Training loss 3.0125, Validation loss 3.5756
tensor([
5.1964, -16.7512], requires_grad=True)
Here we are not being entirely fair to our model. The validation set is really small, so
the validation loss will only be meaningful up to a point. In any case, we note that the
validation loss is higher than our training loss, although not by an order of magnitude.
We expect a model to perform better on the training set, since the model
parameters are being shaped by the training set. Our main goal is to also see both the
training loss and the validation loss decreasing. While ideally both losses would be
roughly the same value, as long as the validation loss stays reasonably close to the
training loss, we know that our model is continuing to learn generalized things about
our data. In figure 5.14, case C is ideal, while D is acceptable. In case A, the model
isn’t learning at all; and in case B, we see overfitting. We’ll see more meaningful examples
of overfitting in chapter 12.
loSs
loSs
A
B
iterations
iterations
loSs
loSs
C
D
iterations
iterations
Figure 5.14 Overfitting scenarios when looking at the training (solid line) and validation (dotted line)
losses. (A) Training and validation losses do not decrease; the model is not learning due to no
information in the data or insufficient capacity of the model. (B) Training loss decreases while
validation loss increases: overfitting. (C) Training and validation losses decrease exactly in tandem.
Performance may be improved further as the model is not at the limit of overfitting. (D) Training and
validation losses have different absolute values but similar trends: overfitting is under control.