Deep-Learning-with-PyTorch

What does an ideal dataset look like?
343
E2 val 0.0603 loss, 98.5% correct, 0.1271 precision, 0.8456 recall,
➥ 0.2209 f1 score
E2 val_ben 0.0584 loss, 98.6% correct (54181 of 54971)
E2 val_mal 0.8471 loss, 84.6% correct (115 of 136)
...
E5 trn 0.0578 loss, 98.3% correct, 0.9839 precision, 0.9823 recall,
➥ 0.9831 f1 score
E5 trn_ben 0.0665 loss, 98.4% correct (98388 of 100000)
E5 trn_mal 0.0490 loss, 98.2% correct (98227 of 100000)
E5 val 0.0361 loss, 99.2% correct, 0.2129 precision, 0.8235 recall,
➥ 0.3384 f1 score
E5 val_ben 0.0336 loss, 99.2% correct (54557 of 54971)
E5 val_mal 1.0515 loss, 82.4% correct (112 of 136)...
...
E10 trn 0.0212 loss, 99.5% correct, 0.9942 precision, 0.9953 recall,
➥ 0.9948 f1 score
E10 trn_ben 0.0281 loss, 99.4% correct (99421 of 100000)
E10 trn_mal 0.0142 loss, 99.5% correct (99530 of 100000)
E10 val 0.0457 loss, 99.3% correct, 0.2171 precision, 0.7647 recall,
➥ 0.3382 f1 score
E10 val_ben 0.0407 loss, 99.3% correct (54596 of 54971)
E10 val_mal 2.0594 loss, 76.5% correct (104 of 136)
...
E20 trn 0.0132 loss, 99.7% correct, 0.9964 precision, 0.9974 recall,
➥ 0.9969 f1 score
E20 trn_ben 0.0186 loss, 99.6% correct (99642 of 100000)
E20 trn_mal 0.0079 loss, 99.7% correct (99736 of 100000)
E20 val 0.0200 loss, 99.7% correct, 0.4780 precision, 0.7206 recall,
➥ 0.5748 f1 score
E20 val_ben 0.0133 loss, 99.8% correct (54864 of 54971)
E20 val_mal 2.7101 loss, 72.1% correct (98 of 136)
Ugh. That’s a lot of text to scroll past to get to the numbers we’re interested in. Let’s
power through and focus on the val_mal XX.X% correct numbers (or skip ahead to
the TensorBoard graph in the next section.) After epoch 2, we were at 87.5%; on
epoch 5, we peaked with 92.6%; and then by epoch 20 we dropped down to 86.8%—
below our second epoch!
NOTE As mentioned earlier, expect each run to have unique behavior due to
random initialization of network weights and random selection and ordering
of training samples per epoch.
The training set numbers don’t seem to be having the same problem. Negative training
samples are classified correctly 98.8% of the time, and positive samples are 99.1%
correct. What’s going on?
12.4.3 Recognizing the symptoms of overfitting
What we are seeing are clear signs of overfitting. Let’s take a look at the graph of our
loss on positive samples, in figure 12.18.