Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Very High Learning Rate
Wait, it may get worse than that! Let’s use a really high learning rate, say, a step
size of 1.1!
Figure 0.12 - Using a really high learning rate
"He chose … poorly."
Grail Knight
Ok, that is bad. On the right plot, not only did we end up on the other side of the
curve again, but we actually climbed up. This means our loss increased, instead of
decreased! How is that even possible? You’re moving so fast downhill that you end up
climbing it back up?! Unfortunately, the analogy cannot help us anymore. We need
to think about this particular case in a different way.
First, notice that everything is fine on the left plot. The enormous learning rate did
not cause any issues, because the left curve is less steep than the one on the right.
In other words, the curve on the left can take higher learning rates than the curve
on the right.
What can we learn from it?
Step 4 - Update the Parameters | 47