Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Choosing a learning rate that works well for a given model (and dataset) is a
difficult task, one mostly done by trial-and-error since there is no analytical way of
finding the optimal learning rate. One thing we can say for sure is that it should be
less than 1.0, and it is likely higher than 1e-6.
"Well, that doesn’t help much…"
Indeed, it doesn’t. So, let’s discuss how we can make it a bit more specific.
In previous chapters, we used 0.1 as the learning rate, which is kind of high but
worked well for really simple problems. As models grow more complex, though,
that value is definitely too high, and one order of magnitude lower (0.01) is a better
starting point.
"What if it is still too high and the loss doesn’t go down?"
That’s a real possibility, and one possible way of handling this is to perform a grid
search, trying multiple learning rates over a few epochs each and comparing the
evolution of the losses. This is expensive, computationally speaking, since you need
to train the model multiple times, but it may still be feasible if your model is not too
large.
"How do I choose values for the grid search?"
It is common to reduce the learning rate by a factor of 3 or a factor of 10. So, your
learning rate values could very well be [0.1, 0.03, 0.01, 3e-3, 1e-3, 3e-4, 1e-4] (using
a factor of 3) or [0.1, 0.01, 1e-3, 1e-4, 1e-5] (using a factor of 10). In general, if you
plot the learning rates against their corresponding losses, this is what you can
expect:
• If the learning rate is too low, the model doesn’t learn much, and the loss
remains high.
• If the learning rate is too high, the model doesn’t converge to a solution, and
the loss gets higher.
• In between those two extremes, the loss should be lower, hinting at the right
order of magnitude for the learning rate.
Learning Rates | 445