Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Finding LR
As it turns out, you don’t have to grid search the learning rate like that. In 2017,
Leslie N. Smith published "Cyclical Learning Rates for Training Neural Networks"
[101]
in which he outlines a procedure to quickly find an appropriate range for the
initial learning rate (more on the cyclical part of his paper later!). This technique is
called LR Range Test, and it is quite a simple solution to help you get a first
estimate for the appropriate learning rate.
The general idea is pretty much the same as the grid search: It tries multiple
learning rates and logs the corresponding losses. But here comes the difference: It
evaluates the loss over a single mini-batch, and then changes the learning rate
before moving on to the next mini-batch.
This is computationally cheap (it is performing ONE training step only for each
candidate) and can be performed inside the same training loop.
"Wait a minute! Wouldn’t the results be affected by the previous
training steps performed using different learning rates?"
Well, technically, yes. But this is not such a big deal: First, we’re looking for a
ballpark estimate of the learning rate, not a precise value; second, these updates
will barely nudge the model from its initial state. It is easier to live with this
difference than to reset the model every single time.
First, we need to define the boundaries for the test (start_lr and end_lr) and the
number of iterations (num_iter) to move from one to the other. On top of that, we
can choose to change how to make the increments: linearly or exponentially. Let’s
build a higher-order function that takes all those arguments and returns another
function, one that returns the multiplying factor given the current iteration
number:
446 | Chapter 6: Rock, Paper, Scissors