Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Perfect! Now let’s build the actual range test. This is what we’re going to do:
• Since we’ll be updating both model and optimizer, we need to store their initial
states so they can be restored in the end.
• Create both custom function and corresponding scheduler, just like in the
snippets above.
• (Re)implement a training loop over mini-batches, so we can log the learning
rate and loss at every step.
• Restore model and optimizer states.
Moreover, since we’re using a single mini-batch to evaluate the loss, the resulting
values will likely jump up and down a lot. So, it is better to smooth the curve using
an exponentially weighted moving average (EWMA) (we’ll talk about EWMAs in
much more detail in the next section) to more easily identify the trend in the values.
This is what the method looks like:
StepByStep Method
def lr_range_test(self, data_loader, end_lr, num_iter=100,
step_mode='exp', alpha=0.05, ax=None):
# The test updates both model and optimizer, so we need to
# store their initial states to restore them in the end
previous_states = {
'model': deepcopy(self.model.state_dict()),
'optimizer': deepcopy(self.optimizer.state_dict())
}
# Retrieves the learning rate set in the optimizer
start_lr = self.optimizer.state_dict()['param_groups'][0]['lr']
# Builds a custom function and corresponding scheduler
lr_fn = make_lr_fn(start_lr, end_lr, num_iter)
scheduler = LambdaLR(self.optimizer, lr_lambda=lr_fn)
# Variables for tracking results and iterations
tracking = {'loss': [], 'lr': []}
iteration = 0
# If there are more iterations than mini-batches in the data
# loader, it will have to loop over it more than once
while (iteration < num_iter):
Learning Rates | 449