Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Higher-Order Learning Rate Function Builder
1 def make_lr_fn(start_lr, end_lr, num_iter, step_mode='exp'):
2 if step_mode == 'linear':
3 factor = (end_lr / start_lr - 1) / num_iter
4 def lr_fn(iteration):
5 return 1 + iteration * factor
6 else:
7 factor = (np.log(end_lr) - np.log(start_lr)) / num_iter
8 def lr_fn(iteration):
9 return np.exp(factor)**iteration
10 return lr_fn
Now, let’s try it out: Say we’d like to try ten different learning rates between 0.01
and 0.1, and the increments should be exponential:
start_lr = 0.01
end_lr = 0.1
num_iter = 10
lr_fn = make_lr_fn(start_lr, end_lr, num_iter, step_mode='exp')
There is a factor of 10 between the two rates. If we apply this function to a
sequence of iteration numbers, from 0 to 10, that’s what we get:
lr_fn(np.arange(num_iter + 1))
Output
array([ 1. , 1.25892541, 1.58489319, 1.99526231,
2.51188643, 3.16227766, 3.98107171, 5.01187234,
6.30957344, 7.94328235, 10. ])
If we multiply these values by the initial learning rate, we’ll get an array of learning
rates ranging from 0.01 to 0.1 as expected:
start_lr * lr_fn(np.arange(num_iter + 1))
Learning Rates | 447