Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

each parameter using the chain rule, let me show you how Wikipedia describes
backpropagation (highlights are mine):
The backpropagation algorithm works by computing the gradient of the loss
function with respect to each weight by the chain rule, computing the
gradient one layer at a time, iterating backward from the last layer to avoid
redundant calculations of intermediate terms in the chain rule;
…
The term backpropagation strictly refers only to the algorithm for computing
the gradient, not how the gradient is used; but the term is often used loosely
to refer to the entire learning algorithm, including how the gradient is used,
such as by stochastic gradient descent.
Does it seem familiar? That’s it; backpropagation is nothing more than "chained"
gradient descent. That’s, in a nutshell, how a neural network is trained: It uses
backpropagation, starting at its last layer and working its way back, to update the
weights through all the layers.
In our example, we have a single layer, even a single neuron, so there is no need to
backpropagate anything (more on that in the next chapter).
Step 4 - Update the Parameters
In the final step, we use the gradients to update the parameters. Since we are
trying to minimize our losses, we reverse the sign of the gradient for the update.
There is still another (hyper-)parameter to consider: the learning rate, denoted by
the Greek letter eta (that looks like the letter n), which is the multiplicative factor
that we need to apply to the gradient for the parameter update.
Equation 0.5 - Updating coefficients b and w using computed gradients and a learning rate
We can also interpret this a bit differently: Each parameter is going to have its
Step 4 - Update the Parameters | 41