Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Recap
This finishes our journey through the inner workings of gradient descent. By now, I
hope you have developed better intuition about the many different aspects
involved in the process.
In time, with practice, you’ll observe the behaviors described here in your own
models. Make sure to try plenty of different combinations: mini-batch sizes,
learning rates, etc. This way, not only will your models learn, but so will you :-)
This is a (not so) short recap of everything we covered in this chapter:
• defining a simple linear regression model
• generating synthetic data for it
• performing a train-validation split on our dataset
• randomly initializing the parameters of our model
• performing a forward pass; that is, making predictions using our model
• computing the errors associated with our predictions
• aggregating the errors into a loss (mean squared error)
• learning that the number of points used to compute the loss defines the kind of
gradient descent we’re using: batch (all), mini-batch, or stochastic (one)
• visualizing an example of a loss surface and using its cross-sections to get the
loss curves for individual parameters
• learning that a gradient is a partial derivative and it represents how much the
loss changes if one parameter changes a little bit
• computing the gradients for our model’s parameters using equations, code,
and geometry
• learning that larger gradients correspond to steeper loss curves
• learning that backpropagation is nothing more than "chained" gradient
descent
• using the gradients and a learning rate to update the parameters
• comparing the effects on the loss of using low, high, and very high learning
rates
• learning that loss curves for all parameters should be, ideally, similarly steep
58 | Chapter 0: Visualizing Gradient Descent