Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Step 2 - Compute the Loss
There is a subtle but fundamental difference between error and loss.
The error is the difference between the actual value (label) and the predicted
value computed for a single data point. So, for a given i-th point (from our dataset
of N points), its error is:
Equation 0.2 - Error
The error of the first point in our dataset (i = 0) can be represented like this:
Figure 0.3 - Prediction error (for one data point)
The loss, on the other hand, is some sort of aggregation of errors for a set of data
points.
It seems rather obvious to compute the loss for all (N) data points, right? Well, yes
and no. Although it will surely yield a more stable path from the initial random
parameters to the parameters that minimize the loss, it will also surely be slow.
This means one needs to sacrifice (a bit of) stability for the sake of speed. This is easily
accomplished by randomly choosing (without replacement) a subset of n out of N
data points each time we compute the loss.
30 | Chapter 0: Visualizing Gradient Descent