Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Output
tensor([[109, 124],
[124, 124],
[124, 124]])
It is almost the same result as before. The model is probably overfitting, but it
doesn’t matter because the purpose of this exercise was to show you how to use
transfer learning and how you can pre-process your dataset to speed up model
training.
AlexNet was fun to work with, but it is time to move on. In the next sections, we’ll
focus on new architectural elements that are part of the Inception and ResNet
models.
Auxiliary Classifiers (Side-Heads)
The first version of the Inception model (depicted in the figure below) introduced
auxiliary classifiers; that is, side-heads attached to intermediate parts of the
model that would also try to perform classification, independently from the typical
main classifier at the very end of the network.
Figure 7.3 - Inception model: simplified diagram
The cross-entropy loss was also computed independently for each of the three
classifiers and added together to the total loss (although auxiliary losses were
multiplied by a factor of 0.3). The auxiliary classifiers (and losses) were used during
training time only. During the evaluation phase, only the logits produced by the
main classifier were considered.
The technique was originally developed to mitigate the vanishing gradients
problem (more on that in the next chapter), but it was later found that the auxiliary
classifiers were more likely to have a regularizer effect instead. [125]
Auxiliary Classifiers (Side-Heads) | 521