Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Figure 5.15 - LeNet-5 architecture
Source: Generated using Alexander Lenail’s NN-SVG and adapted by the author. For
more details, see LeCun, Y., et al. (1998). "Gradient-based learning applied to
document recognition," Proceedings of the IEEE, 86(11), 2278–2324. [92]
Do you see anything familiar? The typical convolutional blocks are already there
(to some extent): convolutions (C layers), activation functions (not shown), and
subsampling (S layers). There are some differences, though:
• Back then, the subsampling was more complex than today’s max pooling, but
the general idea still holds.
• The activation function, a sigmoid at the time, was applied after the
subsampling instead of before, as is typical today.
• The F6 and Output layers were connected by something called Gaussian
connections, which is more complex than the typical activation function one
would use today.
Typical Architecture | 369