Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Figure 10.26 - Two patch embeddings
The patch embeddings are obviously different for each image, but the embedding
corresponding to the special classifier token that’s prepended to the patch
embeddings is always the same.
Figure 10.27 - Two patch embeddings + [CLS] embedding
"How do we do that?"
It’s actually simple: We need to define a parameter in our model (using
nn.Parameter) to represent this special embedding and concatenate it at the
beginning of every sequence of embeddings. Let’s start by creating the parameter
itself (it will be an attribute of our model later):
cls_token = nn.Parameter(torch.zeros(1, 1, 16))
cls_token
Vision Transformer | 855