Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Output
Parameter containing:
tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0.]]], requires_grad=True)
It is just a vector full of zeros. That’s it. But, since it is a parameter, its values will be
updated as the model gets trained. Then, let’s fetch a mini-batch of images and get
their patch embeddings:
images, labels = next(iter(train_loader))
images.shape # N, C, H, W
Output
torch.Size([16, 1, 12, 12])
embed = patch_embed(images)
embed.shape # N, L, D
Output
torch.Size([16, 9, 16])
There are 16 images, each represented by a sequence of nine patches with 16
dimensions each. The special embedding should be the same for all 16 images, so
we use tensor.expand() to replicate it along the batch dimension before
concatenation:
cls_tokens = cls_token.expand(embed.size(0), -1, -1)
embed_cls = torch.cat((cls_tokens, embed), dim=1)
embed_cls.shape # N, L+1, D
Output
torch.Size([16, 10, 16])
856 | Chapter 10: Transform and Roll Out