Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Visualizing the Context Vector
Let’s start by creating one dummy "query" and three dummy "keys":
q = torch.tensor([.55, .95]).view(1, 1, 2) # N, 1, H
k = torch.tensor([[.65, .2],
[.85, -.4],
[-.95, -.75]]).view(1, 3, 2) # N, L, H
Then, let’s visualize them as vectors, together with their norms and the
cosines of the angles between each "key" and the "query."
We can use the values in the figure above to compute the dot product
between each "key" and the "query":
Equation 9.7 - Dot products
# N, 1, H x N, H, L -> N, 1, L
prod = torch.bmm(q, k.permute(0, 2, 1))
prod
718 | Chapter 9 — Part I: Sequence-to-Sequence