Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Figure 9.46 - Consistent distances
Awesome, isn’t it? The encoded distance between any two positions T steps apart
is constant now. In our encoding, the distance between any two positions one step
apart will always be 2.03.
"Great! But how do I choose the 'bases' for the encoding?"
It turns out, you don’t have to. As the first vector, simply move along the circle as
many radians as the index of the position (one radian is approximately 57.3
degrees). Then, for each new vector added to the encoding, move along the circle
with exponentially slower angular speeds. For example, in the second vector, we
would move only one-tenth of a radian (approximately 5.73 degrees) for each new
position. In the third vector, we would move only one-hundredth of a radian, and
so on and so forth. Figure 9.47 depicts the red arrow moving at increasingly slower
angular speeds.
Positional Encoding (PE) | 769