Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Output
(0.016193246061448008, 0.034184777294285595)
Recap
In this chapter, we’ve introduced sequence-to-sequence problems and the
encoder-decoder architecture. At first, we used recurrent neural networks to
encode a source sequence so that its representation (hidden state) could be used
to generate the target sequence. Then, we improved the architecture by using a
(cross-)attention mechanism that allowed the decoder to use the full sequence of
hidden states produced by the encoder. Next, we replaced the recurrent neural
networks with self-attention mechanisms, which, although more efficient, cause
the loss of information regarding the order of the inputs. Finally, the addition of
positional encoding allowed us to account for the order of the inputs once again.
This is what we’ve covered:
• generating a synthetic dataset of source and target sequences
• understanding the purpose of the encoder-decoder architecture
• using the encoder to generate a representation of the source sequence
• using encoder’s final hidden state as the decoder’s initial hidden state
• using the decoder to generate the target sequence
• using teacher forcing to help the decoder during training
• combining both encoder and decoder into a single encoder-decoder model
• understanding the limitations of using a single hidden state to encode the
source sequence
• defining the sequence of (transformed) hidden states from the encoder as
"values" (V)
• defining the sequence of (transformed) hidden states from the encoder as
"keys" (K)
• defining (transformed) hidden states produced by the decoder as "queries" (Q)
• computing similarities (alignment scores) between a given "query" and all the
"keys" using scaled dot-product
• visualizing the geometric interpretation of the dot-product
Recap | 793