Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Before shuffling, the pixels were ordered; that is, they had an underlying twodimensional
structure. This structure can be exploited by the convolutional neural
networks: The kernel moving around the image looks at a pixel in the center and all
its neighbors in both dimensions, height and width.
If the underlying structure has a single dimension, though, that’s a sequence. This
particular structure can be exploited by recurrent neural networks and their many
variants, as well as by 1D convolutional neural networks, which constitute the
subject of this chapter.
There are two main types of sequence problems: time series and natural language
processing (NLP). We’ll start by generating a synthetic dataset and then use it to
illustrate the inner workings of recurrent neural networks, encoder-decoder
models, attention mechanisms, and even Transformers! Only then will we get to
the natural language processing part.
I chose to follow this sequence of topics because I find it much easier to develop
intuition (and to produce meaningful visualizations) while working with a twodimensional
dataset, as opposed to 100-dimensional word embeddings.
Data Generation
Our data points are two-dimensional, so they can be visualized as an image, and
ordered, so they are a sequence. We’ll be drawing squares! Each square, as
depicted in the figure below, has four corners (duh!), with each corner assigned a
letter and a color. The bottom-left corner is A and gray, the bottom-right one is D
and red, and so on.
Figure 8.1 - Our colored square
The coordinates (x 0 , x 1 ) of the four corners are our data points. The "perfect" square
has the coordinates depicted in the figure above: A=(-1, -1), B=(-1, 1), C=(1, 1),
and D=(1,-1). Sure enough, we’ll generate a dataset full of noisy squares, each
Sequences | 589