Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

RNN Cell
Let’s take a look at some of the internals of an RNN cell:
Figure 8.6 - Internals of an RNN cell
On the left, we have a single RNN cell. It has three main components:
• A linear layer to transform the hidden state (in blue)
• A linear layer to transform the data point from the sequence (in red)
• An activation function, usually the hyperbolic tangent (TanH), which is applied
to the sum of both transformed inputs
We can also represent them as equations:
Equation 8.1 - RNN
I chose to split the equation into smaller colored parts to highlight the fact that
these are simple linear layers producing both a transformed hidden state (t h ) and a
transformed data point (t x ). The updated hidden (h t ) state is both the output of this
particular cell and one of the inputs of the "next" cell.
But there is no other cell, really; it is just the same cell over and over again, as
depicted on the right side of the figure above. So, in the second step of the
sequence, the updated hidden state will run through the very same linear layer the
initial hidden state ran through. The same goes for the second data point.
594 | Chapter 8: Sequences