Statistical Language Models based on Neural Networks - Faculty of ...

w(t-2)
s(t-3)
U
W
w(t-1)
s(t-2)
U
W
w(t)
s(t-1)
Figure 3.2: Recurrent neural network unfolded as a deep feedforward network, here
for 3 time steps back in time.
are propagated from the hidden layer s(t) to the hidden layer from the previous time step
s(t−1) and the recurrent weight matrix (denoted as W in Figure 3.2) is updated. Error
propagation is done recursively as follows (note that the algorithm requires the states of
U
W
s(t)
the hidden layer from the previous time steps to be stored):
V
y(t)
eh(t−τ−1) = dh eh(t−τ) T W, t−τ−1 . (3.18)
The function dh is defined in equation 3.13. The unfolding can be applied for as many
time steps as many training examples were already seen, however the error gradients
quickly vanish as they get backpropagated in time [4] (in rare cases the errors can explode),
so several steps of unfolding are sufficient (this is sometimes referred to as truncated
BPTT). While for word <strong>based</strong> LMs, it seems to be sufficient to unfold network for about
5 time steps, it is interesting to notice that this still allows the network to learn to store
34