Statistical Language Models based on Neural Networks - Faculty of ...

y(t) = g (Vs(t)) (3.5)
The output layer y represents a probability distribution of the next word wt+1 given
the history. The time complexity of one training or test step is proportional to
O = H × H + H × V = H × (H + V ) (3.6)
where H is size of the hidden layer and V is size of the vocabulary.
3.3 Learning Algorithm
Both the feedforward and the recurrent architecture of the neural network model can be
trained by stochastic gradient descent using a well-known backpropagation algorithm [65].
However, for better performance, a so-called Backpropagation through time algorithm can
be used to propagate gradients of errors in the network back in time through the recurrent
weights, so that the model is trained to capture useful information in the state of the
hidden layer. With simple BP training, the recurrent network performs poorly in some
cases, as will be shown later (some comparison was already presented in [50]). The BPTT
algorithm has been described in [65], and a good description for a practical implementation
is in [9].
With the stochastic gradient descent, the weight matrices of the network are updated
after presenting every example. A cross entropy criterion is used to obtain gradient of an
error vector in the output layer, which is then backpropagated to the hidden layer, and in
case of BPTT through the recurrent connections backwards in time. During the training,
validation data are used for early stopping and to control the learning rate. Training
iterates over all training data in several epochs before convergence is achieved - usually,
8-20 epochs are needed. As it will be shown in Chapter 6, the convergence speed of the
training can be improved by randomizing order of sentences in the training data, effectively
reducing the number of required training epochs (this was already observed in [5], and we
provide more details in [52]).
The learning rate is controlled as follows. Starting learning rate is α = 0.1. The
same learning rate is used as long as significant improvement on the validation data is
observed (in further experiments, we consider as a significant improvement more than
0.3% reduction of the entropy). After no significant improvement is observed, the learning
30