Statistical Language Models based on Neural Networks - Faculty of ...

where d(t) is a target vector that represents the word w(t + 1) that should have been
predicted (encoded again as 1-of-V vector). Note that it is important to use cross entropy
and not mean square error (MSE), which is a common mistake. The network would still
work, but the results would be suboptimal (at least, if our objective is to minimize entropy,
perplexity, word error rate or to maximize compression ratio). Weights V between the
hidden layer s(t) and the output layer y(t) are updated as
vjk(t+1) = vjk(t) + sj(t)eok(t)α (3.9)
where α is the learning rate, j iterates over the size of the hidden layer and k over the
size of the output layer, sj(t) is output of j-th neuron in the hidden layer and eok(t) is
error gradient of k-th neuron in the output layer. If L2 regularization is used, the equation
changes to
vjk(t+1) = vjk(t) + sj(t)eok(t)α − vjk(t)β (3.10)
where β is regularization parameter, in the following experiments its value is β = 10 −6 .
Regularization is used to keep weights close to zero 2 . Using matrix-vector notation, the
equation 3.10 would change to
V(t+1) = V(t) + s(t)eo(t) T α − V(t)β. (3.11)
Next, gradients of errors are propagated from the output layer to the hidden layer
eh(t) = dh eo(t) T V, t , (3.12)
where the error vector is obtained using function dh() that is applied element-wise
dhj(x, t) = xsj(t)(1 − sj(t)). (3.13)
Weights U between the input layer w(t) and the hidden layer s(t) are then updated as
uij(t+1) = uij(t) + wi(t)ehj(t)α − uij(t)β (3.14)
2 Quick explanation of using regularization is by using Occam’s razor: simper solutions should be
preferred, and small numbers can be stored more compactly than the large ones; thus, models with small
weights should generalize better.
32