Statistical Language Models based on Neural Networks - Faculty of ...

output layer changes to computation between the hidden and the class layer:
and the hidden layer and a subset of the output layer:
The probability of word w(t + 1) is then computed as
⎛
cm(t) = g ⎝
⎞
sj(t)xmj⎠
, (3.23)
yV
j
⎛
′(t) = g ⎝
sj(t)vV ′ ⎞
⎠ j . (3.24)
j
P (wt+1|s(t)) = P (ci|s(t))P (wi|ci, s(t)), (3.25)
where wi is an index of the predicted word and ci is its class. During training, the weights
are accessed in the same way as during the forward pass, thus the gradient of the error
vector is computed for the word part and for the class part, and then is backpropagated
back to the hidden layer, where gradients are added together. Thus, the hidden layer is
trained to predict both the distribution over the words and over the classes.
An alternative to simple frequency binning is a slightly modified approach, that min-
imizes access to words and classes: instead of using frequencies of words for the equal
binning algorithm, one can apply square root function on the original frequencies, and
perform the binning on these modified frequencies. This approach leads to even larger
speed-up 3 .
Factorization of the computation between the hidden and output layers using simple
classes can easily lead to 15 - 30 times speed-up against a fair baseline, and for the net-
works with huge output layers (more then 100K words), the speedup may be even an order
of magnitude larger. Thus, this speedup trick is essential for achieving reasonable perfor-
mance on larger data sets. Additional techniques for reducing computational complexity
will be discussed in more detail in Chapter 6.
3 Thanks to Dan Povey who suggested this modification.
39