Statistical Language Models based on Neural Networks - Faculty of ...

n-gram probabilities. However, it was shown in [40] that this simple technique degrades
performance for small values of S very significantly, and even with small S such as 2000,
the complexity induced by the H × V term is still very large.
More successful approaches are <strong>based</strong> on Goodman’s trick for speeding up maximum
entropy models using classes [25]. Each word from the vocabulary is assigned to a single
class, and only the probability distribution over the classes is computed first. In the
second step, the probability distribution over words that are members of a particular class
is computed (we know this class from the predicted word whose probability we are trying
to estimate). As the number of classes can be very small (several hundreds), this is a
much more effective approach than using shortlists, and the performance degradation is
smaller. We have shown that meaningful classes can be formed very easily, by considering
only unigram frequencies of words [50]. Similar approaches have been described in [40]
and [57].
6.2.4 Reduction of Size of the Hidden Layer
Another way to reduce H ×V is to choose a small value of H. For example, in [8], H = 100
is used when the amount of the training data is over 600M words. However, we will show
that the small size of the hidden layer is insufficient to obtain good performance when the
amount of training data is large, as long as the usual neural net LM architecture is used.
In Section 6.6, a novel architecture of neural net LM is described, denoted as RNNME
(recurrent neural network trained jointly with maximum entropy model). It allows small
hidden layers to be used for models that are trained on huge amounts of data, with
very good performance (much better than what can be achieved with the traditional
architecture).
6.2.5 Parallelization
Computation in artificial neural network models can be parallelized quite easily. It is
possible to either divide the matrix times vector computation between several CPUs, or to
process several examples at once, which allows going to matrix times matrix computation
that can be optimized by existing libraries such as BLAS. In the context of NN LMs,
Schwenk has reported a speedup of several times by exploiting parallelization [68].
It might seem that recurrent networks are much harder to parallelize, as the state of
the hidden layer depends on the previous state. However, one can parallelize just the
75