Statistical Language Models based on Neural Networks - Faculty of ...

Chapter 6
Strategies for Training Large Scale
Neural Network <strong>Language</strong> <strong>Models</strong>
The experiments on the Penn Treebank Corpus have shown that mixtures of recurrent
neural networks trained by backpropagation through time provide state of the art results
in the field of statistical language modeling. However a remaining question is, if the
performance would be also this good with much larger amount of the training data -
the PTB corpus with about 1M training tokens can be considered as very small, because
language models are typically trained on corpora with orders of magnitude more data.
It is not unusual to work with huge training corpora that consist of much more than a
billion words. While application to low resource domains (especially for new languages
and for domains where only small amount of relevant data exists) is also a very interesting
research problem, the most convincing results are those obtained with well tuned state of
the art systems, which are trained on large amounts of data.
The experiments in the previous chapter focused on obtaining the largest possible
improvement, however some of the approaches would become computationally difficult
to apply to large data sets. In this chapter, we briefly mention existing approaches for
reducing the computational complexity of neural net language models (most of these ap-
proaches are also applicable to maximum entropy language models). We propose two new
simple techniques that can be used to reduce computational complexity of the training
and the test phases. We show that these new techniques are complementary to existing
approaches.
Most interestingly, we show that a standard neural network language model can be
70