Statistical Language Models based on Neural Networks - Faculty of ...

Chapter 3
Neural Network <strong>Language</strong> <strong>Models</strong>
The use of artificial neural networks for sequence prediction is as old as the neural network
techniques themselves. One of the first widely known attempts to describe language using
neural networks was performed by Jeff Elman [17], who used recurrent neural network
for modeling sentences of words generated by an artificial grammar. The first serious at-
tempt to build a statistical neural network <strong>based</strong> language model of real natural language,
together with an empirical comparison of performance to standard techniques (n-gram
models and class <strong>based</strong> models) was probably done by Yoshua Bengio in [5]. Bengio’s
work was followed by Holger Schwenk, who did show that NNLMs work very well in a
state of the art speech recognition systems, and are complementary to standard n-gram
models [68].
However, despite many scientific papers were published after the original Bengio’s
work, no techniques or modifications of the original model that would significantly improve
ability of the model to capture patterns in the language were published, at least to my
knowledge 1 . Integration of additional features into the NNLM framework (such as part
of speech tags or morphology information) has been investigated in [19] [1]. Still, the
accuracy of the neural net models remained basically the same, until I have recently shown
that recurrent neural network architecture can work actually better than the feedforward
one [49] [50].
Most of the research work did focus on overcoming practical problems when using
these attractive models: the computational complexity was originally too high for real
world tasks. It was reported by Bengio in 2001 that training of the original neural net
1 With the exception of Schwenk, who reported better results by using linear interpolation of several
neural net models trained on the same data, with different random initialization of the weights - we denote
this approach further as a combination of NNLMs.
26