Statistical Language Models based on Neural Networks - Faculty of ...

ecognition). Maybe even more importantly, several researchers have already pointed out
that building large look-up tables from huge amounts of training data (which is equal to
standard n-gram modeling) is not going to provide the ultimate answer to the language
modeling problem, as because of curse of dimensionality, we will never have that much
data [5].
The other way around, building language models from huge amounts of data (hundreds
of billion words or more) is also a very challenging task, and has received recently a lot
of attention [26]. The problems that arise include smoothing, as well as compression
techniques, because it is practically impossible to store the full n-gram models estimated
from such amount of data in computer memory. While amount of text that is available
on the Internet is ever-increasing and computers are getting faster and memory bigger, we
cannot hope to build a database of all possible sentences that can ever be said.
In this thesis, recurrent neural network language model (RNN LM) which I have re-
cently proposed in [49, 50] is described, and compared to other successful language mod-
eling techniques. Several standard text corpora are used, which allows to provide detailed
and fair comparison to other advanced language modeling techniques. The aim is at
obtaining the best achievable results by combining all studied models, which leads to a
new state of the art performance on the standard setup involving part of Penn Treebank
Corpus.
Next, it is shown that the RNN <strong>based</strong> language model can be applied to large scale
well-tuned system, and that it provides significant improvements in speech recognition
accuracy. The baseline system for these experiments from IBM (RT04 Broadcast News
speech recognition) has been recently used in the 2010 Summer Workshop at Johns Hop-
kins University [82]. This system was also used as a baseline for a number of papers
concerning novel type of maximum entropy language model, a so-called model M [30] lan-
guage model, which is also used in the performance comparison as it was previously the
state-of-the-art language model on the given task.
Finally, I try to answer some fundamental questions of language modeling. Namely,
whether the progress in the field is illusory, as is sometimes suggested. And ultimately,
why the new techniques did not reach human performance yet, and what might be the
missing parts and the most promising areas for the future research.
10