Statistical Language Models based on Neural Networks - Faculty of ...

words, but also additional symbols such as brackets and quotation marks. These follow
simple patterns that cannot be described efficiently with n-gram models (such as that after
a left bracket, a right bracket should be expected etc.). Even with a limited RNNME-
160 model, the entropy reduction over KN5 model with no count cutoffs was about 7% -
perplexity was reduced from 110 to 80. By combining static and dynamic RNNME-160
model, perplexity decreased further to 58. It would be not surprising if a combination
of several larger static and dynamic RNNME models trained on the MT corpora would
provide perplexity reduction over 50% against the KN5 model with no count cutoffs - it
would be interesting to investigate this task in the future.
7.2 Data Compression
Compression of text files is a problem almost equivalent to statistical language modeling.
The objective of state of the art text compressors is to predict the next character or word
with as high probability as possible, and encode it using algorithmic coding. Decompres-
sion is then a symmetric process, where models are built while processing the decoded
data in the same way as they were built during the compression. Thus, the objective is
to simply maximize P (w|h). An example of such compression approach is the state of the
art compression program ’PAQ’ from M. Mahoney [45, 46].
The difference between language modeling and data compression is that the data is not
known in advance, and has to be processed on-line. As it was described in the previous
chapters, training of neural network models with a hidden layer requires several training
passes over the training data for reaching the top performance. Thus, it was not clear if
RNN LMs can be practically applied to data compression problems.
As data compression is not topic of this thesis, I will not describe experiments in great
detail. However, it is worth observing that it is possible to accurately estimate compression
ratio given by RNN LMs using the RNNLM toolkit. The size of a compressed text file is
size of (possibly also compressed) vocabulary plus entropy of the training data during the
first training epoch (this is simply average entropy per word times number of words).
For the further experiments, I have implemented arithmetic coding into a special ver-
sion of the RNNLM toolkit, thus the following results were obtained with real RNN-<strong>based</strong>
data compressor. As the class <strong>based</strong> RNN architecture was used, first the class of predicted
word is encoded using arithmetic coding, and then the specific word, as the text data are
96