Statistical Language Models based on Neural Networks - Faculty of ...

ALP can be used to obtain prior probabilities of any sequential data, thus it provides
theoretical solution to the statistical language modeling. As mentioned before, ALP is
not computable (because of the halting problem), however it is mentioned here to justify
our later experiments with model combination. Different language modeling techniques
can be seen as individual components in eq. 2.2, where instead of using description length
of individual models for normalization, we use the performance of the model on some
validation data to obtain its weight 2 . More details about concepts such as ALP and
Minimum description length (MDL) will be given in Chapter 8.
Another work worth of mentioning was done by Mahoney [44], who has shown that the
problem of finding the best models of data is actually equal to the problem of general data
compression. Compression can be seen as two problems: data modeling, and coding. Since
coding is optimally solved by Arithmetic coding, data compression can be seen just as a
data modeling problem. Mahoney together with M. Hutter also organize a competition
with the aim to reach the best possible compression results on a given data set (mostly
containing wikipedia text), known as a Hutter prize competition. As the data compression
of text is almost equal to the language modeling task, I follow the same idea and try
to reach the best achievable results on a single well-known data set, the Penn Treebank
Corpus, where it is possible to compare (and combine) results of techniques developed by
several other researchers.
The important drawback of perplexity is that it obscures achieved improvements. Usu-
ally, improvements of perplexity are measured as percentual decrease over the baseline
value, which is a mistaken but widely accepted practice. In Table 2.1, it is shown that
constant perplexity improvement translates to different entropy reductions. For example,
it will be shown in Chapter 7 that advanced LM techniques provide similar relative reduc-
tions of entropy for word and character <strong>based</strong> models, while perplexity comparison would
completely fail in such case. Thus, perplexity results will be reported as a good measure
for quick comparison, but improvements will be mainly reported by using entropy.
2 It can be argued that since most of the models that are commonly used in language modeling are
not Turing-complete - such as finite state machines - using description length of these models would be
inappropriate.
13