Statistical Language Models based on Neural Networks - Faculty of ...

The notion ”probability of a sentence” is an entirely useless one, under any known
interpretation of this term. (Chomsky, 1969)
Still, we can consider entropy and perplexity as very useful measures. The simple
reason is that in the real-world applications (such as speech recognizers), there is a strong
positive correlation between perplexity of involved language model and the system’s per-
formance [24].
More theoretical reasons for using entropy as a measure of performance come from
an artificial intelligence point of view [42]. If we want to build an intelligent agent that
will maximize its reward in time, we have to maximize its ability to predict the outcome
of its own actions. Given the fact that such agent is supposed to work in the real world
and it can experience complex regularities including the natural language, we cannot hope
for a success unless this agent has an ability to find and exploit existing patterns in such
data. It is known that Turing machines (or equivalent) have the ability to represent any
algorithm (in other words, any pattern or regularity). However, algorithms that would
find all possible patterns in some data are not known. Contrary, it was proved that such
algorithms cannot exist in general, due to the halting problem (for some algorithms, the
output is not computationally decidable due to potential infinite recursion).
A very inspiring work on this topic was done by Solomonoff [70], who has shown an
optimal solution to the general prediction problem called Algorithmic probability. Despite
the fact that it is uncomputable, it provides very interesting insight into concepts such
as patterns, regularities, information, noise and randomness. Solomonoff’s solution is to
average over all possible (infinitely many) models of given data, while normalizing by their
description length. Algorithmic probability (ALP) of string x is defined as
PM(x) =
∞
2 −|Si(x)|
, (2.2)
i=0
where PM(x) denotes probability of string x with respect to machine M and |Si(x)| is
the description length of x (or any sequence that starts with x) given the i-th model of
x. Thus, the shortest descriptions dominate the final value of algorithmic probability of
the string x. More information about ALP, as well as proofs of its interesting properties
(for example invariance to the choice of the machine M, as long as M is universal) can be
found in [70].
12