Statistical Language Models based on Neural Networks - Faculty of ...

are usually achieved by combining class <strong>based</strong> model and the n-gram model. There exists a
lot of variations of class <strong>based</strong> models, which often focus on the process of forming classes.
So-called soft classes allow one word to belong to multiple classes. Description of several
variants of class <strong>based</strong> models can be found in [24].
While perplexity improvements given by class <strong>based</strong> models are usually moderate, these
techniques have noticeable effect on the word error rate in speech recognition, especially
when only small amount of training data is available. This makes class <strong>based</strong> models quite
attractive as opposed to the cache models, which usually work well only in experiments
concerning perplexity.
The disadvantages of class <strong>based</strong> models include high computational complexity during
inference (for statistical classes) or reliance on expert knowledge (for manually assigned
classes). More seriously, improvements tend to vanish with increased amount of the train-
ing data [24]. Thus, class <strong>based</strong> models are more often found in the research papers, than
in real applications.
From the critical point of view, there are several theoretical difficulties involving class
<strong>based</strong> models:
• The assumption that words belong to some higher level classes is intuitive, but
usually no special theoretical explanation is given to the process how classes are
constructed; in the end, the number of classes is usually just some tunable parameter
that is chosen <strong>based</strong> on performance on development data
• Most techniques do attempt to cluster individual words in the vocabulary, but the
idea is not extended to n-grams: by thinking about character-level models, it is obvi-
ous that with increasing amount of the training data, classes can only be successful
if longer context can be captured by a single class (several characters for this case)
2.3.3 Structured <strong>Language</strong> <strong>Models</strong>
The statistical language modeling was criticized heavily by the linguists from the first
days of its existence. The already mentioned Chomsky’s statement that ”the notion of
probability of a sentence is completely useless one” can be nowadays easily seen as a big
mistake due to indisputable success of applications that involve n-gram models. However,
further objections from the linguistic community usually address the inability of n-gram
models to represent longer term patterns that clearly exist between words in a sentence.
20