Statistical Language Models based on Neural Networks - Faculty of ...

main (static) n-gram model. As the cache models provide truly significant improvements
in perplexity (sometimes even more than 20%), there exists a large number of more refined
techniques that can capture the same patterns as the basic cache models - for example,
various topic models, latent semantic analysis <strong>based</strong> models [3], trigger models [39] or
dynamically evaluated models [32] [49].
The advantage of cache (or similar) models is in large reduction of perplexity, thus
these techniques are very popular in the language modeling related papers. Also, their
implementation is often quite easy. The problematic part is that new cache-like techniques
are compared to weak baselines, like bigram or trigram models. It is unfair to not include
at least unigram cache model to the baseline, as it is very simple to do so (for example by
using standard LM toolkits such as SRILM [72]).
The main disadvantage is in questionable correlation between perplexity improvements
and word error rate reductions. This has been explained by [24] as a result of the fact
that the errors are locked in the system - if the speech recognizer decodes incorrectly a
word, it is placed in the cache which hurts further recognition by increasing chance of
doing the same error again. When the output from the recognizer is corrected by the user,
cache models are reported to work better; however, it is not practical to force users to
manually correct the output. Advanced versions, like trigger models or LSA models were
reported to provide interesting WER reductions, yet these models are not commonly used
in practice.
Another explanation of poor performance of cache models in speech recognition is
that since the output of a speech recognizer is imperfect, the perplexity calculations that
are normally performed on some held-out data (correct sentences) are misleading. If the
cache models were using the highly ambiguous history of previous words from a speech
recognizer, the perplexity improvements would be dramatically lower. It is thus important
to be careful when conclusions are made about techniques that access very long context
information.
2.3.2 Class Based <strong>Models</strong>
One way to fight the data sparsity in higher order n-grams is to introduce equivalence
classes. In the simplest case, each word is mapped to a single class, which usually repre-
sents several words. Next, n-gram model is trained on these classes. This allows better
generalization to novel patterns which were not seen in the training data. Improvements
19