The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 2. FOUNDATIONS 9The number of parameters in ¢ -gram models grows exponentially with ¢ , and only the cases¢56 2 (bigram models) and ¢56 3 (trigram models) are of practical importance. Bigram and trigrammodels are popular for various applications, especially speech decoding (Ney 1984), to approximate the truedistributions of language elements (characters, words, etc.), which are known to violate the independenceassumption embodied in (2.1).Because (2.1) is essentially a truncated version of the true joint probability given by (2.2), ¢ -gramsare in some sense a natural choice for this approximation, and are appropriate to the extent that symboloccurrences tend be more and more independent as the distance between the occurrences increases. Of coursethere are important cases in natural language and elsewhere where this assumption is blatantly wrong. Forexample, the distribution of lexical elements in natural languages are constrained by phrase structures thatcan relate two (or more) words over essentially arbitrary distances. This is the main motivation for moving,at a minimum, to the stochastic context-free models that are one of the main subjects of this thesis.2.2.3 Probabilistic grammars as random string generatorsOne can always abstractly associate a probabilistic language ¨ with a corresponding random stringgenerator, i.e., a device that generates strings stochastically according to the distribution ¨ . However, aprobabilistic grammar usually describes ¨ by making this generator concrete. For example, for ¢ -gramsa generator would output string symbols left-to-right, always choosing the next symbol 1M@ according to aprobability lookup table indexed by the ¢"L 10 -tuple of previous symbols 1