The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 3. HIDDEN MARKOV MODELS 48complexity and data fit. Second, since the finite-state models they investigate act as encoder/decoders oftext they are deterministic, i.e., the current state and the next input symbol determine a unique next state (itfollows that each string has a unique derivation). This constrains the model space and allows states to beidentified with string suffixes, which is the basis of all their algorithms. Finally, the models have no end statessince they are supposed to encode continuous text. This is actually a minor difference since we can viewthe end-of-sentence as a special symbol, so that the final state is simply one that is dedicated to emitting thatspecial symbol.Bell et al. (1990) suggest state splittingas a more efficient induction technique for adaptively findinga finite-state model structure. In this approach, states are successively duplicated and differentiated accordingto their preceding context, whenever such a move promises to help the prediction of the following symbol.Ron et al. (1994) give a reformulation and formal analysis of this idea in terms of an information-theoreticevaluation function.Interestingly, Bell et al. (1990) show that such a state splitting strategy confines the power of thefinite-state model to that of a finite-context model. In models of this type there is always a finite bound",such that the last"preceding symbols uniquely determine the distributionof the next symbol. In other words,state-based models derived by this kind of splittingare ¢ essentially -gram models with variable (but bounded)context. This restriction applies equally to the algorithm of Ron et al. (1994).By contrast, consider the HMM depicted in Figure 3.2, which is used below as a benchmark model.It describes a language in which the context needed for correct prediction of the final symbol is unbounded.Such a model can be found without difficulty by simple best-first merging. The major advantage of splittingapproach is that it is guaranteed to find the appropriate model if enough data is presented and if the targetlanguage is in fact finite-context.3.5.4 Other probabilistic approachesAnother probabilistic approach to HMM structure induction similar to ours is described by Thomason& Granum (1986). The basic idea is to incrementally build a model structure by incorporating newsamples using an extended form of Viterbi alignment. New samples are aligned to the existing model so asto maximize their likelihood, while allowing states to be inserted or deleted for alignment purposes. Theprocedure is limited to HMMs that have a left-to-right ordering of states, however; in particular, no loops areallowed. In a sense this approach can be seen as an approximation to Bayesian HMM merging for this specialclass of models. The approximation in this case is twofold: the likelihood (not the posterior) is maximized,and only the likelihood of a single sample (rather than the entire data set) is considered.Haussler et al. (1992) apply HMMs trained by the Baum-Welch method to the problem of proteinprimary structure alignment. Their model structures are mostly of a fixed, linear form, but subject to limitedmodification by a heuristic that inserts states (‘stretches’ the model) or deletes states (‘shrinks’ the model)based on the estimated probabilities.Somewhat surprisingly, the work by Brown et al. (1992) on the construction of ¢ class-based -gram