The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 3. HIDDEN MARKOV MODELS 34Returning to the example, we now chose to merge states 2 and 6 (4 3). This step decreases the loglikelihood (from L 0£ 602 to L 0£ 829) but it is the smallest decrease that can be achieved by any of the potentialmerges.Following that, states 1 and 5 can be merged without (4 penalty 4). The resulting HMM is theminimal model generating the target language 0# , but what prevents us from merging further, to obtain anHMM for , ? #It turns out that merging the remaining two states reduces the likelihood much more drastically thanthe previous, ‘good’ generalization step, from L 0£ 829 to L 3£ 465 (i.e., three decimal orders of magnitude). Apreliminary answer, therefore, is to set the threshold small enough to allow only desirable generalizations. Amore satisfactory answer is provided by the Bayesian methods described below.Note that further data may well justify the generalization to a model for # . This data-drivencharacter is one of the central aspects of model merging.A domain-specific justification for model merging in the case of HMMs applies. It can be seen fromthe example that the structure of the generating HMM can always be recovered by an appropriate sequence ofstate merges from the initial model, provided that the available data ‘covers’ all of the generating model, i.e.,each emission and transition is exercised at least once. Informally, this is because the initial model is obtainedby ‘unrolling’ the paths used in generating the samples in the target model. The iterative merging process,then, is an attempt to undo the unrolling, tracing a search through the model space back to the generatingmodel. Of course, the best-first heuristic is not guaranteed to find the appropriate sequence of merges, or, lesscritically, it may result in a model that is only weakly equivalent to the generating model.3.3.3 Priors for Hidden Markov ModelsFrom the previous discussion it is clear that the choice of the prior distribution is important sinceit is the term in (2.13) that drives generalization. We take the approach that priors should be subject toexperimentation and empirical comparison of their ability to lead to useful generalization. The choice of aprior represents an intermediate level of probabilisticmodeling, between the global choice of model formalism(HMMs, in our case) and the choice of a particular instance from a model class (e.g., a specific HMM structureand parameters). The model merging approach ideally replaces the usually poorly constrained choice of lowlevelparameters with a more robust choice of (few) prior parameters. As long as it doesn’t assign zeroprobability to the correct model, the choice of prior is eventually overwhelmed by a sufficient amount of data.In practice, the ability to find the correct model may be limited by the search strategy used, in our case, themerging process.HMMs are a special kind of parameterized graph structure. Unsurprisingly, many aspects of thepriors discussed in this section can be found in Bayesian approaches to the induction of graph-based modelsin other domains (e.g., Bayesian networks (Cooper & Herskovits 1992; Buntine 1991) and decision trees(Buntine 1992)).