The dissertation of Andreas Stolcke is approved: University of ...

4ä Ö ÕÃ,CHAPTER 3. HIDDEN MARKOV MODELS 37Description Length priorsWe can use the MDL framework as discussed in Section 2.5.6 to derive simplepriors for HMM structures from various coding schemes. For example, a natural way to encode the transitionsand emissions in an HMM is to simply enumerate them. Each transitioncan be encoded using log , .ñÞ9.ñN 10 bits,since there are .ñÞ9. possible transitions, plus a special ‘end’ marker which allows us not to encode the missingtransitions explicitly. The total description length for all transitions from ¢state is thusSimilarly, u Õ ä Öall emissions from can ¢ ú be coded . 3§.N 10 using log bits. 4 , uä Ö Õlog , .ñÞã.N 10 . àThe resulting priorû…ü;ý , ?B= ?E= ü;ý ûÿ(3.8)10 3§.N . þ. 4Tø&0%Ò.ƒÞã.N 10+-,has the property that small differences in the number of states matter little compared to differences in the totalnumber of transitions and emissions.We have seen in Section 2.5.7 that the preferred criterion for maximization is the posterior ofstructure +-, 4ÀÃM. )g0 , which requires integrating out the parameters UÄ . In Section 3.4 we give a solution forthis computation that relies on the approximation of sample likelihoods by Viterbi paths.3.3.4 Why are smaller HMMs preferred?Intuitively, we want an HMM induction algorithm to prefer ‘smaller’ models over ‘larger’ ones,other things being equal. This can be interpreted as a special case of ‘Occam’s razor,’ or the scientific maximthat simpler explanations are to be preferred unless more complex explanations are required to explain thedata.Once the notions of model size (or explanation complexity) and goodness of explanation arequantified, this principle can be modified to include a trade-off between the criteria of simplicity and datafit. This is precisely what the Bayesian approach does, since in optimizing the product 450+-,)/. 450 a+-,compromise between simplicity (embodied in the prior) and fit to the data (high model likelihood) is found.But how is it that the HMM priors discussed in the previous section lead to a preference for ‘smaller’or ‘simpler’ models? Two answers present themselves: one has to do with the general phenomenon of ‘Occamfactors’ found in Bayesian inference; the other is related, but specific to the way HMMs partition data forpurposes of ‘explaining’ it. We will discuss each in turn.3.3.4.1 Occam factorsConsider the following scenario. Two pundits, 4 1 and 4 2, are asked for their predictions regardingan upcoming election involving a number of candidates. Each pundit has his/her own ‘model’ of the politicalprocess. We will identify these models with their respective proponents, and try to evaluate each according4 The basic idea of encoding transitions and emissions by enumeration has various more sophisticated variants. For example, onecould base the enumeration of transitions on a canonical ordering of states, such that only log¤£¦¥1‡§¥log£¦¥©¨¨¨¥s££¥1‡bits are required. Or one could use the-out-of-£-bit integer coding scheme described in Cover & Thomas (1991) and used for MDLinference in Quinlan & Rivest (1989). Any reasonable Bayesian inference procedure should not be sensitive to such minor difference inthe prior, unless it is used with too little data. Our goal here is simply to suggest priors that have reasonable qualitative properties, andare at the same time computationally convenient.