The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 3. HIDDEN MARKOV MODELS 50distribution for a target language was generated by assigning uniform probabilities at all choice points in agiven HMM topology.The induced models were compared using a variety of techniques. A simple quantitativecomparisonis obtained by computing the log likelihood on a test set. This is proportional to the negative of (the empiricalestimate of) the cross-entropy, which reaches a minimum when the two distributions are identical.To evaluate the HMM topology induced by Baum-Welch training, the resulting models are pruned,i.e., transitions and emissions with probability close to zero are deleted. The resulting structure can then becompared with that of the target model, or one generated by merging. The pruning criterion used throughout?was that a transition or emission had an expected count of less than 103 given the training set.Specifically, we would like to check that the resulting model generates exactly the same discretelanguage as the target model. This can be done empirically (with arbitrarily high accuracy) using a simpleMonte-Carlo experiment. First, the target model is used to generate a reasonably large number of samples,which are then parsed using the HMM under evaluation. Samples which cannot be parsed indicate that theinduction process has produced a model that is not sufficiently general. This can be interpreted as overfittingthe training data.Conversely, we can generate samples from the HMM in question and check that these can be parsedby the target model. If not, the induction has overgeneralized.In some cases we also inspected the resulting HMM structures, mostly to gain an intuition for thepossible ways in which things can go wrong. Some examples of this are presented below.Note that the outcome of the Baum-Welch algorithm may (and indeed does, in our experience) varygreatly with the initial parameter settings. We therefore set the initial transition and emission probabilitiesrandomly from a uniform distribution, and repeated each Baum-Welch experiment 10 times. Merging, on theother hand, is deterministic, so for each group of runs only a single merging experiment was performed forcomparison purposes.Another source of variation in the Baum-Welch method is the fixed total number of model parameters.In our case, the HMMs are fully parameterized for a given number of states and set of possible emissions;so the number of parameters can be simply characterized by the number of states in the HMM. 12In the experiments reported here we tried two variants: one in which the number of states was equalto that of the target model (the minimal number of states for the language at hand), and a second one in whichadditional states were provided.Finally, the nature of the training samples was also varied. Besides a standard random sample fromthe target distribution we also experimented with a ‘minimal’ selection of representative samples, chosento be characteristic of the HMM topology in question. This selection is heuristic and can be characterizedas follows: “From the list of all strings generated by the target model, pick a minimal subset, in order ofdecreasing probability, such that each transition and emission in the target model is exercised at least once, andsuch that each looping transitionis exemplified by a sample that traverses the loop at least twice.” The minimal12 By convention, we exclude initial and final states in these counts.