The dissertation of Andreas Stolcke is approved: University of ...

ÆCHAPTER 3. HIDDEN MARKOV MODELS 69At the very least, the method chosen should bewell-defined, i.e., correspond to some probabilistic model that represents a proper distribution over all¡strings;unbiased with respect to the methods being compared, to the extent possible.¡Standard back-off models (where a second model is consulted if, and only if, the first one returnsprobability zero) do not yield consistent probabilities unless they are combined with ‘discounting’ of probabilitiesto ensure that the total probabilitymass sums up to unity (Katz 1987). The discounting scheme, as wellas various smoothing approaches (e.g., adding a fixed number of virtual ‘Dirichlet’ samples into parameterestimates) tend to be specific to the model used, and are therefore inherently problematic when comparingdifferent model-building methods.To overcome these problems, we chose to use the mixture models approach described in Section2.3.1. The target models to be evaluated are combined with a simple back-off model that guaranteesnon-zero probabilities, e.g., a bigram grammar with smoothed parameters. This back-off grammar is identicalin structure for all target models. Unlike discrete back-off schemes, the target and the back-up are alwaysconsulted both for the probability they assign to a given sample, which are then weighted and averagedaccording to a mixture proportion.When comparing two model induction methods, we first let each induce a structure. Each is builtinto a mixture model, and both the component model parameters and the mixture proportions are estimatedusing the EM procedure for generic mixture distributions. To get meaningful estimates for the mixtureproportions, the HMM structure is induced based on a subset of the training data, and the full training datais then used to estimate the parameters, including the mixture weights. This holding-out of training datamakes the mixture model approach similar to the deleted interpolation method (Jelinek & Mercer 1980). Themain difference is that the component parameters are estimated jointly with the mixture proportions. 18 In ourexperiments we always used half of the training data in the structure induction phase, adding the other halfduring the EM estimation phase. Also, to ensure that the back-off model receives a non-zero prior probability,we estimate the mixture proportions under a simple symmetrical Dirichlet prior with Æ 1 62 6 1£ 5.3.6.2.4 Results and discussionHMM merging was evaluated in two variants, with and without the single-output constraint. Ineach version, three settings of the structure prior weight. were tried: 0.25, 0.5 and 1.0. Similarly, forBaum-Welch training the preset number of states in the fully parameterized HMM was set to 1.0, 1.5 and1.75 times the longest sample length. For comparison purposes, we also included the performance of theunmerged maximum-likelihood HMM, and a biphone grammar of the kind used in the mixture models usedto evaluate the other model types. Table 3.1 summarizes the results of these experiments.18 This difference can be traced to the different goals: in deleted interpolation the main goal is to gauge the reliability of parameterestimates, whereas here we want to assess the different structures.