The dissertation of Andreas Stolcke is approved: University of ...

. canCHAPTER 3. HIDDEN MARKOV MODELS 71The table also includes useful summary statistics of the model sizes obtained, and the time it took tocompute the models. The latter figures are obviously only a very rough measure of computational demands,and their comparison suffers from the fact that the implementation of each of the methods may certainly beoptimized in idiosyncratic ways. Nevertheless these figures should give an approximate idea of what to expectin a realistic application of the induction methods involved.One important general conclusion from these experiments is that both the merged models and thoseobtained by Baum-Welch training do significantlybetter than the two ‘dumb’ approaches, the bigram grammarand the ML HMM (which is essentially a list of observed samples). We can therefore conclude that it pays totry to generalize from the data, either using our Bayesian approach or Baum-Welch on an HMM of suitablesize.Overall the difference in scores even between the simplest approach (bigram) and the best scoring(merging,.T6 1£ one 0) are quite small, with phone perplexities ranging from 1.985 to 1.849. This is notsurprising given the specialized nature and small size of the sample corpus. Unfortunately, this also leavesvery little room for significant differences in comparing alternate methods. However, the advantage of thebest model merging result (unconstrained with. 6 1£ outputs 0) is still significant compared to the bestBaum-Welch (size factor 1.5) result (t Ï 0£ 041). Such small differences in log probabilities would probablybe irrelevant when the resulting HMMs are embedded in a speech recognition system.Perhaps the biggest advantage of the merging approach in this application is the compactness of theresulting models. The merged models are considerably smaller than the comparable Baum-Welch HMMs.This is important for any of the standard algorithms operating on HMMs, which typically scale linearly withthe number of transitions (or quadratically with the number of states). Besides this advantage in productionuse, the trainingtimes for Baum-Welch grow quadratically with the number of states for the structure inductionphase since it requires fully parameterized HMMs. This scaling is clearly visible in the run times we observed.Although we haven’t done a word-by-word comparison of the HMM structures derived by mergingand Baum-Welch, the summary of model sizes seem to confirm our earlier finding (Section 3.6.1) that Baum-Welch training needs a certain redundancy in ‘model real estate’ to be effective in finding good-fitting models.Smaller size factors give poor fits, whereas sufficiently large HMMs will tend to overfit the training data.The choice of the weights. prior for HMM merging (Section 3.4.4) controls the model size inan indirect way: larger values lead to more generalization and smaller HMMs. For best results this valuecan be set based on previous experience with representative data. This could effectively be done in a crossvalidationlike procedure, in which generalization is successively increased starting with small.’s. Due tothe nature of the merging algorithm, this can be done incrementally, i.e., the outcome of merging with a smallbe submitted to more merging at a larger. value, until further increases reduce generalization on thecross-validation data.