The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 8. FUTURE DIRECTIONS 1828.2 Noisy samplesA related problem concerns the potential presence of noise in the sample, i.e., samples randomlyaltered by an independent process, such as random replacement of symbols. Plain model merging algorithmswill simply try to model the language resulting from the composition of the undistorted language and thenoise process. If the probabilities of distortions by noise are small enough one could expect to recover theoriginal language model by pruning low probability parts of the induced model structures.Pruning techniques have indeed been applied successfully in conjunction with the HMM mergingalgorithm as applied to multiple-pronunciation models (Section 3.6.2) in cases were the training corpus wasknown to contain outlier (mislabeled) samples, as reported by Wooters (1993). Again, a formal characterizationof the conditions under which this method is successful would be obviously useful.8.3 More informative search heuristics and biasesThe work so far clearly shows that, as expected, increased model expressiveness requires a morevaried repertoire of search operators, which in turn necessitates increasingly non-local search strategies.Carefully chosen heuristics and macro operators can counter the increase in search complexity in some cases.This raises the question of what principled sources of search bias might be used. A main themeof the present studies was that a considerable amount of linguistic structure can be found by model merging,based solely on non-informative priors and ‘dumb’ search strategies. However, linguistic theories with stronga priori bases should help in cases where the uninformed approaches turn out to be insufficient. A prerequisitefor this approach is that the predictions of the linguistic theory can be cast into effective search heuristics.8.4 Induction by model specializationPure model merging is based on operators that leave the weak generative capacity of the modelsunchanged or produce a strictly more inclusive model. We already mentioned an alternative, inverse approachin Chapter 3: model splitting or specialization. Given the local nature of the search process, we expectimprovements from adding operators that can effectively undo previous merging steps.Also, a learning algorithm that goes from general-to-specific incorporates a different global bias inthe face of local search: it will tend to err on the side of the too general, rather than the too specific model.Having both types of biases available allows choosing or mixing them according to the needs of particularapplications. For example, if it is important that the resulting model have high coverage of new data, onemight prefer over-general models.The approaches in the literature that make use of state splittingare so far based entirely on likelihoodcriteria. However, the Bayesian evaluation methods used in this thesis are clearly separable from the mergingand search components of the overall approach, and should apply to other systems as well. A first practical step