The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 1. INTRODUCTION 2Instance-based parts of a model can coexist with generalized ones, depending on the degree of similarity¡among the observed samples, allowing the model to adapt to non-uniform coverage of the sample space.The generalization process is driven and controlled by a uniform, probabilistic metric: the Bayesian¡posterior probability of a model, integrating both criteria of goodness-of-fit with respect to the data anda notion of model simplicity (‘Occam’s Razor’).Our 1 approach is quite general in nature and scope (comparable to, say, Mitchell’s version spaces(Mitchell 1982)) and needs to be instantiated in concrete domains to study its utility and practicality. Wewill do that with three different types of probabilistic models: Hidden Markov Models (HMMs), stochasticcontext-free grammars (SCFGs), and simple probabilistic attribute grammars (PAGs).Following this introduction, Chapter 2 presents the basic concepts and mathematical formalismsunderlying probabilistic language models and Bayesian learning, and also introduces our approach to learningin general terms.Chapter 3 (HMMs), Chapter 4 (SCFGs) and Chapter 5 (attribute grammars) describe the particularversions of the learning approach for the various types of languages models. Unfortunately, these chapters(except for Chapter 3) are not entirely self-contained, as they form a natural progression in both ideas andformalisms presented.The following two chapters address various computational problems aside from learning that arisein connection with probabilistic context-free language models. Chapter 6 deals with probabilistic parsing andChapter 7 gives an algorithm for approximating context-free grammars with much simpler ¢ -gram models.These two chapters are nearly self-contained and need not be read in any particular order (with respect eachother or the preceding chapters).future research.Chapter 8 discusses general open issues arising from the present work and gives an outlook onVirtually all probabilistic grammar types and algorithms described in the following chapters havebeen implemented and integrated in an object-oriented framework in CommonLisp/CLOS. The result purportsto be a flexible and extensible environment for experimentation with probabilistic language models. Thedocumentation for this system is available separately (Stolcke 1994).background.The remainder of this introductiongives general motivationand highlights,as well as some historical1.2 Structural Learning of Probabilistic GrammarsProbabilistic language models (or grammars) have firmly established themselves in a number ofareas in recent time (automatic speech recognition being one of the major applications). One importantfactor is their probabilistic nature itself: they can be used to make weighted predictions about future data1 The first person plural will be used throughout, both for stylistic uniformity and to reflect the fact that much of this work was donein collaboration with others. Bibliographic references to co-authored publications are given at the end of this chapter.