The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 1. INTRODUCTION 3based (or conditioned) on evidence seen in the past, using the framework of probability theory as a consistentmathematical basis.However, this fundamental feature is only truly useful because probabilistic models are also adaptable:there are effective algorithms for tuning a model based on previously observed data, so as to optimizeits predictions on new data (assuming old and new data obey the same statistics).1.2.1 Probabilistic finite-state modelsAn example in point are the probabilistic finite-state models known as Hidden Markov models(HMMs) routinely used in speech recognition to model the phone sequences making up the words to berecognized. The top part of Figure 1.1 shows a simple word model for “and.” Each phonetic realization ofthe word corresponds to a path through the network of states and transitions, with the probabilities indicated.Given a network structure and a training corpus data to be modeled, there are standard algorithms foroptimizing (or estimating) the probability parameters of the HMM to fit the data.However, a more fundamental problem is how to obtain a suitable model structure in first place.The basic idea here will be to construct initial model networks from the observed data (as shown in the bottompart of Figure 1.1), and then gradually transform them into a more compact and general form by a processcalled model merging. We will see that there is a fundamental tension between optimizing the fit of the modelto the observed data, and the goal of generalizing to new data. We will use the Bayesian notions of prior andposterior model probabilities to formalize these conflicting goals and derive a combined criterion that allowsfinding a compromise between them.Chapter 3 describes this approach to structural learning of HMMs and discusses many of the issuesand methods recurring in later chapters.1.2.2 The Miniature Language (* Learning 0) TaskAn additional motivation for the present work came from a seemingly simple task proposed byFeldman et al. (1990): construct a machine learner that could generalize from usage examples of a naturallanguage fragment to novel instances, for an arbitrary natural language. Figure 1.2 shows the essentialelements of this miniature language learning problem, informally known as “¨ the 0” task. The goal isto ‘learn’ ¨ the 0 language from exposure to pairs of corresponding two-dimensional pictures and naturallanguage descriptions. Both the syntax and semantics of the language were intentionally limited to makethe problem more manageable. The purpose of the proposal was to highlight certain fundamental problemswith traditional cognitive science theories, including issue such as dependence on the underlying conceptualsystem, grounding of meaning and categorization in perception, and others which are explored in recent andongoing research (Regier 1992; Feldman et al. 1994).For our purposes we can abstract a (much simpler) subproblem from this interdisciplinary task:given pairs of sentences and associated idealized semantics (e.g., in first-order logic formulae), construct anadequate formal description of the relation between these two for the given language.