The dissertation of Andreas Stolcke is approved: University of ...

@For example, for multinomials an unbiased, consistent estimator of the component probabilities Us@v,vv,,,6@,t000CHAPTER 2. FOUNDATIONS 11is given by the observed averagesU @£ (2.8)eZero bias is not always desirable, however, as attested by the extensive literature on how to modifyestimates for ¢ -gram (and similar) models so as to avoid estimates from taking on 0 values, specifically inlinguistic domains. According to (2.8) this would be the case whenever some outcome has not been observedat all (6 0). In many domains and for realistic sample sizes one expects many counts to remain zero eventhough the underlying parameters are not; hence one wants to bias the estimates away from zero. See Church& Gale (1991) for a survey and comparison of various methods for doing that. Another reason for introducingbias is to reduce the variance of an estimator (Geman et al. 1992).2.2.6 Likelihood and cross-entropyIf the ) data is given and fixed, we can view )/. 450 as a function of 4 , the likelihood (function).A large class of estimators, so called maximum likelihood (ML) estimators, can be defined as the maxima of+-,likelihood functions. For example, the simple estimator (2.8) for multinomials happens to also be the MLestimator, as setting the parameters U @ to their empirical averages maximizes the probability of the observeddata.Intuitively, a high likelihood means that the 4 model ‘fits’ the data well. This is because a modelallocates a fixed probability mass (unity) over the space of possible strings (or sequences of strings). Tomaximize the probability of the observed strings the unobserved ones have as little probability as possible,within the constraints of the model. In fact, if a model class allows assigning probabilities to samplesaccording to their relative frequencies as in (2.8), this is the best one can do.An alternative measure of the fit or closeness of a model to a distribution is based on the concept ofentropy. The relative entropy (also known as the Kullback-Leibler distance) between distributionst two anduis defined as0 logu ,21(2.9)tw.o. u 0w6xLgyz{t,21,;1This can be written as,210 log u ,21,;10 logt,21tw.o. u 0&6|L y z5t0%L y z{tThe second sum is the familiar } entropyterm in which both distributions appear. We will call this first term the cross-entropy 3 }¦~tE0 of the distribution t , whereas the first sum is an entropy-like, u 0 , givingu 0