The dissertation of Andreas Stolcke is approved: University of ...

,,vv,v,v,,directly. However, note that for minimization purposes only the } ~,– derives 1 +-,v,”,CHAPTER 2. FOUNDATIONS 12follows thatu 0Œ‹ },u 0Ž‹ 0, with t%.o.between distributions, or between distributions and models. 4Computingtw.o. u 0w6 0 iff t 6 u. This justifies thinking oftw.o. u 0 as a pseudo-distancescenarios at most one is known, e.g., because it is given by a model. For example, we might use the relativet%. . u 0 exactly presumes knowledge of the full distributions t and u , but in typicalentropy to define an estimator for model parameters, such that the estimated value is that which vminimizes45070 , where t is the distribution from which the samples are drawn. Since t is not known, we cannotIt can be shown that }Š~tB0 always, with equality if and only if the two distributions are identical. Itcomputet%.o. ¨u 0 term is relevant, } tB0 is,u 0 to be minimized, which,an unknown constant, but one that remains fixed as 4 is varied. This leaves } ~is the expected value of L log , u 0 under the true distributiont0 . It can therefore be estimated by averagingover the sample corpus )10 (2.10),;1} ~. )/.y zs“” log u ,;1tB0&‘’LWe thus see that estimated cross-entropy is proportional (by a factor L1 • of) to the log of the•likelihood. Therefore, ML estimators are effectively also minimum relative entropy estimators.2.3 Grammars with hidden variablesAlthough all grammars considered here generate their samples through a combination of multinomials,the sequence of choices that can give rise to a given sample is not always uniquely determined, unlikefor ¢ -gram grammars. There, one can uniquely identify the sequence of choices leading to the generationof a complete string, by inspecting the ¢ -grams occurring in the string in left-to-right order. Knowing the-grams, one can then compute their probabilities, and hence the probability of the string itself (by taking¢products).A complete sequence of generator events (multinomial samples) that generate string 1 is called aderivation of 1 . (Thus, for ¢ -gram models the only derivation of a strings is the string itself.) Grammarsthat generate strings with more than one derivation are called ambiguous. Each derivation – has a probabilitywhich is the product of the probabilities of the multinomial outcomes making up the derivation. In general,then, a string probability is a sum of derivation probabilities, namely, of all derivations generating the samestring:. 450—6 y+-,;1–