The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 2. FOUNDATIONS 22¡An explicit, probabilistic bias expressed by the prior +-, 450 .An implicit, heuristic bias as part of the choice of the topology of the search space (the search bias).¡Even if the merging operator(s) are defined such that all models in the space are reachable from theinitialmodel, this bias is still significant due to the local and sequential nature of the posterior probabilitymaximization.We will return to the question of how to relax the search bias by using less constrained search methods(Section 3.4.5).The remaining section of this chapter present some of the common mathematical tools needed ininstantiating the Bayesian model merging approach in the domain of probabilistic grammars. The followingchapters 3, 4 and 5 each describe one such instantiation.2.5.4 Minimum Description LengthBayesian inference based on posterior probabilities has an alternative formulation in terms offinformation-theoretic concepts. The dualism between the two formulations is useful both for a deeperunderstanding of the underlying principles, and for the construction of prior distributions (see Section 2.5.6below).The maximization of+-,+-,+-,4|_)g0w6450450 )À.implicit in Bayesian model inference is equivalent to minimizingL log +-, 4|_)h0'6|L log +-, 450ÁL log +-, )/. 450F£Information theory tells us that the negative logarithm of the probability of a discrete event b is the optimalcode word length for communicating an instance of b , so as to minimize the average code length of arepresentative message.description lengths.Accordingly, the terms in the above equation can be interpreted as message orSpecifically, L log +9, 450 is the description length of the model under the prior distribution;L log +-, )/. 450 corresponds to a description of the data ) using 4 as the model on which code lengthsare based. The negative log of the joint probability can therefore be interpreted as the total description lengthof model and data.Inference or estimation by minimum description length (MDL) (Rissanen 1983; Wallace & Freeman1987) is thus equivalent to, and a useful alternative conceptualization of posterior probability maximization. 99 The picture become somewhat more complex when distributions over continuous spaces are involved. For those the correspondingMDL formulation has to consider the optimal granularity of the discrete encoding. We will can conveniently avoid this complication asthe only formal use of description lengths will be to devise priors for discrete objects, namely, grammar structures.