The dissertation of Andreas Stolcke is approved: University of ...

ÊS==66@N,ÆÆ=NÆ00ÆÊ=S=N0Æ=#@0#=,CHAPTER 2. FOUNDATIONS 24a conjugate prior, i.e., of the same functional form as the likelihood function for the multinomial. Thelikelihood for a sample from the multinomial with total observed outcomes 1£££7is given by equation(2.4). This means that the prior (2.14) and the likelihood (2.4) combine according to Bayes’ law to give anexpression for the posterior density that is again of the same form, namely:11@ £ (2.15)\@YX1U ] ^+-,Ç ^ ? 1 N1£££S%. 1£££ZÅ-, 0'6Æ =Furthermore, it is convenient that the integral over the product of (2.14) and (2.4) has a closed-form solution.11\@YX1+-,SB0+-,Ç ^ ? 1£££Z. SE07–S 6Å-,2ÆÆ =–SU] ^1£££(2.16)ÆM=Å-,Å-,;ÆÆ—= 1 N1£££ZThis integral will be used to compute the posterior for a given model structure, as detailed in Sections 2.5.7and 3.4.3.To get an intuition for the effect of the Dirichlet prior it is helpful to look at the two-dimensionalcase. For ¢g6 2 there is only one free parameter, say U 1 6Ét , which we can identify with the probability ofheads in a biased coin flip ( U 2 61 LËt being the probability of tails). Assume there is no a priori reasonto prefer either outcome, i.e., the prior distribution should be symmetrical about the value t/60£ 5. This1£££symmetry entails a choice of ’s which are equal, in our case 1 6 2 6 . The resulting prior distributionis depicted in Figure 2.4(a), for various values of Æ%@ . For 1 Æ the prior has the effect of adding Æ%@ L 1Æ&@!Ì Æ‘virtual’ samples to the likelihood expression, resulting in a MAP estimate ofÆ @L 1£ (2.17)eU @Æ—@For 6 1 the prior is uniform and the MAP estimate is identical to the ML estimate.Æw@Figure 2.4(b) shows the effect of varying amounts of data (total number of samples) on theposterior distribution. With no data (Ð6 0) the posterior is identical to the prior, illustrated here for Æ 6 2. As increases the posterior peaks around the ML parameter setting.W d;dÍNdÎL 10For 0 ÏÏ 1 the MAP estimate is biased towards the extremes of the parameter space, U@ 6 0 and U @ 6 1.2.5.6 Description Length priorsThe MDL framework is most useful for designing priors for the discrete, structural aspects ofgrammars. Any (prefix-free) coding scheme for models that assigns 4a code length Ñ450 can be used toinduce a prior distribution over models with?>Ô7ÕoÄ Ö+-,450wÒ|Ó£We can take advantage of this fact to design ‘natural’ priors for many domains. This idea will be extensivelyused with all grammar types.