The dissertation of Andreas Stolcke is approved: University of ...

..1 £££1; 450,1 £££1; 450CHAPTER 2. FOUNDATIONS 8distinction between language (distribution) and grammar (description), and simply write+-,+-,)/. 450'6)/. ¨450702.2.1 Interpretation of probabilitiesThe probabilities assigned to strings by a model can be interpreted as long-term relative frequenciesof those strings, assuming conditionalindependence of samples. This frequentist interpretationbecomes problematicas we introduce probabilities of entities that do not correspond to outcomes of repeatable experiments,such as the models themselves.In such cases it is more plausible to think of probabilities as degrees of belief, the subjectivist interpretationof probabilities. The classic paper by Cox (1946) shows that the two interpretations are compatiblein that they observe the same calculus if some simple assumptions about beliefs and their compositionalproperties are made.The unification of the frequentist and the subjectivist treatment of probabilities is fundamental tothe Bayesian approach described later. It allows probabilities to be used as the common ‘currency’ relatingobservations and beliefs about underlying explanations for observations.2.2.2 An example: 8 -gram modelsA simple example both illustrates these notions and introduces a useful standard tool for later use.An ¢ -gram model defines the probabilityof a string +9,21 . 450 as a product of conditionalprobabilities+9,21+9,21+-,;1+-,21$=. 450:61. $; 4502. $ 1 1; 450=$?£££+9,21$@1>@A?B=1$@C?#1 ££££££+9,21ED1ED ?B=1ED ?+-,1EDwhere G is the string length, 1 @ is the H th symbol in string 1 , and $ is a special delimiter marking beginningand end of a string.The parameters of an ¢ -gram model are thus the probabilities$.1BD ?E=#1 £££#2 £££10+-,21 =1 =>?for1 =JI1£££ all 1$ . (It is convenient to allow a prefix of 1 1 £££1 =>?3"K.11 to take on the value $ to denotethe left end of the string. The context always makes it clear which end of a string $ stands for.)It is useful to compare definition (2.1) to the expression for +-,21 . 450 arrived at by repeated conditioning,which is true of any probability distribution over strings:0 (2.2)We see that that ¢ -gram models represent exactly those distributions in which each symbol is independent ofthe beginning of the string given just the immediately preceding ¢JL 1 symbols (including the position of theleft string boundary).. 450%61. $02. $ 1 10