The dissertation of Andreas Stolcke is approved: University of ...

) In particular, the string probability +9,9 c) The prefix probability C +9,91 C+-,91+-,9 1y @ 10&6 ¸:Œ”'/åŠ@_,6)600 ¸ ¸y 0%6 ¸ .1BD ?Œ”'/z§‡1åKÆ%DY,6)CHAPTER 6. EFFICIENT PARSING WITH STOCHASTIC CONTEXT-FREE GRAMMARS 132set H . Having said that, we will refer to Æ simply as a probability, both for the sake of brevity, and to keep theanalogy to the HMM terminology of which this is a generalization. 7 Note that for scanned states, is alwaysa probability, since by definition a scanned state can occur only once along a path.ÆThe inner probabilities, on the other hand, represent the probability of generating a substring of theinput from a given nonterminal, using a particular production. Inner probabilities are thus conditional on thepresence of a given )nonterminal with expansion starting at position", unlike the forward probabilities,which include the generation history starting with the initial state. The inner probabilities as defined herecorrespond closely to the quantities of the same name in Baker (1979). The sum ofŠof all states with a givenLHS ) is exactly Baker’s inner probability for ) .The following lemma is essentially a restatement of Lemma 6.2 in terms of forward and innerprobabilities. It shows how to obtain the sentence and string probabilities we are interested in, provided thatforward and inner probabilities can be computed effectively.a) that9 Provided CÀprobability that a nonterminal )0å…å…å6?Lemma 6.3 The followingassumes an Earley chart constructed by the parser on an input string 1 with .1a possible left-most derivation of the grammar (for some@), thegenerates the substring 16£££1 can be computed as the sum.6 G .1>@A?is+9,1)z@ @C? 16å…å…å ) C(sum of inner probabilities over all complete states with LHS )and start index")..—£ 00 can be computed as 89%£…0C10°6 ŠD ,9%£…0C1Æ D ,0 , with .1.6|G , can be computed as1£ ˜M0(sum of forward probabilities over all scanned states).The restriction in (a) that )be preceded by a possible prefix is necessary since the Earley parserat position H will only pursue derivations that are consistent with the input up to position H . This constitutesthe main distinguishing feature of Earley parsing compared to the strict bottom-up computation used inthe standard inside probability computation (Baker 1979). There, inside probabilities for all positions andnonterminals are computed, regardless of possible prefixes.7 The same technical complication was noticed by Wright (1990) in the computation of probabilistic LR parser tables. The relationto LR parsing will be discussed in Section 6.7.3. Incidentally, a similar interpretation of forward ‘probabilities’ is required for HMMswith non-emitting states.8 The definitions of forward and inner probabilities coincide for the final state.