The dissertation of Andreas Stolcke is approved: University of ...

£££§C11y‚“sꄃ+-,9 0&61CCHAPTER 6. EFFICIENT PARSING WITH STOCHASTIC CONTEXT-FREE GRAMMARS 129where@1?@2£££?@6are string of terminals and/or nonterminals, ) ¸.is production of{, and@2 isderived from@1 by replacing one occurrence of )with..derivations ) C1producing from ) . 6 1 Cb) The string probability +-, ) 0 (of 1 given ) ) is the sum of the probabilities of all left-most1c) The sentence 1probability C +9,9(of given{) is the string probability given the symbol9 start 0 of{. By definition, this is also the 1 probability assigned to by the grammar{.1 .{§0 +-,21d) The prefix probability C¬+-,9having 1 as a prefix,0 (of 1 given{) is the sum of the probabilities of all sentence strings C+-,9÷s0£(In particular, 1). CTzA0w6 +9,9In the following, we assume that the probabilities in a SCFG are proper and consistent as definedin Booth & Thompson (1973), and that the grammar contains no useless nonterminals (ones that can neverappear in a derivation). These restrictions ensure that all nonterminals define probability measures overstrings, i.e., +-, )C10 is a proper distribution over 1 for all ) . Formal definitions of these conditions aregiven in Section 6.4.8.6.4.2 Earley paths and their probabilitiesIn order to define the probabilities associated with parser operation on a SCFG, we need the conceptof a path, or partial derivation, executed by the Earley parser.Definition 6.2a) An (unconstrained) Earley path, or simply path, is a sequence of Earley states linkedby prediction, scanning, or completion. For the purpose of this definition, we allow scanning to operatein ‘generation mode,’ i.e., all states with terminals to the right of the dot can be scanned, not just thosematching the input. (For completed states, the predecessor state is defined to be the complete state fromthe same state set contributing to the completion.)b) A path is said to be constrained by (or generate) a string 1 if in all scanned states the terminalsimmediately to the left of the dot, in sequence, form the string 1 .c) A path is complete if the last state on it matches the first, except that the dot has moved to the end ofthe RHS.d) We say that a path starts with nonterminal ) if the first state on it is a predicted state with ) on theLHS.6 In a left-most derivation each step replaces the nonterminal furthest to the left in the partially expandedstring. The order of expansionis actually irrelevant for this definition, due to the multiplicative combination of production probabilities. We restrict summation toleft-most derivations to avoid counting duplicates, and because left-most derivations will play an important role later.