The dissertation of Andreas Stolcke is approved: University of ...

eingDefinition 6.1 The following quantities are defined relative to a SCFG{, a nonterminal ) , and a string 1CHAPTER 6. EFFICIENT PARSING WITH STOCHASTIC CONTEXT-FREE GRAMMARS 127It is easy to see that Earley parser operations are correct, in the sense that each chain of transitions(predictions, scanning steps, completions) corresponds to a possible (partial) derivation. Intuitively, it is alsotrue that a parser that performs these transitions exhaustively is complete, i.e., it finds all possible derivations.Formal proofs of these properties are given in the literature, e.g., Aho & Ullman (1972). The relationshipbetween Earley transitions and derivations will be stated more formally in the next section.The parse trees for sentences can be reconstructed from the chart contents. We will illustrate this inSection 6.5 when discussing Viterbi parses.Table 6.1 gives an example for Earley parsing, in the form of a trace of transitions as they areperformed by our implementation.Earley’s parser can deal with any type of context-free rule format, even with null orA-productions,i.e., those that replace a nonterminal with the empty string. Such productions do however require specialattention, and make the algorithm and its description more complicated than otherwise necessary. In thefollowing sections we assume that no null productions have to be dealt with, and then summarize thenecessary changes in Section 6.4.7. One might chose to simply preprocess the grammar to eliminate nullproductions, a process which is also described.6.4 Probabilistic Earley Parsing6.4.1 Stochastic context-free grammarsA stochastic context-free grammar (SCFG) extends the standard context-free formalism by addingprobabilities to each production:¸ )òtróôwhere the rule probabilityt is usually written as +-, ) ¸.t is a conditional probability, of production ) ¸ .. This notation to some extent hides the fact thatchosen, given that is up for expansion. The.E0)probabilities of all rules with the same )nonterminal on the LHS must therefore sum to unity. Contextfreenessin a probabilistic setting translates into conditional independence of rule choices. As a result,complete derivations have joint probabilities that are simply the products of the rule probabilities involved.The probabilities of interest mentioned in Section 6.1 can now be defined formally.over the alphabet of{. 5a) The probability of a (partial) derivation@1C~@2C £££