The dissertation of Andreas Stolcke is approved: University of ...

are then summed over all nonterminals>, and the result is once multiplied by the rule probability +-,= ¸CHAPTER 6. EFFICIENT PARSING WITH STOCHASTIC CONTEXT-FREE GRAMMARS 1586.6 Implementation IssuesThis section briefly discusses some of the experience gained from implementing the probabilisticEarley parser. Implementation is mainly straightforward and many of the standard techniques for context-freegrammars can be used (Graham et al. 1980). However, some aspects are unique due to the addition ofprobabilities.6.6.1 PredictionDue to the collapsing of transitive predictions, this step can be implemented in a very efficient andstraightforward manner. As explained in Section 6.4.5, one has to perform a single pass over the currentstate set, identifying all nonterminals> occurring to the right of dots, and add states corresponding to all@that are reachable through the relation> left-corner =. As indicated in equation (6.3), C/¸contributions to the forward probabilities of new states have to be summed when several paths lead to theproductions=same state. However, the summation in equation (6.3) can be mostly eliminated if the Æ values for all oldstates with the same nonterminal> are summed first, and then multiplied by © . These quantities CTœ=ë0,>to give the forward probability for the predicted state.@E06.6.2 CompletionUnlike prediction, the completion step still involves iteration. Each complete state derived bycompletion can potentially feed other completions. An important detail here is that to ensure that allcontributions to a state’s andŠ are summed before proceeding with using that state as input to furthercompletion steps.ÆOne approach to this problem is to insert complete states into a prioritized queue. The queue ordersstates by their start indices, highest first. This is because states corresponding to later expansion always haveto be completed first before they can lead to the completion of earlier expansions. For each start index, theentries are managed as a first-in-first-out queue, ensuring that the directed dependency graph formed by thestates is traversed in breadth-first order.A completion pass can now be implemented as follows. Initially, all complete states from theprevious scanning step are inserted in the queue. States are then removed from the front of the queue, andused to complete other states. Among the new states thus produced, complete ones are again added to thequeue. The process iterates until no more states remain in the queue. Because the computation of probabilitiesalready includes chains of unit productions, states derived from such productions need not be queued, whichalso ensures that the iteration terminates.A similar queuing scheme, with the start index order reversed, can be used for the reverse completionstep needed in the computation of outer probabilities (Section 6.5.2).