The dissertation of Andreas Stolcke is approved: University of ...

1CHAPTER 6. EFFICIENT PARSING WITH STOCHASTIC CONTEXT-FREE GRAMMARS 1234. How should the parameters (e.g., rule in{ probabilities) be chosen to maximize the probability over atraining set of strings?The incremental model merging algorithm for SCFGs (Chapter 4) requires either (1) or (2) forefficient operation. Traditional grammar parameter estimation is essentially (4), and is typically also usedas a post-processing step to model merging (after the grammar structure has been learned). The algorithmdescribed in this chapter can compute solutions to all four of these problems in a single framework, with anumber of additional advantages over previously presented isolated solutions. It was originally developedsolely as a general and efficient tool and accessory to the model merging algorithm. We then realized that italso solves task (3) in an efficient and elegant fashion, greatly expanding its usefulness, as described below.Most probabilistic parsers are based on a generalization of bottom-up chart parsing, such as theCYK algorithm. Partial parses are assembled just as in non-probabilistic parsing (modulo possible pruningbased on probabilities), while substring probabilities (also known as ‘inside’ probabilities) can be computedin a straightforward way. Thus, the CYK chart parser underlies the ‘standard’ solutions to problems (1) and(4) (Baker 1979), as well as (2) (Jelinek 1985). While the Jelinek & Lafferty (1991) solution to problem (3)is not a direct extension of CYK parsing they nevertheless present their algorithm in terms of its similaritiesto the computation of inside probabilities.In our algorithm, computations for tasks (1) and (3) proceed incrementally, as the parser scans itsinput from left to right; in particular, prefix probabilities are available as soon as the prefix has been seen, andare updated incrementally as it is extended. Tasks (2) and (4) require one more (reverse) pass over the parsetable constructed from the input.Incremental, left-to-right computation of prefix probabilities is particularly important since thatis a necessary condition for using SCFGs as a replacement for finite-state language models in many applications,such a speech decoding. As pointed out by Jelinek & Lafferty (1991), knowing probabilities+-,;1£££1 @0 0 for arbitrary prefixes 0 £££1 @enables probabilistic prediction of possible follow-words 1 @ # 1,1as 1. 0 £££1>@0Í6+-,210 £££1$@C1$@ +-,;1# 10_³ 0 £££1$@0 . These conditional probabilities can then be used as#word transition probabilities in a Viterbi-style decoder or to incrementally compute the cost function for a+-,21$@stack decoder (Bahl et al. 1983).Another application where prefix probabilities play a central role is the extraction ¢ of -gramprobabilities from SCFGs, a problem that is the subject of Chapter 7. Here, too, efficient incrementalcomputation saves time since the work for common prefix strings can be shared.The key to most of the features of our algorithm is that it is based on the top-down parsing methodfor non-probabilistic CFGs developed by Earley (1970). Earley’s algorithm is appealing because it runs withbest-known efficiency on a number of special classes of grammars. In particular, Earley parsing is moreefficient than the bottom-up methods in cases where top-down prediction can rule out potential parses ofsubstrings. The worst-case computational expense of the algorithm (either for the complete input, or theincrementally for each new word) is as good as that of the other known specialized algorithms, but can besubstantially better on well-known grammar classes.