The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 4. STOCHASTIC CONTEXT-FREE GRAMMARS 88The criterion used for model evaluation is the posterior of the model structure, as discussed inSection 2.5.7. It is approximated using the same Viterbi method as described for HMMs (Section 3.4.3).This again has the advantage that the posterior is decomposed into a product of terms, one for each grammarnonterminal. Changes to the posterior are computed efficiently by recomputing just the terms pertaining tononterminals affected by the merging or chunking operation.4.3.5 Search strategiesThe question of how to search efficiently for good sequences of merging operators becomes morepressing in the case of SCFGs. The main reason is that the introductionof the new operator, chunking, createsa more complex topology in the search space. In addition, the evaluation of chunking step is not directlycomparable to merging, as chunking does not have a generalizing effect on the grammar (it can only affectthe prior contribution to the posterior).We have experimented with several search strategies for SCFG learning, discussed below. Clearlymore sophisticated ones are possible, and await further study.Best-first search This is the straightforwardextension of our approach to merging with HMMs. All operatortypes and application instances are pooled for the purpose of comparison, and at each step the locally bestone is chosen. This is combined with the simple linear look-ahead extension described in Section 3.4.5 tohelp overcome local maxima.This simple approach often fails because chunking typically has to be followed by several mergingsteps to produce an overall improvement. The look-ahead feature often doesn’t help here as other chunks getin the way between a chunking step and the ‘right’ successive merging choices.Multi-level best-first search One possible solution to the above problem is to make the search procedureaware of the different nature of the two operators, by constraining the way in which they interact. Empirically,the following simple extension of the best-first paradigm seems to work generally well for many SCFGs.The basic idea is that the search operates on two distinct levels, associated with merging andchunking, respectively. Search at the merging level consists of a best-first sequence of merging steps (withlook-ahead). Search at the second level chooses the locally best chunking step, and then proceeds with asearch at level 1. (Clearly, this approach could be generalized to any number of search levels).Notice that in this approach, the chunking steps are not evaluated by trying an exhaustive sequenceof merges following each possible choice. This would entail an overhead that is quite significant even in smallcases.Beam search In a beam search the locality of the search is relaxed by considering a pool a relatively goodmodels simultaneously, rather than only a single one as in best-first search. In Section 3.3 we remarked thatbeam-search for HMMs seems to only very rarely give worthwhile improvements over the best-first approach.