The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 4. STOCHASTIC CONTEXT-FREE GRAMMARS 904.3.6 Miscellaneous4.3.6.1 Restricted chunkingWe already observed that unrestricted chunking can produce arbitrary CFG structures. For practicalpurposes, however, the set of potential chunks needs to be restricted to avoid generating an infeasibly largenumber of hypothesis in each search step. Specifically, the following restrictions can optionally be imposed.1. No null productions. These would result from proposing empty chunks.2. No unit (or chain) productions. These would be the result of singleton chunks.3. A sequence of nonterminals (of any length exceeding 1) needs to occur at least twice in the grammar tobe a candidate for chunking.4. A chunk is replaced wherever it occurs, as opposed to choosing only a subset of the occurrences.It is not known what sort of global constraint the last two restrictions place on the grammars that can beinferred, since both make reference to the form of an intermediate grammar hypothesis, which depends onthe actually occurring samples and the dynamics of the search process.4.3.6.2 Chunking undoneOccasionally a chunking operation and the nonterminals created for it become superfluous inretrospect, because only one occurrence of the nonterminal remains in the grammar as a result of productionsmerging. In this case it is beneficial to undo the chunking operation, a step we call unchunking. Note thatthe final outcome in such cases could have been achieved by not choosing to chunk in the first place, butunchunking provides a trivial and convenient way to recover from chunks that seem temporarily advantageous. 74.3.6.3 Efficient sample incorporationThe simple extension of the batch merging procedure to an incremental, on-line version was alreadydiscussed for the HMM case (Section 3.3.5), and can be applied unchanged for SCFGs. This includes theuse of a prior factor. weighting to control generalization and prevent overgeneralization during the earlyrounds of incremental merging (Section 3.4.4). Incremental merging is the default method used in all theexperiments reported below, unless otherwise noted.As a result of incremental nonterminal merging, a sequence of nonterminals that was previouslythe subject of chunking can reappear. In that case the chunking operation is re-applied and the previouslyallocated LHS nonterminal is used in replacing the re-occurring sequence. This special form of the chunkingoperator is known as rechunking.Various additional strategies are possible in order to reduce the number of new nonterminalscreated during sample incorporation, thereby reducing subsequent merging work. The drawback of all these7 The unchunkingoperation was adapted from Cook et al. (1976) after we had noticed the similarity between the two approaches (seethe discussion in Section 4.4.3)