The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 4. STOCHASTIC CONTEXT-FREE GRAMMARS 91approaches is that they reuse previously hypothesized grammar structures, possibly preventing the algorithmfrom considering better alternatives.1. Avoid duplicate samples: incorporate duplicate samples only once, with appropriately adjusted counts.This is a trivial optimization that can never do harm.2. Try parsing samples first before resorting to the ordinary creation of new productions. If a new sample isparsed successfully counts on the old productions are updated to reflect the new sample. 8 This methodsubsumes strategy 1 above. (See Section 6.5.3 for ways to efficiently handle the parsing of bracketedsamples, which is needed if this method is to applied to structured samples.)3. To save initial merging of preterminals, reuse existing preterminals where possible. This precludes thecreation of grammars with ambiguity at the level of lexical productions.4. Try to parse the new sample into string fragments using existing rules, and add only a top-levelproduction to link these fragments to the start symbol. This subsumes both strategy 2 and strategy 3.(Section 6.5.4 describes one approach to parsing ungrammatical samples into fragments that can beused here.)Unless noted otherwise, only strategy 2 was used in obtaining the results reported here.4.4 Related Work4.4.1 Bayesian grammar learning by enumerationWe already mentioned Horning (1969) as an early proponent of the Bayesian version of grammarinference by enumeration, as the principle is general enough to be applied (in theory) to any type of probabilisticgrammar. Horning’s focus was actually on probabilistic CFGs, and the formal device used to enumerategrammars, as well as to assign prior probabilities, was a grammar-generating grammar, or grammar grammar.As expected, enumeration is not practical beyond the simplest target grammars, but Horning’s work istheoretically important and was one of the first to point out the use of posterior probabilities as a formalizationof the simplicity vs. data fit trade-off.4.4.2 Merging and chunking based approachesThe idea of combining merging and chunking with a hill-climbing style search procedure to induceCFG structures seems to have been developed independently by several researchers. Below is a list of thosewe are aware of.8 If the sample is ambiguous the counts could be updated for all derivation according to their respective probabilities, or using onlyfor the Viterbi derivation. In either case the likelihood of the sample will be underestimated by the Viterbi-based computation of theposterior probability. Updating according to the Viterbi derivation should favor the creation of unambiguous grammar structures, but nodetailed comparisons have been done on this issue.