The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 4. STOCHASTIC CONTEXT-FREE GRAMMARS 93likelihood loss in that case. Algorithms based on just the relative frequencies of samples, on the otherhand, will be indifferent to this change in absolute frequencies. Also, the Bayesian approach gives anintuitive interpretation to the weighting factor that is useful in practice to globally balance the complexity anddiscrepancy terms (Section 3.4.4).Cook et al. (1976) propose a larger set of operators which partly overlaps with our merging andchunking operations. Chunking is known under the name ‘substitution.’ Merging is not directly available,but similar effects can be obtained by an operation called ‘disjunction,’ which creates new nonterminalsthat expand to one of a number of existing nonterminals. They also have special operations for removingproductions which are subsumed or made redundant by others. These can mostly me emulated with merging,although explicit testing for, and elimination of redundant productions is also useful in our algorithm, sinceit shortcuts combinations of induction steps (i.e., they are macro operators in search parlance). 11Cook et al. (1976) evaluate their algorithm using a number of benchmark grammars; these will bereexamined below using our Bayesian algorithm.4.5 EvaluationThe merging algorithm for SCFGs has been evaluated in a number of experiments. These fallnaturally into two broad categories: simple formal languages and various grammar fragments modelingaspects of natural language syntax.4.5.1 Formal language benchmarksThis group of test grammars has been extracted from the article by Cook et al. (1976) discussed inSection 4.4. Except for the last two, they represent examples of simple context-free formal languages as theyare typically given in textbooks on the subject.The main advantage of using this same set of grammars is that the results can be compared. Sincethe two algorithms (ours and Cook’s) have similar underlying intuitions about structural grammar inductionwe expect similar results, but it is important to verify that expectation.Experimental setup To replicate the examples given by Cook, we used the following procedure. Thesamples given in the paper are used unchanged, except that sample probabilities were converted into countssuch that the total number was 50 for each experiment. These were then incorporated into initial grammarsand merged in batch mode. Our SNF grammar format could introduce a subtle inductive bias not presentin Cook original experiments. The merging procedure was therefore constrained to always maintain aone-to-one correspondence between terminals and preterminals, effectively making terminals redundant andletting preterminals function as the true terminals, in terms of which arbitrary productions are now possible.Chunking of single nonterminals was allowed.11 Notice how repeated merging and chunking effectively eliminate redundant productions in the example of Section 4.3.2.