The dissertation of Andreas Stolcke is approved: University of ...

==Ì==ÌCHAPTER 4. STOCHASTIC CONTEXT-FREE GRAMMARS 92Cook et al. (1976) exhibit a procedure similar to ours, using a somewhat different set of operators. 9Their approach is also aimed at probabilistic SCFGs, but uses a conceptually quite different evaluationfunction, which will be discussed in more detail below, as it illustrates a fundamental feature of the Bayesianphilosophy adopted here.Langley (1994) discusses a non-probabilistic CFG induction approach using the same merging andchunking operators as described here, which in turn is based on that of Wolff (1978). Langley’s CFG learneralso alternates between merging and chunking. No incremental learning strategy is described, although addingone along the lines presented here seems straightforward. The evaluation function is non-probabilistic, butincorporates several heuristics to control data fit and a bias towards grammar ‘simplicity,’ measured by thetotal length of production RHSs. A comparison with our Bayesian criterion highlights the considerableconceptual and practical simplification gained from using probabilities as the universal ‘currency’ of theevaluation metric.The present approach was derived as a minimal extension of the HMM merging approach toSCFGs (see Section 3.5 for origins of the state merging concept). As such, it is also related to variousinduction methods for non-probabilistic CFGs that rely on structured (parse-tree skeleton) samples to formtree equivalence classes that correspond to the nonterminals in a CFG (Fass 1983; Sakakibara 1990). As wehave seen, merging alone is sufficient as an induction operator if fully bracketed samples are provided.4.4.3 Cook’s Grammatical Inference by Hill ClimbingCook et al. (1976) present a hill-climbing search procedure for SCFGs that shares many of thefeatures and ideas of ours. Among these is the best-first approach, and an evaluation metric that aims tobalance ‘complexity’ of the grammar against ‘discrepancy’ relative to the target distribution. A crucialdifference is that only the relative frequencies of the samples, serving as an approximation to the true targetdistribution, are used.Discrepancy of grammar and samples is evaluated by a metric that combines elements of the standardrelative entropy with an ad-hoc measure of string complexity. Complexity of the grammar is likewise measuredby a mix of rule entropy and rule complexity. 10 Discrepancy and complexity are then combined in a weightedsum, where the weighting factor is set empirically (although the induction procedure is apparently quite robustwith respect to the exact value of this parameter).To see the conceptual difference to the Bayesian approach, consider the introductory example fromSection 4.3.2. The four samples , `Z0 observed with relative frequencies (10, 5, 2,1) are good evidence for a generalization to the target grammar that Z¢ 0 generates . However, ifthe same samples were observed with hundred-fold frequencies (1000, 500, 200, 100), then the hypothesis0 should become rather unlikely (in the absence of any additional samples, such as 5 5 , 6 6 ,etc.) Indeed our Bayesian learner will refrain from this generalization, due to the 100-fold increased log¢9 Thanks to Eugene Charniak for pointing out this reference, which seems to be less well-known and accessible than it deserves.10 The exact rationale for these measures is not entirely clear, as the complexities of strings are determined independently of theunderlying model, which is inconsistent with the standard information theoretic (and MDL) approach.