The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 7. -GRAMS FROM STOCHASTIC CONTEXT-FREE GRAMMARS 177Training corpus Test corpusNo. of sentences 2621 364No. of words 16974 2208Bigram vocabulary 1064Bigram coverage 100% 77%SCFG productions 1177SCFG vocabulary 655SCFG coverage 63% 51%Table 7.1: BeRP corpora and language model statistics.Coverage is measured by the percentage of sentences parsed with non-zero probability by a given languagemodel.expectations and 108959 linear systems for bigram expectations.SPARCstation 10 using a non-optimized Lisp implementation. 6The process takes about 9 hours on aThe experiments and results described below overlap with those reported in Jurafsky et al. (1994b).In experiment 1, the recognizer used bigrams that were estimated directly from the training corpus,without any smoothing, resulting in a word error rate of 33.7%.In experiment 2, a different set of bigram probabilities was used, computed from the context-freegrammar, whose probabilities had previously been estimated from the same training corpus, using standardEM techniques. This resulted in a word error rate of 32.9%. This may seem surprisingly good given thelow coverage of the underlying CFGs, but notice that the conversion into bigrams is bound to result in aless constraining language model, effectively increasing coverage. For comparison purposes we also ran thesame experiment with bigrams computed indirectly by Monte-Carlo sampling from the SCFG, using 200,000samples. The result was slightly worse (33.3%), confirming that the precise computation has an inherentadvantage, as it cannot omit words or constructions that the SCFG assigns very low probability.Finally, in experiment 3, the bigrams generated from the SCFG were augmented by those from theraw training data, in a proportion of 200,000 : 2500. We have not attempted to optimize this mixture proportion,e.g., by deleted interpolation (Jelinek & Mercer 1980). 7 With the bigram estimates thus obtained, the worderror rate dropped to 29.6%, which represents a statistically significant improvement over experiments 1and 2.Table 7.2 summarizes these figures and also adds two more points of comparison: a pure SCFGlanguage model and a mixture model that interpolates between bigram and SCFG. Notice that the latter caseis different from experiment 3, where the language model used is a standard bigram, albeit one that wasobtained by ‘mixing’ counts obtained both from the data and from the SCFG. The system referred to here, on6 One inefficiency is that the actual number of nonterminals (and hence the rank of the coefficient matrix) is 445, as the grammar isconverted to the Simple Normal Form introduced in Chapter 4.7 This proportion comes about because in the original system, predating the method described here, bigrams had to be estimated fromthe SCFG by random sampling. Generating 200,000 sentence samples was found to give good converging estimates for the bigrams.The bigrams from the raw training sentences were then simply added to the randomly generated ones. We later verified that the bigramsestimated from the SCFG were indeed identical to the ones computed directly using the method described here.