The dissertation of Andreas Stolcke is approved: University of ...

¸ > ) ¸are all productions with LHS ) , the entry indexed by ) in the right-hand side vector would be +9, ) ¸¸ )¸ ˜.CHAPTER 7. -GRAMS FROM STOCHASTIC CONTEXT-FREE GRAMMARS 179The algorithm has been found practical in the context of a medium-scale speech understandingsystem, were it gave improved estimates for a bigram language model, based on a hand-written SCFG andvery small amounts of available training data.¢ Deriving -gram probabilities from more sophisticated language models appears to be a generallyuseful technique which can both improve upon direct estimation ¢ of -grams, and allows available higherlevellinguistic knowledge to be effectively integrated into speech decoding or other tasks that place strongconstraints on usable language models.7.8 Appendix: Related ProblemsWe have seen ¢ how -gram expectations for SCFGs can be obtained by solving linear systemsbased on the matrix L I A, where I is the identity matrix and A is the first-moment (expectation) matrix ofnonterminal occurrences for a single nonterminal expansion. As it turns out, a number of apparently unrelatedproblems arising in connection with SCFGs and other probabilistic grammars have solutions based on thissame matrix. These are briefly surveyed below, without detailed proofs.7.8.1 Expected string lengthTo compute the expected number of terminals in a string, the IL system A is solved for the right-handside vector containing the average number of terminals generated in a single production, for each nonterminal.For example, ifThe solution vector contains the expected lengths for the sublanguages generated by each of the2 N+9,) ¸ 0; 1.>!0Œ¾nonterminals. Thus, the expected sentence string length is the9-entry in the solution vector.The problems and its solution are easily generalized to obtain the expected number of terminals ofa particular type occuring in a string (Booth & Thompson 1973).7.8.2 Derivation entropyThe derivation entropy is the average number of bits required to specify a derivation from a SCFG.Is is computed from a right-hand side vector that contains the average negative log probabilities for theproductions of each LHS nonterminal. For example, based on