The dissertation of Andreas Stolcke is approved: University of ...

The forward and inner probabilities of the states thus created are those of the first state )First, the unit-production relationhas to be extended to allow for unit-production chains due to null productions. A rule ) ¸\¸ £=1Ó‘Œ\XE@ Ó‘Œ 6­¸CHAPTER 6. EFFICIENT PARSING WITH STOCHASTIC CONTEXT-FREE GRAMMARS 1426.4.7.2 Prediction with null productionsPrediction is mediated by the left-corner relation. For each )generate states for all= that are reachable from ) by way of the )occurring to the right of a dot, weThis reachability criterionhas to be extended in the presence of null productions. Specifically, if ) has a production)¸=1 £££+=@A?then=@is a left corner of ) iff=1£££=@A?of such a production to the left-corner probability +-, )1 all have a non-zero probabilityof expanding toA.The contribution£=@0 is¸relation.11=@.@A?+-,1=@.B0) ¸=1 £££+=@A?6XThe old prediction procedure can now be modified in two steps. First, replace the old relation+by the one that takes into account null productions, as sketched above. From the resulting compute thereflexive transitive closure + , and use it to generate predictions as before.©Second, when predicting a left corner= with a production=¸£££˜=@A?=1dot positions up to the first RHS nonterminal that cannot expand toA, say ) ¸from £££˜=@A?£=1) ¸=1 £££+=@A?1=@., add states for all1=@.through1£=@.. We will call this procedure ‘spontaneous dot shifting.’ It accounts precisely for thosederivations that expand the RHS prefix=1 £££+=@A?1 without consuming any of the input symbols.£££=@C?£=1d6X ¬1Ó‘Œ,1=@., multiplied by factors that account for the impliedA-expansions. This factor is just the productwhere is the dot position.6.4.7.3 Completion with null productionsModification of the completion step follows a similar pattern.if all other nonter-£££+=@A?1=@=@#=1 £££+=d 1 can effectively act like a unit production that ) and=@linksminals on the RHS can expand toA. Its contribution to the unit production relation ) ¸ =@ +9,be0 will thenFrom the resulting revised + matrix we compute the closure ©§ as usual.The second modification is another instance of spontaneous dot shifting. When completing a state, additional states have to be added, obtained by moving the˜) ¸ +-,£££=@A?1=@=@=1#1 £££+=df0dot further over any nonterminals in ˜ that have non-zeroA-expansion probability. As in prediction, forward) ¸.M£=§˜ and moving the dot to get ) ¸ .V=i£and inner probabilities are multiplied by the correspondingA-expansion probabilities.6.4.7.4 Eliminating null productionsGiven these added complications one might consider simply eliminating allA-productions in apreprocessing step. This is mostly straightforward and analogous to the corresponding procedure for non-