The dissertation of Andreas Stolcke is approved: University of ...

)g£?CHAPTER 6. EFFICIENT PARSING WITH STOCHASTIC CONTEXT-FREE GRAMMARS 155For robust parsing, we want to identify all nonterminals that can possibly generate any substring of the input.This can be accomplished by also placing dummy states¸ £ ) for all positions"and nonterminals ) , in the Earley chart prior to the start of normal operation. (In practice,":6dummy states need to be added only for those )nonterminalsinput symbol. This technique is discussed in Section 6.6.3.2.)whose expansion can start with the currentThe immediate effect of these extra states is that more predictions will be generated, from whichmore completions follow, etc. After finishing the processing of the th state set, the chart will contain statesindicating that )nonterminal 1.Table 6.3(a) illustrates the robust parsing process using the example grammar from Table 6.1(p. 128).¸generates the substring 16å…å…åd:6Probabilities in the extra states are handled as follows. The initial dummy states6¸ £ ) areinitialized with a forward probability of zero. This will ensure that the forward probabilities of all extra statesremain at zero and don’t interfere with the computation of prefix probabilities from the regular Earley states.Inner probabilities on dummy states are initialized to unity just as the9 for start state, and processedin the usual way. The inner probabilities for the each substring/nonterminal pair can then be read off of thecomplete dummy states.Viterbi probabilities and Viterbi back-pointers can also be processed unchanged. Applying theViterbi-parse procedure from Section 6.5.1 to the complete dummy states yields Viterbi parses for allsubstring/nonterminal pairs.6.5.4.2 Assembling partial parsesInstead of consulting the chart for individual substring/nonterminal pairs it may be useful to obtaina list of all complete partial parses of the input. A complete partial parse is a sequence of nonterminals thattogether generate the input. For example, using the grammar in Table 6.1, the input a circle touches abovea square has the complete partial parses ‘NP VT PP’ and ‘Det N VT P NP’, among others. The input isgrammatical if9 exactly is among the complete partial parses.First note that there may not exist a complete partial parse if the input contains unknown symbols.As a preprocessing step, or on-line during parsing, one may have to create new preterminals to account forsuch new input symbols.The Earley algorithm can be minimally extended to also generate the list of all partial parses. Whatis needed is some device that assembles abutting nonterminals from partial parses left-to-right. This workcan be carried out as a by-product of the normal completion process using the concept of a variable state. Avariable state is a special kind of dummy state in which the RHS can have any number of nonterminals to the