The dissertation of Andreas Stolcke is approved: University of ...

CHAPTER 5. PROBABILISTIC ATTRIBUTE GRAMMARS 109on adjacent parse tree nodes in a way similar to the belief propagation algorithm of Pearl (1988), conditioningon the top-level feature assignments and those given by the choice of SCFG rules. Once the conditionaldistribution of feature values on adjacent nodes is known, the likelihood is maximized by choosing a featureequation’s probability to be proportional to the number of times the two features had the same value.While this scheme is in principle capable of finding full PAGs of the kind shown above startingfrom randomly initialized feature probabilities, we found that it is also very prone to local maxima and highlydependent on the initial conditions of the parameters. As expected, the problem is observed increasingly in‘deep’ grammars with many intermediate of features, and hence hidden variables as part of a derivation.Thus, the problems with the parameter estimation approach to structural learning that were previouslyobserved with HMMs (Section 3.6.1) and SCFGs (Pereira & Schabes 1992) seem to show up even moreseverely in the PAG formalism. This gives further motivation to our investigation of merging-based learningapproaches.5.3 PAG MergingThe following sections describe the usual ingredients necessary to specify a merging-based learningalgorithm for the PAG formalism. Not surprisingly, we can build on the corresponding components of theSCFG merging algorithm.5.3.1 Sample incorporationThe first step in the learning algorithm is the creation of ad hoc rules that allow new samples to begenerated. We use newly created top-level productions to generate the sample strings as before, but augmentthese with feature value assignments to generate the observed attribute-value pairs. Thus, a sample(a circle is below a square, { tr = ‘circle’, lm = ‘square’, rel = ‘below’ })is incorporated by creating productionsS --> A1 CIRCLE1 IS1 BELOW1 A2 SQUARE1 (1)S.tr = ‘circle’ (1)S.lm = ‘square’ (1)S.rel = ‘below’ (1)A1 --> a (1)CIRCLE1 --> circle (1)IS1 --> is (1)BELOW1 --> below (1)A2 --> a (1)SQUARE1 --> square (1)We extend our earlier notation to associate counts with both productions and individual feature equations.Counts on feature equations indicate how often an equation has been used in feature derivations, and can beused to compute likelihoods, probability estimates, etc. (Section 3.4.3).