The dissertation of Andreas Stolcke is approved: University of ...

Óz ?ts%Õ> ÖvuwCHAPTER 5. PROBABILISTIC ATTRIBUTE GRAMMARS 119theory of Markov networks (Pearl 1988) tells us that such a marginal probabilities cannot be assigned locallyin a consistent fashion.A partial solution to this particular problem is the imposition of a global total ordering amongagreement features, so that the entire system of constraints is again expressible as a product of conditionalprobabilities. An ordering based on the linguistic notion of phrase head might accomplish this: all RHSfeatures could then only depend on RHS features associated with the distinguished head constituent. In anycase, the convenient global bottom-up ordering of feature constraints would have to be given up, and theprobabilities of the – Ã (string) component of a derivation would no longer be independent of the featuralaspects of the grammar.Systems of unordered constraints can still be given a probabilistic interpretation using well-knownconcepts from statistical physics. Instead of directly defining probabilities for local feature assignments, wecould instead define an energy function that expresses the ‘badness’ of an assignment as an arbitrary positivenumber. The energy function can be defined by local components, namely as a product of local contributions,e.g., one for each rule. The above rule would thereby be translated into a term that gives low energy if andonly if NP.number = VP.number. The total energy b-,;1 0 of a complete feature assignment 1 is thenused to generate probabilities according to the Boltzmann distribution+-,210%6is the normalizing constant (integral over the numerator), andx is a parameter (the temperature)that controls the ‘peakedness’ of the distribution. 7where>This formulation is elegant and intuitively appealing (although it obviates the traditional conceptof derivation). However, it carries a heavy computational price: Simply obtaining the probability of a givenstring/feature assignment pair for various alternative grammars generally requires stochastic simulationsin order to compute the constant>.The posterior probabilities of models can also be evaluated usingMonte-Carlo techniques (Neal 1993), but the approach seems too inefficient for the generate-and-test searchstrategies we have explored so far. On the other hand, the formulation also suggests investigating otherlearning paradigms, such as stochastic optimization via simulated annealing and Boltzmann machine learning(Geman & Geman 1984; Hinton & Sejnowski 1986).5.5.2 Hierarchical featuresIn Chapter 4 we saw that the merging algorithm is quite capable of inducing recursive syntacticrules. However, in the PAG formalism there is no matching recursive structure in the feature descriptions: it isconstrained to ‘flat’ feature structures. As a result, even the ¨ 0 semantics of sentences with simple embeddedrelative clausesa circle is below a square which is to the left of a triangle which ...7 all Aa"zyconfigurations are equally probable, regardless of their energy, whereas ata" 0 all probability mass is concentrateon the lowest energy configuration.