Nonparametric Bayesian Discrete Latent Variable Models for ...

More documents

Recommendations

Info

3 Dirichlet Process Mixture Models The no gaps algorithm augments the state to have N components, some of which will be empty. The values of φ1, . . . , φN are not altered during indicator variable updates. Only group labels are changed if needed and the component parameters are updated only after updating the indicators of all data points. Note that although theoretically the state includes N components, we only need to represent K ‡ − + 1 of them since the rest of the components are not considered when updating ci. We can include new components in the representation by sampling new parameter values from the prior as needed. Gibbs Sampling Using Auxiliary Variables (Neal, 2000) Neal (2000) presents another Gibbs sampling algorithm that uses auxiliary variables that only exist temporarily, instead of augmenting the state to include empty components. The state of the Markov chain consists of the indicator variables c1, . . . , cN and the active component parameters φc. We repeatedly update the indicator variables and the component parameters. The component parameters can be updated by simply sampling from their posterior. When we update the indicator variable for a particular data point, ζ ≥ 1 auxiliary components will be used to avoid the intractable integral. Treating xi as the last data point, we can either assign ci to be equal to one of the active components, or create a new component. The auxiliary components will represent the possible new components. Using the same notation as above, there are K ‡ − active components associated with the data points other than i. The prior probability of assigning ci to be equal to one of the active components is n−i,k/(N − 1 + α) and the probability of creating a new component is α/(N − 1 + α), which will be split among the ζ auxiliary components. If ci is a singleton, then we assign the parameter of one of the auxiliary variable to be φci , and draw values for the parameters of the rest of the auxiliary variables from G0. Otherwise, if ci belongs to a component that is also associated with other data points, then we sample parameters of all auxiliary components form G0. We update ci using the following probabilities: P (ci = k | xi, c−i, φ) ∝ P (ci = k | xi, c−i, φ) ∝ n−i,k N − 1 + α F (xi | φk) for k = 1, . . . , K ‡ − , α/ζ N − 1 + α F (xi | φk) for k = (K ‡ − + 1), . . . , (K‡ − + ζ). (3.34) Note the differences between this method and the no gaps algorithm. Here, we have used k = 1, . . . , K ‡ − as the labels for the active components, and k = K‡ − +1, . . . , K‡ − +ζ for the auxiliary components only for notational convenience. On the other hand, the values of the labels were significant for the no gaps algorithm of the previous section. Furthermore, the prior probability of creating a new component is K ‡ − +1 greater for this algorithm. For large values of ζ, the auxiliary variables can be thought of approximating G0 and giving an approximation of the integral in eq. (3.29). But the algorithm is valid for any ζ ≥ 1. ζ can be chosen as a compromise between computational cost and mixing performance. 30
3.2 MCMC Inference in Dirichlet Process Mixture Models Algorithm 5 Gibbs sampling for non-conjugate DPM models using auxiliary components The state of the Markov chain consists of the indicator variables c = {c1, . . . , cN} and the parameters of the active components Φ = {φ1, . . . , φ K ‡} Repeatedly sample: for all i = 1, . . . , N do {indicator updates} if ci is a singleton then Assign φci to be the parameter of one of the auxiliary components Draw values from G0 for the rest of the auxiliary parameters else Draw values from G0 for all the ζ auxiliary parameters end if Update ci using eq. (3.34) Discard the inactive components end for for all k = 1, . . . , K ‡ do {parameter updates} Update φk by sampling from its posterior, eq. (3.31) end for Metropolis-Hastings Updates Neal (2000) proposes to use Metropolis-Hastings updates combined with partial Gibbs updates. We can use a Metropolis Hastings algorithm using the conditional priors as the proposal distribution, leading to the acceptance probability to be the ratio of the likelihoods. That is, we propose ci to be equal to one of the existing components with probability n−i,k/(N − 1 + α) and propose to create a singleton with probability α/(N − 1 + α). The proposal is evaluated with the ratio of the likelihoods. The probability of considering to create a new component would be very low if α is small relative to the number of data points N. Therefore Neal (2000) changes the proposal distribution so as to increase the probability of considering to propose forming a new component. And, to facilitate mixing, he adds partial Gibbs sampling steps for the members of the non-singleton components in which only changing ci to one of the existing components is considered. In detail, if the data point i belongs to a singleton, it will be considered to be assigned to only one of the existing components with assignment probability n−i,k/(N − 1). The acceptance ratio of this proposal is min 1, α N − 1 F (xi | φc ∗ i ) F (xi | φci ) . (3.35) And, whenever ci is not a singleton, proposing to change it to a newly created component will be proposed with probability 1, and a parameter for this component will be sampled 31
Page 1: Nonparametric Bayesian Discrete Lat
Page 4 and 5: Matrizen mit unendlich vielen Spalt
Page 7 and 8: Contents Zusammenfassung iii Abstra
Page 9: List of Algorithms 1 Gibbs sampling
Page 13 and 14: Notation Matrices are capitalized a
Page 15: Symbol Meaning IBP Z binary latent
Page 18 and 19: 1 Introduction belief in the prior.
Page 20 and 21: 2 Nonparametric Bayesian Analysis b
Page 22 and 23: 2 Nonparametric Bayesian Analysis s
Page 25 and 26: 3 Dirichlet Process Mixture Models
Page 27 and 28: 3.1 The Dirichlet Process the perfo
Page 29 and 30: α G o G θi x i N 3.1 The Dirichle
Page 31 and 32: 15 10 5 −0.5 0 0.5 2 1 G 0 0 −0
Page 33 and 34: increment process with the correspo
Page 35 and 36: α G o π k c i θk x i 8 N 3.1 The
Page 37 and 38: 3.1 The Dirichlet Process Eq. (3.21
Page 39 and 40: number of components, K number of c
Page 41 and 42: 3.2 MCMC Inference in Dirichlet Pro
Page 43 and 44: and Bush and MacEachern (1996). 3.2
Page 45: 3.2.2 Algorithms for non-Conjugate
Page 55 and 56: ∗ π ∗ π s 3.2 MCMC Inference
Page 57 and 58: model can be written in the form of
Page 59 and 60: −1 µ y Σy Σy D ξ normal R 3.3
Page 61 and 62: the log likelihood term is: where a
Page 63 and 64: 3.3 Empirical Study on the Choice o
Page 65 and 66: autocovariance coefficient 1 0.8 0.
Page 67 and 68: # of data points # of data points 5
Page 69 and 70: 3.4 Dirichlet Process Mixtures of F
Page 71 and 72: µ y Σy ξ R 0 ν w normal µ −1
Page 73 and 74: ch1 ch2 ch3 ch4 3.4 Dirichlet Proce
Page 75 and 76: 3.4 Dirichlet Process Mixtures of F
Page 77 and 78: # of components # of components # o
Page 79 and 80: ch 2 ch 3 ch 1 ch 2 ch 3 ch 4 3.4 D
Page 81: 3.5 Discussion 3.5 Discussion In th
Page 84 and 85: 4 Indian Buffet Process Models matr
Page 86 and 87: 4 Indian Buffet Process Models In t
Page 88 and 89: 4 Indian Buffet Process Models α z
Page 90 and 91: 4 Indian Buffet Process Models α
Page 92 and 93: 4 Indian Buffet Process Models The
Page 94 and 95: 4 Indian Buffet Process Models colu
Page 96 and 97:
4 Indian Buffet Process Models Pois
Page 98 and 99:
4 Indian Buffet Process Models z α
Page 100 and 101:
4 Indian Buffet Process Models For
Page 102 and 103:
4 Indian Buffet Process Models ciat
Page 104 and 105:
4 Indian Buffet Process Models rati
Page 106 and 107:
4 Indian Buffet Process Models repr
Page 108 and 109:
4 Indian Buffet Process Models samp
Page 110 and 111:
4 Indian Buffet Process Models feat
Page 112 and 113:
4 Indian Buffet Process Models Algo
Page 114 and 115:
4 Indian Buffet Process Models mixi
Page 116 and 117:
4 Indian Buffet Process Models Figu
Page 118 and 119:
4 Indian Buffet Process Models pres
Page 120 and 121:
4 Indian Buffet Process Models ε
Page 122 and 123:
4 Indian Buffet Process Models LL P
Page 124 and 125:
4 Indian Buffet Process Models P+ P
Page 126 and 127:
4 Indian Buffet Process Models tEBA
Page 128 and 129:
4 Indian Buffet Process Models dist
Page 130 and 131:
5 Conclusions has been defined as a
Page 132 and 133:
A Details of Derivations for the St
Page 135 and 136:
B Mathematical Appendix B.1 Dirichl
Page 137 and 138:
p3 α 1 0.5 0 0 0.5 0.4 0.3 0.2 0.1
Page 139 and 140:
Construction of A Process B.4 Equal
Page 141 and 142:
Bibliography D. Aldous. Exchangeabi
Page 143 and 144:
Bibliography T. S. Ferguson. Prior
Page 145 and 146:
Bibliography L. F. James and J. W.
Page 147 and 148:
Bibliography R. M. Neal. Probabilis
Page 149 and 150:
Bibliography Y. W. Teh, M. I. Jorda
show all

Nonparametric Bayesian Discrete Latent Variable Models for ...

Create successful ePaper yourself

Delete template?

Save as template?