Nonparametric Bayesian Discrete Latent Variable Models for ...

More documents

Recommendations

Info

3 Dirichlet Process Mixture Models the data comes from the base distribution, (Walker et al., 1999). The DP has been introduced as a random probability measure, and its use as a prior for modeling the uncertainty in the functional form of the distribution of parameters in a model has been motivated. In this case, larger α values pronounces stronger belief in the base distribution. The derivation of the DPM as the limit of a finite mixture model shows that it can be used as a mixture model with intrinsic selection of number of components. In this case, the concentration parameter does not necessarily need to be considered as the prior belief about the distribution of the parameters. It can be regarded as a prior guess for the number of active components in the infinite mixture model. The likelihood for α may be derived by taking the limit K → ∞ of the joint distribution for the indicator variables: P (c1, . . . , cn|α) ∝ α K‡ n i=1 1 i − 1 + α αK‡ Γ(α) = , (3.22) Γ(n + α) where K ‡ denotes the number of active components. It is interesting that this expression depends only on the number of active components and the total number of data points, and not on how the observations are distributed among the components (unlike the finite mixture model case). To allow for flexibility in the model, we need to specify a prior for α. For the simulations in this work, we chose the prior for α −1 to have gamma shape with unit mean and degree of freedom 1: P (α −1 ) ∼ G(1/2, 1/2) =⇒ P (α) ∝ α −3/2 exp(−1/2α). (3.23) This prior is asymmetric, having a short tail for small values of α, expressing our prior belief, that we don’t expect a very small number of active classes (say K ‡ = 1). Figure 3.7 shows the prior distribution on the number of components for different prior specifications on α. This figure is inspired from (Navarro et al., 2006). The base distribution of the DP is the prior guess of the distribution of the parameters in a model. This corresponds to the distribution of the component parameters in an infinite mixture model. The shape of the priors is governed by a trade-off between modelling properties and computational considerations. In particular the use of conjugate priors makes the analysis in DPM models much more tractable, as we will see in the following section. However, the computational convenience is not a justification for using a certain type of prior that fails to represent prior knowledge. There has been a great deal of research in developing inference methods that do not require conjugacy. We describe MCMC methods for both cases in the following section, and in Section 3.3, we present an empirical study for comparing conjugate and conditionally conjugate models. It is often difficult to directly assign the base distribution with high confidence. Thus it is customary to utilize hierarchical priors to allow sufficient flexibility of the priors when they are of parametric form (West et al., 1994; Escobar and West, 1995; Ras- 22
number of components, K number of components, K 20 15 10 5 Gamma(0.5, 0.5) 3.1 The Dirichlet Process 0 0 5 10 15 20 25 number of data points, n InvGamma(0.5, 0.5) 25 20 15 10 5 0 0 5 10 15 20 25 number of data points, n Figure 3.7: The effect of the distribution of the concentration parameter α on the prior distribution for the number of components. The plots show the change in the expected number of mixture components with increasing number of data points. The area of the squares are proportional to the probability of the corresponding number of components for a given number of data points. Note that the gamma prior favours less number of components. mussen, 2000). In the hierarchical scheme, the priors linking the parameters of the component mixtures can all be parameterized using hyperparameters, which are themselves given vague priors. Alternatively, classical nonparametric priors such as kernel density estimation (KDE) can be used (McAuliffe et al., 2006). In this work, we consider the hierarchical model formulation. 23
Page 1: Nonparametric Bayesian Discrete Lat
Page 4 and 5: Matrizen mit unendlich vielen Spalt
Page 7 and 8: Contents Zusammenfassung iii Abstra
Page 9: List of Algorithms 1 Gibbs sampling
Page 13 and 14: Notation Matrices are capitalized a
Page 15: Symbol Meaning IBP Z binary latent
Page 18 and 19: 1 Introduction belief in the prior.
Page 20 and 21: 2 Nonparametric Bayesian Analysis b
Page 22 and 23: 2 Nonparametric Bayesian Analysis s
Page 25 and 26: 3 Dirichlet Process Mixture Models
Page 27 and 28: 3.1 The Dirichlet Process the perfo
Page 29 and 30: α G o G θi x i N 3.1 The Dirichle
Page 31 and 32: 15 10 5 −0.5 0 0.5 2 1 G 0 0 −0
Page 33 and 34: increment process with the correspo
Page 35 and 36: α G o π k c i θk x i 8 N 3.1 The
Page 37: 3.1 The Dirichlet Process Eq. (3.21
Page 41 and 42: 3.2 MCMC Inference in Dirichlet Pro
Page 43 and 44: and Bush and MacEachern (1996). 3.2
Page 45 and 46: 3.2.2 Algorithms for non-Conjugate
Page 55 and 56: ∗ π ∗ π s 3.2 MCMC Inference
Page 57 and 58: model can be written in the form of
Page 59 and 60: −1 µ y Σy Σy D ξ normal R 3.3
Page 61 and 62: the log likelihood term is: where a
Page 63 and 64: 3.3 Empirical Study on the Choice o
Page 65 and 66: autocovariance coefficient 1 0.8 0.
Page 67 and 68: # of data points # of data points 5
Page 69 and 70: 3.4 Dirichlet Process Mixtures of F
Page 71 and 72: µ y Σy ξ R 0 ν w normal µ −1
Page 73 and 74: ch1 ch2 ch3 ch4 3.4 Dirichlet Proce
Page 75 and 76: 3.4 Dirichlet Process Mixtures of F
Page 77 and 78: # of components # of components # o
Page 79 and 80: ch 2 ch 3 ch 1 ch 2 ch 3 ch 4 3.4 D
Page 81: 3.5 Discussion 3.5 Discussion In th
Page 84 and 85: 4 Indian Buffet Process Models matr
Page 86 and 87: 4 Indian Buffet Process Models In t
Page 88 and 89:
4 Indian Buffet Process Models α z
Page 90 and 91:
4 Indian Buffet Process Models α
Page 92 and 93:
4 Indian Buffet Process Models The
Page 94 and 95:
4 Indian Buffet Process Models colu
Page 96 and 97:
4 Indian Buffet Process Models Pois
Page 98 and 99:
4 Indian Buffet Process Models z α
Page 100 and 101:
4 Indian Buffet Process Models For
Page 102 and 103:
4 Indian Buffet Process Models ciat
Page 104 and 105:
4 Indian Buffet Process Models rati
Page 106 and 107:
4 Indian Buffet Process Models repr
Page 108 and 109:
4 Indian Buffet Process Models samp
Page 110 and 111:
4 Indian Buffet Process Models feat
Page 112 and 113:
4 Indian Buffet Process Models Algo
Page 114 and 115:
4 Indian Buffet Process Models mixi
Page 116 and 117:
4 Indian Buffet Process Models Figu
Page 118 and 119:
4 Indian Buffet Process Models pres
Page 120 and 121:
4 Indian Buffet Process Models ε
Page 122 and 123:
4 Indian Buffet Process Models LL P
Page 124 and 125:
4 Indian Buffet Process Models P+ P
Page 126 and 127:
4 Indian Buffet Process Models tEBA
Page 128 and 129:
4 Indian Buffet Process Models dist
Page 130 and 131:
5 Conclusions has been defined as a
Page 132 and 133:
A Details of Derivations for the St
Page 135 and 136:
B Mathematical Appendix B.1 Dirichl
Page 137 and 138:
p3 α 1 0.5 0 0 0.5 0.4 0.3 0.2 0.1
Page 139 and 140:
Construction of A Process B.4 Equal
Page 141 and 142:
Bibliography D. Aldous. Exchangeabi
Page 143 and 144:
Bibliography T. S. Ferguson. Prior
Page 145 and 146:
Bibliography L. F. James and J. W.
Page 147 and 148:
Bibliography R. M. Neal. Probabilis
Page 149 and 150:
Bibliography Y. W. Teh, M. I. Jorda
show all

Nonparametric Bayesian Discrete Latent Variable Models for ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?