Nonparametric Bayesian Discrete Latent Variable Models for ...

More documents

Recommendations

Info

4 Indian Buffet Process Models Poisson(α) nonzero entries, and the distribution of the total nonzero entries in the matrix follows a Poisson(αN) distribution. The equivalence classes have been defined by using the lof (.) function. The ordering of the columns is not important for the lof representation, however the stick-breaking representation has a particular ordering. The columns are sorted with decreasing feature presence probabilities µ (k). This representation allows the µ (k) being represented in computations rather than being integrated out. The construction of the feature presence probabilities is such that the rate of decay is exponential. Given a part of the matrix, this representation allows judgment about the distribution of entries of the rest of the matrix. In particular, after a column with a feature presence probability µ (k), the expected number of nonzero entries in the rest of the row has a Poisson(αµ (k)), and the rest of the matrix a Poisson(αµ (k)N) distribution. The stick-breaking construction for the IBP and for the DP has an interesting relation. For the IBP, the feature presence probabilities µ (k) correspond to the stick lengths that we have left after breaking a piece off the stick, and for the DP the mixing proportions πk correspond to the piece we discard. For this reason, the stick-breaking construction for the IBP has strictly decreasing weights and the DP has size-biased ordering of the weights. The direct correspondence to stick-breaking in DP implies that a range of techniques for and extensions to the DP can be adapted for the IBP. For example, we can generalize the IBP by replacing the Beta(α, 1) distribution on νk’s with other distributions. One possibility is a Pitman-Yor extension (Pitman and Yor, 1997) of the IBP, defined as νk ∼ Beta(α + kd, 1 − d) µ (k) = k l=1 νl (4.32) where d ∈ [0, 1) and α > −d. The Pitman-Yor IBP weights decrease in expectation as 1 − a O(k d ) power-law that has heavier tails for the distribution of the number features. This may be a better fit for some naturally occurring data which have a larger number of features with significant but small weights Goldwater et al. (2006). An example technique for the DP which we can adapt to the IBP is to truncate the stick-breaking construction after a certain number of break points and to perform inference in the reduced space. Ishwaran and James (2001) give a bound for the error introduced by the truncation in the DP case which can be used here as well. Thibaux and Jordan (2007) show the connection between the beta process and the IBP. This suggests that the beta process, which has been extensively used in survival analysis, can be used for defining binary latent feature models. Furthermore, they use the connection between the IBP and the beta process to develop a new algorithm to sample beta processes with size-biased ordering of the weights. Wolpert and Ickstadt (1998) show how to construct the beta process with strictly decreasing weights using the inverse Lévy measure. The stick-breaking construction is a neat way of describing this algorithm. A direct consequence of the stick-breaking construction and the relation of the IBP to 80
4.2 MCMC Sampling algorithms for IBLF models the beta process is that a draw from a beta process has the form ∞ (·) with k=1 µ (k)δθk µ (k) drawn form the stick breaking prior for the feature presence probabilities, and θ (k) is independent from µ (k) and is drawn from the base measure H. Generalizations of the stick-breaking constructions lead to generalizations of the beta process. The IBP parameter α affects the number of active features therefore updating this parameter would give more flexibility to the model. The likelihood for α can be derived from the joint distribution of the features given in Equation 34 of Griffiths and Ghahramani (2005) to be, p(Z | α) ∝ α K‡ exp − αHN , where K ‡ is the number of active components and N is the number of rows. We can put a gamma prior on α, α ∼ G(1, 1). Combining this likelihood with the prior, we get the posterior distribution for α p(α | Z) = G 1 + K ‡ , 1 + HN . (4.33) Since this posterior is of standard form, it can be sampled from easily. Note that the construction of IBP and related distributions described above concerns the prior for Z only. Using Z in a IBLF, we update it by sampling its entries using the full conditional distributions which involves conditioning on the data. In the following section, we describe methods for inference on IBLF models. 4.2 MCMC Sampling algorithms for IBLF models The previous section summarized several different approaches for defining the distribution induced by the Indian buffet process, which has strong connections to the Dirichlet Process. In this section, we describe MCMC methods for inference on the latent feature models that use the IBP as the nonparametric prior over the feature matrix, which we refer to as the infinite binary latent feature (IBLF) models. The general form of the models we will consider assumes the data generating process to be living in a latent space of possibly infinite dimension. Each data point xi is described by an infinite dimensional binary latent vector zi = zi,1:∞ that encodes the presence or absence of the infinitely many features characterized by the parameters Θ = θ1:∞, and possibly other parameters Φ that live in the observable space. The state of the binary latent variables determine which features are actively used, that is, the effective latent dimensionality. Equivalently, we refer to the vector zi as the binary latent feature vector, and Θ as the set of parameters of the features. We put an IBP prior on the feature presence matrix Z, which has the vectors zi as its rows. The parameters Θ and Φ are given standard parametric priors. The data distribution is assumed not to depend on the features that do not belong to any of the data points, that is, the zero columns of Z and the parameters θk that are associated with those columns. The model can be 81
Page 1:
Nonparametric Bayesian Discrete Lat
Page 4 and 5:
Matrizen mit unendlich vielen Spalt
Page 7 and 8:
Contents Zusammenfassung iii Abstra
Page 9:
List of Algorithms 1 Gibbs sampling
Page 13 and 14:
Notation Matrices are capitalized a
Page 15:
Symbol Meaning IBP Z binary latent
Page 18 and 19:
1 Introduction belief in the prior.
Page 20 and 21:
2 Nonparametric Bayesian Analysis b
Page 22 and 23:
2 Nonparametric Bayesian Analysis s
Page 25 and 26:
3 Dirichlet Process Mixture Models
Page 27 and 28:
3.1 The Dirichlet Process the perfo
Page 29 and 30:
α G o G θi x i N 3.1 The Dirichle
Page 31 and 32:
15 10 5 −0.5 0 0.5 2 1 G 0 0 −0
Page 33 and 34:
increment process with the correspo
Page 35 and 36:
α G o π k c i θk x i 8 N 3.1 The
Page 37 and 38:
3.1 The Dirichlet Process Eq. (3.21
Page 39 and 40:
number of components, K number of c
Page 41 and 42:
3.2 MCMC Inference in Dirichlet Pro
Page 43 and 44:
and Bush and MacEachern (1996). 3.2
Page 45 and 46: 3.2.2 Algorithms for non-Conjugate
Page 47 and 48: 3.2 MCMC Inference in Dirichlet Pro
Page 55 and 56: ∗ π ∗ π s 3.2 MCMC Inference
Page 57 and 58: model can be written in the form of
Page 59 and 60: −1 µ y Σy Σy D ξ normal R 3.3
Page 61 and 62: the log likelihood term is: where a
Page 63 and 64: 3.3 Empirical Study on the Choice o
Page 65 and 66: autocovariance coefficient 1 0.8 0.
Page 67 and 68: # of data points # of data points 5
Page 69 and 70: 3.4 Dirichlet Process Mixtures of F
Page 71 and 72: µ y Σy ξ R 0 ν w normal µ −1
Page 73 and 74: ch1 ch2 ch3 ch4 3.4 Dirichlet Proce
Page 75 and 76: 3.4 Dirichlet Process Mixtures of F
Page 77 and 78: # of components # of components # o
Page 79 and 80: ch 2 ch 3 ch 1 ch 2 ch 3 ch 4 3.4 D
Page 81: 3.5 Discussion 3.5 Discussion In th
Page 84 and 85: 4 Indian Buffet Process Models matr
Page 86 and 87: 4 Indian Buffet Process Models In t
Page 88 and 89: 4 Indian Buffet Process Models α z
Page 90 and 91: 4 Indian Buffet Process Models α
Page 92 and 93: 4 Indian Buffet Process Models The
Page 94 and 95: 4 Indian Buffet Process Models colu
Page 98 and 99: 4 Indian Buffet Process Models z α
Page 100 and 101: 4 Indian Buffet Process Models For
Page 102 and 103: 4 Indian Buffet Process Models ciat
Page 104 and 105: 4 Indian Buffet Process Models rati
Page 106 and 107: 4 Indian Buffet Process Models repr
Page 108 and 109: 4 Indian Buffet Process Models samp
Page 110 and 111: 4 Indian Buffet Process Models feat
Page 112 and 113: 4 Indian Buffet Process Models Algo
Page 114 and 115: 4 Indian Buffet Process Models mixi
Page 116 and 117: 4 Indian Buffet Process Models Figu
Page 118 and 119: 4 Indian Buffet Process Models pres
Page 120 and 121: 4 Indian Buffet Process Models ε
Page 122 and 123: 4 Indian Buffet Process Models LL P
Page 124 and 125: 4 Indian Buffet Process Models P+ P
Page 126 and 127: 4 Indian Buffet Process Models tEBA
Page 128 and 129: 4 Indian Buffet Process Models dist
Page 130 and 131: 5 Conclusions has been defined as a
Page 132 and 133: A Details of Derivations for the St
Page 135 and 136: B Mathematical Appendix B.1 Dirichl
Page 137 and 138: p3 α 1 0.5 0 0 0.5 0.4 0.3 0.2 0.1
Page 139 and 140: Construction of A Process B.4 Equal
Page 141 and 142: Bibliography D. Aldous. Exchangeabi
Page 143 and 144: Bibliography T. S. Ferguson. Prior
Page 145 and 146: Bibliography L. F. James and J. W.
Page 147 and 148:
Bibliography R. M. Neal. Probabilis
Page 149 and 150:
Bibliography Y. W. Teh, M. I. Jorda
show all

Nonparametric Bayesian Discrete Latent Variable Models for ...

Create successful ePaper yourself

Delete template?

Save as template?