Nonparametric Bayesian Discrete Latent Variable Models for ...

More documents

Recommendations

Info

3 Dirichlet Process Mixture Models That is, the data points xi are i.i.d. with distribution function F (xi | θi). The parameters θi specifying the data distribution are i.i.d. draws from G which is a random distribution function with a DP prior. This defines the basic DPM model. Although the DP theory had been formally developed by Ferguson (1973), its use has become popular after the introduction of Markov chain Monte Carlo (MCMC) methods for inference. The first MCMC algorithm for DP models was introduced in the unpublished work of Escobar (1988), later published in Escobar (1994). Extensions and refinements of this method can be found in (Escobar and West, 1995; MacEachern, 1994; West et al., 1994; Bush and MacEachern, 1996). Construction of more flexible models using the DP became possible with the development of methods for the nonconjugate DPM models such as the methods in (MacEachern and Müller, 1998; Walker and Damien, 1998; Neal, 2000; Green and Richardson, 2001). These methods use a representation where the DP is integrated out. There are methods that consider approximations to DP instead of integrating it out, see for example (Muliere and Tardella, 1998; Ishwaran and James, 2002; Kottas and Gelfand, 2001). Recently, methods that use the explicit representation of the DP without having to use approximation have been developed by Papaspiliopoulos and Roberts (2005) and Walker (2006). Other approximate inference techniques used in algorithms for DPM models include sequential importance sampling (Liu, 1996; Quintana, 1998; MacEachern et al., 1999; Ishwaran and Takahara, 2002; Fearnhead, 2004), predictive recursion (Newton and Zhang, 1999), expectation propagation (Minka and Ghahramani, 2003), and variational approximation (Blei and Jordan, 2005; Kurihara et al., 2007a,b). Some theoretical results regarding the posterior consistency can be found in (Diaconis and Freedman, 1986; Ghosal et al., 1999). Convergence rates have been studied by (Ghosal et al., 2000; Shen and Wasserman, 2001; Walker et al., 2006). The DPs are getting increasingly popular in machine learning. There have been several models and model extensions using the DP. Some of these include (Rasmussen, 2000; Rasmussen and Ghahramani, 2002; Beal et al., 2003; Blei et al., 2004; Dubey et al., 2004; Xing et al., 2004; Zhang et al., 2005; Daumé III and Marcu, 2005; Teh et al., 2006; Xing et al., 2006; Meeds and Osindero, 2006; Sudderth et al., 2006; Sohn and Xing, 2007; Beal and Krishnamurthy, 2006; Xue et al., 2007). The structure of this chapter is as follows. We summarize different approaches for defining the DP distribution, and give some of the properties of the DP important for developing models and inference techniques in Section 3.1. In Section 3.2 we describe some of the MCMC algorithms to give an overview of the development of methods for inference on the DPM models. Inference techniques for the DPM models with a conjugate base distribution are relatively easier to implement than for the non-conjugate case. An important question is whether the modeling performance is weakened by using a conjugate base distribution instead of a more flexible distribution. And, a related interesting question is whether the inference is computationally cheaper for the conjugate DPM models. We address these questions in Section 3.3 using one of the most widely used DP model for density estimation and clustering: DP mixtures of Gaussians (DPMoG). The DPMoG model with both conjugate and conditionally conjugate base distributions have been used extensively in applications of the DPM models. However, 10
3.1 The Dirichlet Process the performance of the models using these different prior specifications have not been compared. We present an empirical study on the choice of the base distribution for the DPMoG model. We compare the computational cost and modeling performance of using conjugate and conditionally conjugate base distributions. When the data is believed to have a lower dimensional latent structure, it is possible to incorporate this prior knowledge to the model structure using a mixture of factor analyzers (MFA) model. In Section 3.4 we formulate the Dirichlet process mixture of factor analyzers (DPMFA) model and present experimental results on a challenging clustering problem, known as spike sorting. We conclude this chapter with a discussion in Section 3.5. 3.1 The Dirichlet Process Let X be a space and A be a σ-field of subsets of X . A stochastic process G on (X , A) is said to be a Dirichlet process (DP) with parameters α and G0 if for any partition (A1, . . . , Ak), on the space of support of G0, the random vector (G(A1), . . . , G(Ak)) has a k-dimensional Dirichlet distribution 1 with parameter (αG0(A1), . . . , αG0(Ak)), that is: (G(A1), . . . , G(Ak)) ∼ D((αG0(A1), . . . , αG0(Ak))). (3.2) We denote the random probability measure G that has a DP distribution with concentration parameter α and base distribution G0 by: G ∼ DP (α, G0). (3.3) Ferguson (1973) establishes the existence of the DP by verifying the Kolmogorov consistency conditions, appendix B.3. Some authors define the DP using a single parameter by combining the two parameters to form the random measure α = αG0. Denoting the space of support of G0 as X , the mass of the random measure would be given as α = α(X ), and the base distribution as G0(·) = α(·) α(X ) . In the following, we use the two-parameter notation to denote the random distribution G with a DP prior, eq. (3.3). Some important properties of the DP are as follows: • The mean of the process is the base distribution, E{G} = G0. Thus, G0 can be thought of as the prior guess of the shape of the distribution of G. • Given samples θ1, . . . , θn from G, the posterior distribution is also a DP: G|θ1, . . . , θn ∼ DP α + n, αG0 + n α + n i=1 δθi (·) . (3.4) Note that the concentration parameter becomes α + n after observing n samples, and the contribution of the prior base distribution G0 is scaled by α. Thus, the 1 The definition of the Dirichlet distribution and some of its properties necessary for understanding the DP are given in Appendix B.1. 11
Page 1: Nonparametric Bayesian Discrete Lat
Page 4 and 5: Matrizen mit unendlich vielen Spalt
Page 7 and 8: Contents Zusammenfassung iii Abstra
Page 9: List of Algorithms 1 Gibbs sampling
Page 13 and 14: Notation Matrices are capitalized a
Page 15: Symbol Meaning IBP Z binary latent
Page 18 and 19: 1 Introduction belief in the prior.
Page 20 and 21: 2 Nonparametric Bayesian Analysis b
Page 22 and 23: 2 Nonparametric Bayesian Analysis s
Page 25: 3 Dirichlet Process Mixture Models
Page 29 and 30: α G o G θi x i N 3.1 The Dirichle
Page 31 and 32: 15 10 5 −0.5 0 0.5 2 1 G 0 0 −0
Page 33 and 34: increment process with the correspo
Page 35 and 36: α G o π k c i θk x i 8 N 3.1 The
Page 37 and 38: 3.1 The Dirichlet Process Eq. (3.21
Page 39 and 40: number of components, K number of c
Page 41 and 42: 3.2 MCMC Inference in Dirichlet Pro
Page 43 and 44: and Bush and MacEachern (1996). 3.2
Page 45 and 46: 3.2.2 Algorithms for non-Conjugate
Page 55 and 56: ∗ π ∗ π s 3.2 MCMC Inference
Page 57 and 58: model can be written in the form of
Page 59 and 60: −1 µ y Σy Σy D ξ normal R 3.3
Page 61 and 62: the log likelihood term is: where a
Page 63 and 64: 3.3 Empirical Study on the Choice o
Page 65 and 66: autocovariance coefficient 1 0.8 0.
Page 67 and 68: # of data points # of data points 5
Page 69 and 70: 3.4 Dirichlet Process Mixtures of F
Page 71 and 72: µ y Σy ξ R 0 ν w normal µ −1
Page 73 and 74: ch1 ch2 ch3 ch4 3.4 Dirichlet Proce
Page 75 and 76: 3.4 Dirichlet Process Mixtures of F
Page 77 and 78:
# of components # of components # o
Page 79 and 80:
ch 2 ch 3 ch 1 ch 2 ch 3 ch 4 3.4 D
Page 81:
3.5 Discussion 3.5 Discussion In th
Page 84 and 85:
4 Indian Buffet Process Models matr
Page 86 and 87:
4 Indian Buffet Process Models In t
Page 88 and 89:
4 Indian Buffet Process Models α z
Page 90 and 91:
4 Indian Buffet Process Models α
Page 92 and 93:
4 Indian Buffet Process Models The
Page 94 and 95:
4 Indian Buffet Process Models colu
Page 96 and 97:
4 Indian Buffet Process Models Pois
Page 98 and 99:
4 Indian Buffet Process Models z α
Page 100 and 101:
4 Indian Buffet Process Models For
Page 102 and 103:
4 Indian Buffet Process Models ciat
Page 104 and 105:
4 Indian Buffet Process Models rati
Page 106 and 107:
4 Indian Buffet Process Models repr
Page 108 and 109:
4 Indian Buffet Process Models samp
Page 110 and 111:
4 Indian Buffet Process Models feat
Page 112 and 113:
4 Indian Buffet Process Models Algo
Page 114 and 115:
4 Indian Buffet Process Models mixi
Page 116 and 117:
4 Indian Buffet Process Models Figu
Page 118 and 119:
4 Indian Buffet Process Models pres
Page 120 and 121:
4 Indian Buffet Process Models ε
Page 122 and 123:
4 Indian Buffet Process Models LL P
Page 124 and 125:
4 Indian Buffet Process Models P+ P
Page 126 and 127:
4 Indian Buffet Process Models tEBA
Page 128 and 129:
4 Indian Buffet Process Models dist
Page 130 and 131:
5 Conclusions has been defined as a
Page 132 and 133:
A Details of Derivations for the St
Page 135 and 136:
B Mathematical Appendix B.1 Dirichl
Page 137 and 138:
p3 α 1 0.5 0 0 0.5 0.4 0.3 0.2 0.1
Page 139 and 140:
Construction of A Process B.4 Equal
Page 141 and 142:
Bibliography D. Aldous. Exchangeabi
Page 143 and 144:
Bibliography T. S. Ferguson. Prior
Page 145 and 146:
Bibliography L. F. James and J. W.
Page 147 and 148:
Bibliography R. M. Neal. Probabilis
Page 149 and 150:
Bibliography Y. W. Teh, M. I. Jorda
show all

Nonparametric Bayesian Discrete Latent Variable Models for ...

Create successful ePaper yourself

Delete template?

Save as template?