Nonparametric Bayesian Discrete Latent Variable Models for ...

More documents

Recommendations

Info

4 Indian Buffet Process Models distribution on the binary matrix can be defined suggests generalizations of the IBP. In this chapter, we have summarized the different approaches for defining the distribution induced by the IBP. We have described the MCMC algorithms that have been developed for inference on the IBP models. We have compared the performance of the different samplers on a simple model to have an intuition about their performances. We demonstrated the use of the IBP as a prior on two models: A sparse factor analysis model with an over-complete basis for learning latent features of handwritten digits, and a choice model with infinitely many features describing the alternatives in the choice set. Given the performance of the proposed models, development of nonparametric models using IBP and generalizations for treating many other machine learning problems is an area for future research. There is much insight to gain from studying the connections between the IBP and related distributions, which would lead to further understanding of the theoretical and practical properties of the distribution. Development and improvement of inference algorithms on IBP models is also another direction for future research. 112
5 Conclusions The analysis of real-world problems demands a principled way of summarizing the data and representing the uncertainty in these summaries. Bayesian methods allow representing prior belief about the system generating the data and provides uncertainties about the predictions. The analysis of complex systems often requires flexible models that accurately capture the dependencies in the data. Simple parametric models can become inadequate for complex real-world problems. Nonparametric methods are powerful tools that allow definition of flexible models, since they can be used to approximate any of a wide class of distributions. The Dirichlet process (DP) is one of the most prominent random probability distributions. Although introduced in the 70s, its use have become popular only recently due to the development of sophisticated inference algorithms. The Indian buffet process (IBP) is a generalization of the related Chinese restaurant process which allows definition of more powerful models. In this thesis, we have presented empirical analysis of models using the DP and the IBP. The DP is defined by two parameters, a base distribution and a concentration parameter. The base distribution is a probability distribution, determining the mean of the DP. The concentration parameter is a positive scalar that can be seen as the strength of belief in the prior guess. Use of conjugate priors makes computations for inference much easier for Bayesian models in general. However, conjugate priors may fail to represent one’s prior belief. The trade off between the computational ease and modeling performance is an important modeling question. The Dirichlet process mixtures of Gaussians (DPMoG) model is one of the most widely used Dirichlet process mixture (DPM) models. We have empirically addressed this question in the DPMoG using conjugate and conditionally conjugate base distributions. We have compared the modeling performance and the computational cost of the inference algorithms for both prior specifications. We have empirically shown that it is possible to increase computational efficiency by exploiting conditional conjugacy while obtaining better modeling performance than the fully conjugate model. DPM models can be seen as mixture models with infinitely many components. Although the number of components are not bounded, there is an inherent clustering property in DPM models. We formulated a nonparametric form of the mixtures of factor analyzers (MFA) model using the DP. The MFA models the data as a mixture of Gaussians with reduced parametrization. Hence, the Dirichlet process MFA (DPMFA) model allows modeling high dimensional data efficiently. We have exploited the clustering property of the DPMs for the task of spike sorting, that is, clustering of spike waveforms that arise from different neurons in an extracellular recording. The IBP, a distribution over sparse binary matrices with infinitely many columns, 113
Page 1:
Nonparametric Bayesian Discrete Lat
Page 4 and 5:
Matrizen mit unendlich vielen Spalt
Page 7 and 8:
Contents Zusammenfassung iii Abstra
Page 9:
List of Algorithms 1 Gibbs sampling
Page 13 and 14:
Notation Matrices are capitalized a
Page 15:
Symbol Meaning IBP Z binary latent
Page 18 and 19:
1 Introduction belief in the prior.
Page 20 and 21:
2 Nonparametric Bayesian Analysis b
Page 22 and 23:
2 Nonparametric Bayesian Analysis s
Page 25 and 26:
3 Dirichlet Process Mixture Models
Page 27 and 28:
3.1 The Dirichlet Process the perfo
Page 29 and 30:
α G o G θi x i N 3.1 The Dirichle
Page 31 and 32:
15 10 5 −0.5 0 0.5 2 1 G 0 0 −0
Page 33 and 34:
increment process with the correspo
Page 35 and 36:
α G o π k c i θk x i 8 N 3.1 The
Page 37 and 38:
3.1 The Dirichlet Process Eq. (3.21
Page 39 and 40:
number of components, K number of c
Page 41 and 42:
3.2 MCMC Inference in Dirichlet Pro
Page 43 and 44:
and Bush and MacEachern (1996). 3.2
Page 45 and 46:
3.2.2 Algorithms for non-Conjugate
Page 47 and 48:
Page 49 and 50:
Page 51 and 52:
Page 53 and 54:
Page 55 and 56:
∗ π ∗ π s 3.2 MCMC Inference
Page 57 and 58:
model can be written in the form of
Page 59 and 60:
−1 µ y Σy Σy D ξ normal R 3.3
Page 61 and 62:
the log likelihood term is: where a
Page 63 and 64:
3.3 Empirical Study on the Choice o
Page 65 and 66:
autocovariance coefficient 1 0.8 0.
Page 67 and 68:
# of data points # of data points 5
Page 69 and 70:
3.4 Dirichlet Process Mixtures of F
Page 71 and 72:
µ y Σy ξ R 0 ν w normal µ −1
Page 73 and 74:
ch1 ch2 ch3 ch4 3.4 Dirichlet Proce
Page 75 and 76:
3.4 Dirichlet Process Mixtures of F
Page 77 and 78: # of components # of components # o
Page 79 and 80: ch 2 ch 3 ch 1 ch 2 ch 3 ch 4 3.4 D
Page 81: 3.5 Discussion 3.5 Discussion In th
Page 84 and 85: 4 Indian Buffet Process Models matr
Page 86 and 87: 4 Indian Buffet Process Models In t
Page 88 and 89: 4 Indian Buffet Process Models α z
Page 90 and 91: 4 Indian Buffet Process Models α
Page 92 and 93: 4 Indian Buffet Process Models The
Page 94 and 95: 4 Indian Buffet Process Models colu
Page 96 and 97: 4 Indian Buffet Process Models Pois
Page 98 and 99: 4 Indian Buffet Process Models z α
Page 100 and 101: 4 Indian Buffet Process Models For
Page 102 and 103: 4 Indian Buffet Process Models ciat
Page 104 and 105: 4 Indian Buffet Process Models rati
Page 106 and 107: 4 Indian Buffet Process Models repr
Page 108 and 109: 4 Indian Buffet Process Models samp
Page 110 and 111: 4 Indian Buffet Process Models feat
Page 112 and 113: 4 Indian Buffet Process Models Algo
Page 114 and 115: 4 Indian Buffet Process Models mixi
Page 116 and 117: 4 Indian Buffet Process Models Figu
Page 118 and 119: 4 Indian Buffet Process Models pres
Page 120 and 121: 4 Indian Buffet Process Models ε
Page 122 and 123: 4 Indian Buffet Process Models LL P
Page 124 and 125: 4 Indian Buffet Process Models P+ P
Page 126 and 127: 4 Indian Buffet Process Models tEBA
Page 130 and 131: 5 Conclusions has been defined as a
Page 132 and 133: A Details of Derivations for the St
Page 135 and 136: B Mathematical Appendix B.1 Dirichl
Page 137 and 138: p3 α 1 0.5 0 0 0.5 0.4 0.3 0.2 0.1
Page 139 and 140: Construction of A Process B.4 Equal
Page 141 and 142: Bibliography D. Aldous. Exchangeabi
Page 143 and 144: Bibliography T. S. Ferguson. Prior
Page 145 and 146: Bibliography L. F. James and J. W.
Page 147 and 148: Bibliography R. M. Neal. Probabilis
Page 149 and 150: Bibliography Y. W. Teh, M. I. Jorda
show all

Nonparametric Bayesian Discrete Latent Variable Models for ...

Create successful ePaper yourself

Delete template?

Save as template?