Web Mining and Social Networking: Techniques and ... - tud.ttu.ee

126 6 Web Usage MiningThere are two main problems with PLSA model: 1) it is hard to estimate probability of apreviously unseen document; 2) due to the linear growth in parameters that depend on thedocument itself, PLSA suffers from over-fitting and inappropriate generative semantics.To address these problems, Latent Dirichlet Allocation (LDA) is introduced by combiningthe basic generative model with a prior probability on topics to provide a complete generativemodel for documents. The basic idea of the LDA is that documents are modeled as randommixtures over latent topics with a probability distribution, where each topic is represented bya distribution over word vocabulary. In this sense, any random mixing distribution, T(z) isdetermined by an underlying distribution thereby representing uncertainty over a particularθ(·) as p k (θ(·)), where p k is defined over all θ ∈ P k , the set of all possible (k − l)-simplex.That is, the Dirichlet parameters determine the uncertainty, which models the random mixturedistribution over semantic topics.Latent Dirichlet Allocation (LDA)LDA is a generative probabilistic model of co-occurrence observations. As for a typical applicationin text analysis, here we utilize the document corpus as the co-occurrence observationsto conduct the following formulation. As discussed above, the basic idea of the LDA modelis that the documents within the corpus are represented as a random mixture over the latenttopics and each topic is characterized by a distribution over the words in the documents. Thegraphical illustration of the generative procedure of LDA is shown in Fig.6.4. LDA is performedvia a sequence of the document generation processes (shown in Fig.6.4 and algorithm1). The notations used and the generative procedure of LDA model are outlined as follows.Notations:• M: the number of documents• K: the number of topics• V: the size of vocabulary• α,β: Dirichlet parameters• θ m : the topic assignment of the document m• Θ = θ m ,m = 1,...,M: the topic estimations of the corpus, a M × K matrix• φ k : the word distribution of the topic k• Φ = φ k ,k = 1,...,K: the word assignments of the topics, a K ×V matrix• Dir and Poiss are Dirichlet and Poisson distribution functions respectivelyAlgorithm 6.6: Generation Process of LDAfor each of topicssample the mixture of words φ k ∼ Dir(β)endfor each of documents m = 1:Msample the mixture of topics θ m ∼ Dir(α)sample the lengths of documents N m ∼ Poiss(ξ)for each word n = 1:N m in the document msample the topic index of z m,n ∼ Mult(θ m )sample the weight of word w m,n ∼ Mult(φ zm,n )endend