LDA

Latent Dirichlet Allocation

Motivation

LDA is an algorithm that improves upon

LSA and is suitable for

Let's assume we have a number of topics, each defined as distributions over words. A document is generated through the following process: First, we choose a distribution over the topics. Then, for each word position, we select a topic assignment and choose a word from that corresponding topic.

Method

For each of the topics, draw a multinomial distribution from a
Dirichlet distribution with parameter which controls the mean shape and sparsity of

For each of the documents, draw w a multinomial distribution from a
Dirichlet distribution with parameter which controls the mean shape and sparsity of

For each word position in a document

Select a latent topic from the multinomial distribution
Choose the observation from the multinomial distribution

and has parameters where is the size of the vocabulary across all documents.

Modeling

Posterior is impossible to compute so we approximate it

Approximation using
Gibbs sampling

Initialize probabilities randomly or uniformly

In each step, replace the value of one of the variables by a value drawn from the distribution of that variable conditioned on the values of the remaining variables

Repeat until convergence

Estimate the probability of assigning to each topic, conditioned on the topic assignments () of all other words (notation indicating the exclusion of )

From the above conditional distribution, sample a topic and set it as the new topic assignment of