Topic modelling

Overview

Topic modelling finds the main topics in a set of texts. A review of NLP for student feedback lists it among the techniques for that task, next to feature extraction and feature selection 2301.08826 §2.

The same idea reappears in network models. A review of stochastic block models covers topic models as their own section and as an extension of graph clustering, where the text sits on the edges or nodes of a network 1903.00114 §10 1903.00114 §11.

Two families of plain topic model appear in the cards. Non-probabilistic models include Latent Semantic Analysis (LSA) and Non-negative Matrix Factorisation (NMF) 2301.08826 §2.3.1. Probabilistic models include Probabilistic LSA (PLSA) and Latent Dirichlet Allocation (LDA) 2301.08826 §2.3.2 1903.00114 §10.

The cards give no hyperparameter values, no measured comparison between plain topic models, and no results for them. The graph-based numbers below are the review's summaries of other studies, not measurements made by the review.

Variants

Non-probabilistic models

The student-feedback review places LSA and NMF in this family 2301.08826 §2.3.1. It gives no procedure, cost or quality comparison for either. Card: Shaik et al., 2023.

Latent Dirichlet allocation (probabilistic)

The block-model review calls LDA the reference topic model [arxiv:1903.00114 §10]. In its description, each document has a topic mixture drawn from a symmetric Dirichlet, each topic is a Dirichlet distribution over the vocabulary, and each word draws its own topic 1903.00114 §10. Fitting is mostly Bayesian, with collapsed Gibbs sampling or variational methods 1903.00114 §10.1. The student-feedback review says LDA is the most used topic model in the works it reviews 2301.08826 §2.3.2 2301.08826 §4. That describes usage, not a measured advantage.

Cards: Shaik et al., 2023 and Lee and Wilkinson, 2019.

Choosing the number of topics

Plain LDA takes the number of topics as an input. The block-model review lists three options: fix it, choose it by perplexity or marginal likelihood, or let a Dirichlet process (HDP) or Indian buffet process (IBP) generate it 1903.00114 §10.2 1903.00114 §10.2.1. It does not say which option gives the best topics. Its comparison table lists the number of topics per study, for example fixed at 20 in one study and set by a Dirichlet process in others 1903.00114 tab. 3.

Dynamic topic models

These let topic distributions change over time, with a state-space model, or with a Dirichlet process that evolves over time 1903.00114 §10.3.

Topic models on networks

When the text sits on the edges of a network, for example emails between people, the review lists author-recipient topic models, group-based topic models, and the stochastic topic block model (STBM), which gives each pair of groups its own topic mixture 1903.00114 §11.1.1.

When the text sits on the nodes, for example linked documents, it lists topic-link LDA, a relational topic model, and a hierarchical tree model that mixes word distributions along a path 1903.00114 §11.1.2.

On the Enron email network, one reviewed study selects 10 groups and 5 topics with the STBM, against 8 groups from a block model alone [arxiv:1903.00114 §12.1.2]. This is one study's result as summarised by the review. Card: Lee and Wilkinson, 2019. Its clustering setup is covered on the stochastic block model page.

When to use which

The cards give no rule that links a constraint to a topic model. The table records what they support and what they leave open.

For a real choice, read the original papers on each model. They are not on this wiki yet, so no card supports a claim about them.

Implementation sketch

The cards give no fitting algorithm, equation set or code for any topic model. The sketch below is only the generative story of LDA, as the block-model review describes it 1903.00114 §10. It does not say how to infer the topics. The review names collapsed Gibbs sampling and variational methods for that 1903.00114 §10.1. Hyperparameter values are not stated.

# Sample documents from LDA  [arxiv:1903.00114 §10]
# Input:  K topics, V vocabulary size, alpha, beta (symmetric concentrations)
#         doc_lengths (list of ints)
# Output: documents as lists of word ids

def sample_lda(K, V, alpha, beta, doc_lengths):
    # Each topic is a distribution over the vocabulary  [arxiv:1903.00114 §10]
    topic_words = [draw_dirichlet(beta * ones(V)) for k in range(K)]
    docs = []
    for length in doc_lengths:
        # Each document has its own topic mixture  [arxiv:1903.00114 §10]
        mixture = draw_dirichlet(alpha * ones(K))
        words = []
        for _ in range(length):
            # Each word draws its own topic, then a word from that topic
            z = draw_categorical(mixture)  # [arxiv:1903.00114 §10]
            words.append(draw_categorical(topic_words[z]))
        docs.append(words)
    return docs

Contradictions

Neither card disagrees with the other on a measured claim. They describe different settings. The student-feedback review calls LDA the most used model 2301.08826 §2.3.2, and the block-model review calls it the reference model 1903.00114 §10. Each is a claim about the field, not a test.

Within the student-feedback review, two source inconsistencies affect this page:

• Table 2's Topic Modelling row lists Wang et al. and Fowler et al. The text does not discuss either. In the TeX source, their paragraphs are commented out 2301.08826 §2.3.2 tab. 2.
• Table 2's Feature Extraction row lists LSA and LDA. The text places both under topic modelling 2301.08826 §2.3.1 2301.08826 §2.3.2 tab. 2.

Papers

• Lee and Wilkinson, 2019: a review of stochastic block models. It covers LDA and the topic-model extensions to network models, and compares models in four tables. It runs no experiments of its own.
• Shaik et al., 2023: a review of NLP for student feedback. It names two families of topic model and four models, but runs no experiments and gives no results tied to a specific model.