Improving Topic Modeling by Distilling Soft Labels from Language Models

Raymond Li; Amirhossein Abaskohi; Chuyuan Li; Gabriel Murray; Giuseppe Carenini

cs.CLFeb 20, 2026

Improving Topic Modeling by Distilling Soft Labels from Language Models

Authors: Raymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini

Organizations: University of British Columbia, Vancouver, Canada.

Abstract

Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity. In this work, we introduce a novel topic model training framework by Distilling Soft Labels (DSL) from Language Models (LMs). To construct the contextually enriched reconstruction signals, we project the next token probabilities, conditioned on a specialized prompt, onto a pre-defined vocabulary, and train the topic models to reconstruct the soft labels using the LM hidden states. This produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus. Extensive experiments demonstrate that DSL achieves substantial improvements in topic coherence and assignment accuracy over existing baselines. Additionally, we also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents, highlighting its effectiveness for retrieval-oriented applications.

Figures & tables

Figure 1: Comparison of BoW target (left) versus DSL target (right) for a document in the 20Newsgroups dataset. BoW can only assign mass to words that appear in the document ( bolded ), while the DSL target captures the underlying themes of the sports commentary even when the words don’t explicitly appear in the document (e.g., “ sports ”, “ hockey ”).

Figure 2: Overview of the proposed DSL framework. Given an input document $x$ and a thematic instruction prompt $\pi$ , the language model produces a prompt-conditioned next-token distribution, which is projected onto the topic-model vocabulary to form the semantic soft target $y_{\rm DSL}$ . The LM final hidden-state representation $x_{\rm emb}$ is used as the document representation for the topic model $\mathcal{M}$ , which infers document-topic proportions $\theta$ and predicts a vocabulary-level distribution $\hat{y}(x)$ . Training distills the LM-induced target into the structured topic model by minimizing $D_{\mathrm{KL}}(y_{\rm DSL}\,\|\,\hat{y})$ together with the model-specific regularization term $\mathcal{R}_{\mathcal{M}}$ .

	20NewsGroup				TweetTopic				StackOverflow
Method	$C_{V}$	LLM	IRBO	Purity	$C_{V}$	LLM	IRBO	Purity	$C_{V}$	LLM	IRBO	Purity
Baselines
LDA	.341	2.22	.977	.301	.350	1.95	.992	.441	.354	2.00	.977	.174
ProdLDA	.351	2.49	.992	.356	.355	2.11	.994	.533	.352	2.62	.991	.265
CombinedTM	.351	2.57	.993	.391	.360	2.36	.988	.588	.364	2.85	.986	.306
ZeroShotTM	.354	2.53	.994	.397	.356	2.30	.994	.573	.363	2.84	.993	.307

Table 1: Automatic evaluation results on the top-15 words averaged over 4 numbers of topics (

K=25,50,75,100

), where results for each

K

are averaged over 5 random seeds (20 measurements per cell). For each dataset–metric pair, best-performing method and methods that are not significantly different from the best under Welch’s independent-samples

t

-test with

\alpha=0.05

are highlighted .

Figure 3: Results on 20NewsGroups with ERNIE-4.5-0.3B for various temperature $\tau$ values. The values are normalized to visualize relative trends.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

LLM Rating System Prompt

You are a helpful assistant evaluating the top words of a topic model output for a given topic. Please rate how related the following words are to each other on a scale from 1 to 3 (“1” = not very related, “2” = moderately related, “3” = very related).

Reply with a single number, indicating the overall appropriateness of the topic.

Appendix

Table 6: System prompt used for the LLM topic-coherence rating.

Panel of Figure 4 — Figure 4: Results on 20NewsGroup, TweetTopic, and StackOverflow with ERNIE-4.5-0.3B for different temperature values $\tau$ . Metric values are normalized to visualize relative trends.

Instruction Prompt Variants

1. Name the document’s central theme.

2. Give a single label that sums up the text.

3. Tag the document with its main idea.

4. Read the following document in full, identify its principal theme, and output a one-word label that best conveys that unifying concept.

5. Generate a tag that captures the primary theme running through the entire document.

Appendix

Table 7: Five rephrased instruction prompts used to construct soft labels for the prompt-sensitivity analysis.

Panel of Figure 5 — Figure 5: Effect of vocabulary size on topic model performance across four evaluation metrics on 20NewsGroup. Results are shown for vocabulary sizes 500, 1,000, 2,000, and 4,000. DSL with ERNIE-4.5-0.3B is shown in red.

Table 10: Automatic evaluation results on the top-15 words for each number of topics (

K=25,50,75,100

), with five random seeds per

K

. Results not statistically significantly different from the best method (

p\geq 0.05

) under Welch’s

t

-test are highlighted in blue. Rows without a suffix use the ProdLDA backbone with our proposed method; rows suffixed with + ECRTM or + FASTopic use the ECRTM or FASTopic backbone.

Table 11: Automatic evaluation results on the top-15 words for each number of topics, continued.

Table 12: Automatic evaluation results on the top-15 words for each number of topics, continued.

Table 13: Automatic evaluation results on the top-15 words for each number of topics, continued.

Topic Modeling

Topic modeling discovers recurring themes and document-topic structure in text, helping organize, compare, and track large collections.

See all 57 papers

Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders
Jul 31, 2025Carolina Zheng, Nicolas Beltran-Velez, Sweta Karlekar +5Topic Modeling
Columbia University · Independent · Google Research
TriTopic: Tri-Modal Graph-Based Topic Modeling with Iterative Refinement and Archetypes
Feb 22, 2026Roman EggerTopic Modeling
Smartvisions.at & Modul University Vienna
From Documents to Segments: A Contextual Reformulation for Topic Assignment
May 18, 2026Hoonsang Yoon, Takyoung Kim, Wonkee Lee +3Topic Modeling
1LG AI Research · University of Illinois Urbana-Champaign

Jul 31, 2025J/K move · Enter open · S save

Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders

Carolina Zheng, Nicolas Beltran-Velez, Sweta Karlekar, Claudia Shi+4

Columbia University · Independent · Google Research

Topic Modeling

Open paper AskPDF arXiv

Traditional topic models are effective at uncovering latent themes in large text collections. However, due to their reliance on bag-of-words representations, they struggle to capture semantically abstract features. While some neural variants use richer representations, they are similarly constrained by expressing topics as word lists, which limits their ability to articulate complex topics. We introduce Mechanistic Topic Models (MTMs), a class of topic models that operate on interpretable features learned by sparse autoencoders (SAEs). By defining topics over this semantically rich space, MTMs can reveal deeper conceptual themes with expressive feature descriptions. Moreover, uniquely among topic models, MTMs enable controllable text generation using topic steering vectors. To properly evaluate MTM topics against word list approaches, we propose topic judge, an LLM-based pairwise comparison evaluation framework. Across eight datasets, MTMs match or exceed traditional and neural baselines on coherence metrics, are consistently preferred by topic judge, and enable effective LLM steering.

Abstract

Topic Modeling

Explore similar work

Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders

TriTopic: Tri-Modal Graph-Based Topic Modeling with Iterative Refinement and Archetypes

From Documents to Segments: A Contextual Reformulation for Topic Assignment