Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity. In this work, we introduce a novel topic model training framework by Distilling Soft Labels (DSL) from Language Models (LMs). To construct the contextually enriched reconstruction signals, we project the next token probabilities, conditioned on a specialized prompt, onto a pre-defined vocabulary, and train the topic models to reconstruct the soft labels using the LM hidden states. This produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus. Extensive experiments demonstrate that DSL achieves substantial improvements in topic coherence and assignment accuracy over existing baselines. Additionally, we also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents, highlighting its effectiveness for retrieval-oriented applications.
Figures & tables
Figure 1: Comparison of BoW target (left) versus DSL target (right) for a document in the 20Newsgroups dataset. BoW can only assign mass to words that appear in the document ( bolded ), while the DSL target captures the underlying themes of the sports commentary even when the words don’t explicitly appear in the document (e.g., “ sports ”, “ hockey ”).
Figure 2: Overview of the proposed DSL framework. Given an input document x and a thematic instruction prompt π , the language model produces a prompt-conditioned next-token distribution, which is projected onto the topic-model vocabulary to form the semantic soft target yDSL . The LM final hidden-state representation xemb is used as the document representation for the topic model M , which infers document-topic proportions θ and predicts a vocabulary-level distribution y^(x) . Training distills the LM-induced target into the structured topic model by minimizing DKL(yDSL∥y^) together with the model-specific regularization term RM .
20NewsGroup
TweetTopic
StackOverflow
Method
CV
LLM
IRBO
Purity
CV
LLM
IRBO
Purity
CV
LLM
IRBO
Purity
Baselines
LDA
.341
2.22
.977
.301
.350
1.95
.992
.441
.354
2.00
.977
.174
ProdLDA
.351
2.49
.992
.356
.355
2.11
.994
.533
.352
2.62
.991
.265
CombinedTM
.351
2.57
.993
.391
.360
2.36
.988
.588
.364
2.85
.986
.306
ZeroShotTM
.354
2.53
.994
.397
.356
2.30
.994
.573
.363
2.84
.993
.307
Table 1: Automatic evaluation results on the top-15 words averaged over 4 numbers of topics ( K=25,50,75,100 ), where results for each K are averaged over 5 random seeds (20 measurements per cell). For each dataset–metric pair, best-performing method and methods that are not significantly different from the best under Welch’s independent-samples t -test with α=0.05 are highlighted .
Figure 3: Results on 20NewsGroups with ERNIE-4.5-0.3B for various temperature τ values. The values are normalized to visualize relative trends.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
LLM Rating System Prompt
You are a helpful assistant evaluating the top words of a topic model output for a given topic. Please rate how related the following words are to each other on a scale from 1 to 3 (“1” = not very related, “2” = moderately related, “3” = very related).
Reply with a single number, indicating the overall appropriateness of the topic.
Appendix
Table 6: System prompt used for the LLM topic-coherence rating.
Figure 4: Results on 20NewsGroup, TweetTopic, and StackOverflow with ERNIE-4.5-0.3B for different temperature values τ . Metric values are normalized to visualize relative trends.
Instruction Prompt Variants
1. Name the document’s central theme.
2. Give a single label that sums up the text.
3. Tag the document with its main idea.
4. Read the following document in full, identify its principal theme, and output a one-word label that best conveys that unifying concept.
5. Generate a tag that captures the primary theme running through the entire document.
Appendix
Table 7: Five rephrased instruction prompts used to construct soft labels for the prompt-sensitivity analysis.
Figure 5: Effect of vocabulary size on topic model performance across four evaluation metrics on 20NewsGroup. Results are shown for vocabulary sizes 500, 1,000, 2,000, and 4,000. DSL with ERNIE-4.5-0.3B is shown in red.
Table 10: Automatic evaluation results on the top-15 words for each number of topics ( K=25,50,75,100 ), with five random seeds per K . Results not statistically significantly different from the best method ( p≥0.05 ) under Welch’s t -test are highlighted in blue. Rows without a suffix use the ProdLDA backbone with our proposed method; rows suffixed with + ECRTM or + FASTopic use the ECRTM or FASTopic backbone.
Table 11: Automatic evaluation results on the top-15 words for each number of topics, continued.
Table 12: Automatic evaluation results on the top-15 words for each number of topics, continued.
Table 13: Automatic evaluation results on the top-15 words for each number of topics, continued.
Traditional topic models are effective at uncovering latent themes in large text collections. However, due to their reliance on bag-of-words representations, they struggle to capture semantically abstract features. While some neural variants use richer representations, they are similarly constrained by expressing topics as word lists, which limits their ability to articulate complex topics. We introduce Mechanistic Topic Models (MTMs), a class of topic models that operate on interpretable features learned by sparse autoencoders (SAEs). By defining topics over this semantically rich space, MTMs can reveal deeper conceptual themes with expressive feature descriptions. Moreover, uniquely among topic models, MTMs enable controllable text generation using topic steering vectors. To properly evaluate MTM topics against word list approaches, we propose topic judge, an LLM-based pairwise comparison evaluation framework. Across eight datasets, MTMs match or exceed traditional and neural baselines on coherence metrics, are consistently preferred by topic judge, and enable effective LLM steering.
Carolina Zheng, Nicolas Beltran-Velez, Sweta Karlekar +5
Columbia University · Independent · Google Research
Topic modeling extracts latent themes from large text collections, but leading approaches like BERTopic face critical limitations: stochastic instability, loss of lexical precision ("Embedding Blur"), and reliance on a single data perspective. We present TriTopic, a framework that addresses these weaknesses through a tri-modal graph fusing semantic embeddings, TF-IDF, and metadata. Three core innovations drive its performance: hybrid graph construction via Mutual kNN and Shared Nearest Neighbors to eliminate noise and combat the curse of dimensionality; Consensus Leiden Clustering for reproducible, stable partitions; and Iterative Refinement that sharpens embeddings through dynamic centroid-pulling. TriTopic also replaces the "average document" concept with archetype-based topic representations defined by boundary cases rather than centers alone. In benchmarks across 20 Newsgroups, BBC News, AG News, and Arxiv, TriTopic achieves the highest NMI on every dataset (mean NMI 0.575 vs. 0.513 for BERTopic, 0.416 for NMF, 0.299 for LDA), guarantees 100% corpus coverage with 0% outliers, and is available as an open-source PyPI library.
Traditional topic modeling assigns a single topic to each document. In practice, however, many real-world documents, such as product reviews or open-ended survey responses, contain multiple distinct topics. This mismatch often leads to topic contamination, where unrelated themes are merged into a single topic, making it difficult to identify documents that truly focus on a specific subject. We address this issue by introducing segment-based topic allocation (SBTA), a reformulation of topic modeling that assigns topics not to entire documents, but to segments: short, coherent spans of text that each express a single theme. By modeling topical structure at the segment level, our approach yields cleaner and more interpretable topics and better supports analysis of multi-theme documents. To support systematic evaluation, we construct a SemEval-STM, a new dataset inspired by aspect-based sentiment analysis. Documents are first decomposed into topical segments using large language models (LLMs), followed by human refinement to ensure segment quality. We also propose a segment-level extension of the word intrusion task, enabling human evaluation of topical coherence at the granularity where topics are actually assigned. Across multiple models and evaluation metrics, we show that SBTA improves clustering quality and interpretability. Overall, this work provides a practical, scalable framework for fine-grained topic analysis in heterogeneous text corpora where documents naturally span multiple topics. URL: https://huggingface.co/datasets/LG-AI-Research/SemEval-STM
Hoonsang Yoon, Takyoung Kim, Wonkee Lee +3
1LG AI Research · University of Illinois Urbana-Champaign