Recent advances in neural topic models with pre-trained language models (PLMs) have achieved strong performance by leveraging general-domain pre-training, yet their topic interpretability often degrades on specialized corpora. This limitation primarily stems from the geometry of the embedding space, where domain-specific terms unseen during pre-training collapse into an indistinguishable region, and neither domain-specific re-training, word-level graph enrichment, nor parameter-efficient fine-tuning can restructure this space without inheriting the capacity ceiling of the underlying encoder. Our key insight is that a learnable graph layer operating on token-level PLM embeddings can acquire corpus-specific semantic structure that the frozen encoder lacks, because token-level graphs preserve document-local context that word-level representations discard and joint optimization with the topic objective reshapes embedding geometry directly from target-domain evidence. We instantiate this insight as DARTopic, a domain-agnostic framework that constructs token-level semantic graphs from frozen PLM embeddings and jointly trains a GNN encoder with topic inference. Across three benchmarks spanning general, biomedical, and legal domains, DARTopic consistently outperforms strong baselines in topic coherence and document clus- tering without any encoder fine-tuning, while demonstrating robustness to PLM choice and favorable runtime efficiency over fine-tuning based alternatives.
Table 1: Comparison of topics generated by FASTopic ( Wu et al., 2024b ) and DARTopic on BioASQ dataset. We report low-coherence words in RED font.
Figure 1: The overall architecture of our proposed DARTopic framework. denotes frozen parameters, while denotes trainable parameters.
Dataset
Domain
Metric
ProdLDA
FASTopic
BERTopic
ZeroShotTM
CombinedTM
NeuroMax
PVTM
GINopic
CGTM
DARTopic (ours)
20NG
General
NPMI
0.1581
0.2624
0.2300
0.2604
0.1670
0.1833
0.2707
0.2172
0.2520
0.2736
TU
0.7613
0.8813
0.4547
0.9227
0.7853
1.0000
0.9133
0.4747
0.5227
0.9360
TQ
0.1203
0.2313
0.1046
0.2403
0.1311
0.1833
0.2473
0.1031
0.1317
0.2561
BioASQ
Biomedical
NPMI
0.0142
0.0789
0.1653
0.1448
0.0120
0.0094
0.1596
0.0213
0.1377
0.1641
TU
0.8213
0.4093
0.7320
0.8640
0.8080
1.0000
0.9730
0.8013
0.6947
0.9760
TQ
0.0117
0.0323
0.1210
0.1251
0.0097
0.0094
0.1553
0.0171
0.0957
0.1602
Table 2: Main results on topic-word distribution quality across three domains: general (20NG), biomedical (BioASQ) and legal (Bills). The best performance in each row is shown in bold and the second-best is underlined .
NPMI
TU
TQ
Purity
NMI
MiniLM
0.1641
0.9760
0.1602
0.5235
0.3833
Roberta-large
0.1660
0.9680
0.1606
0.4738
0.3492
Qwen-0.6B
0.1644
0.9653
0.1587
0.4829
0.3555
BioBERT
0.1690
0.9693
0.1638
0.5061
0.3696
Table 3: Comparison of DARTopic performance on BioASQ dataset with different pre-trained models.
20NG
BioASQ
Purity
NMI
Purity
NMI
ProdLDA
0.1186
0.1652
0.2032
0.1030
BERTopic
0.4347
0.3369
0.4666
0.3205
FASTopic
0.4743
0.4089
0.4339
0.3650
ZeroshotTM
0.4555
0.3657
0.3217
0.3312
CombinedTM
0.1963
0.2262
0.2466
0.1566
Table 4: Performance comparison on document clustering task using document-topic distribution vector. The best-performing method is highlighted in bold and second-best performance is underlined .
Methods
PLMs
NPMI
TU
TQ
Purity
NMI
FASTopic
MiniLM
0.0789
0.4093
0.0323
0.4339
0.3650
FASTopic
BioBERT
0.1146
0.9920
0.1137
0.4439
0.3375
ZeroshotTM
MiniLM
0.1448
0.8640
0.1251
0.3217
0.3312
ZeroshotTM
BioBERT
0.1628
0.9680
0.1576
0.4967
0.3652
CombinedTM
MiniLM
0.0120
0.8080
0.0097
0.2466
0.1566
CombinedTM
BioBERT
0.0130
0.7813
0.0102
0.1108
0.1157
Table 5: PLM dependency of PLM-powered neural topic models performance on BioASQ dataset. The better result is highlighted in bold for each metric.
20NG
BioASQ
Bills
Training
Inference
Training
Inference
Training
Inference
PVTM
13.55s
12.79s
2.55s
2.22s
9.82s
9.56s
DARTopic
7.67s
7.65s
2.59s
2.44s
8.67s
8.52s
Table 6: Runtime performance comparison with PVTM.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Domain
#Docs
#Vocab
Avg. length
#Labels
20NewsGroup
General
16,309
1,612
48.02
20
BioASQ
Biomedical
19,448
2,000
7.44
20
Bills
Legal
18,945
2,000
76.28
–
Appendix
Table 7: Statistics of the preprocessed benchmark datasets
Graph Structure
20NG
BioASQ
NPMI
TU
TQ
NPMI
TU
TQ
1-hop
0.2643
0.9307
0.2460
0.1583
0.9600
0.1520
2-hop
0.2667
0.9400
0.2507
0.1687
0.9600
0.1620
3-hop
0.2655
0.9227
0.2449
0.1659
0.9693
0.1608
Semantic (ours)
0.2736
0.9360
0.2561
0.1641
0.9760
0.1602
Appendix
Table 8: Analysis of different graph construction methods in our proposed DARTopic framework. The best-performing method is highlighted in bold .
20NG
BioASQ
Bills
NPMI
TU
TQ
NPMI
TU
TQ
NPMI
TU
TQ
τ = 0.1
0.2670
0.9320
0.2488
0.1668
0.9693
0.1617
0.2503
0.9307
0.2329
τ = 0.2
0.2736
0.9360
0.2561
0.1641
0.9760
0.1602
0.2545
0.9160
0.2331
τ = 0.3
0.2689
0.9307
0.2502
0.1737
0.9533
0.1656
0.2685
0.9240
0.2481
τ = 0.4
0.2647
0.9400
0.2488
0.1609
0.9640
0.1551
0.2489
0.9173
0.2283
Appendix
Table 9: Analysis of graph construction hyperparameter in DARTopic .
Topic modeling extracts latent themes from large text collections, but leading approaches like BERTopic face critical limitations: stochastic instability, loss of lexical precision ("Embedding Blur"), and reliance on a single data perspective. We present TriTopic, a framework that addresses these weaknesses through a tri-modal graph fusing semantic embeddings, TF-IDF, and metadata. Three core innovations drive its performance: hybrid graph construction via Mutual kNN and Shared Nearest Neighbors to eliminate noise and combat the curse of dimensionality; Consensus Leiden Clustering for reproducible, stable partitions; and Iterative Refinement that sharpens embeddings through dynamic centroid-pulling. TriTopic also replaces the "average document" concept with archetype-based topic representations defined by boundary cases rather than centers alone. In benchmarks across 20 Newsgroups, BBC News, AG News, and Arxiv, TriTopic achieves the highest NMI on every dataset (mean NMI 0.575 vs. 0.513 for BERTopic, 0.416 for NMF, 0.299 for LDA), guarantees 100% corpus coverage with 0% outliers, and is available as an open-source PyPI library.
Recent topic models leverage pretrained embeddings, but neural architectures produce latent representations without grounding in specific texts, and clustering-based pipelines assign representative documents only post hoc, relying on absolute distances distorted by hubness and anisotropy in high-dimensional spaces. We introduce MARETopic, a training-free framework that casts topic discovery as rank-based prototype selection. After projecting embeddings onto a low-dimensional manifold, MARETopic builds ranked lists encoding ordinal neighborhood structure. A greedy algorithm selects exactly K exemplar documents, real corpus texts, whose neighborhoods cover the corpus. Two variants share this criterion. MARETopicCorr scores candidates with a query performance predictor and a rank correlation measure, leading Purity and NMI on the two benchmarks with the most categories, ahead of both neural and clustering-based topic models. MARETopicDiff scores them with a rank-based diffusion matrix, needs neither measure, and runs 1.7 to 1.9 times faster. Without a single gradient update, MARETopic leads topic coherence on two of three datasets. A novel inter-topic Maximal Marginal Relevance step raises vocabulary diversity at little cost in coherence. Our code is available at https://github.com/thcastilho/maretopic.
Thiago César Castilho Almeida, Daniel Carlos Guimarães Pedronette
Department of Statistics, Applied Mathematics and Computing State University of São Paulo (UNESP), Rio Claro, Brazil
Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity. In this work, we introduce a novel topic model training framework by Distilling Soft Labels (DSL) from Language Models (LMs). To construct the contextually enriched reconstruction signals, we project the next token probabilities, conditioned on a specialized prompt, onto a pre-defined vocabulary, and train the topic models to reconstruct the soft labels using the LM hidden states. This produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus. Extensive experiments demonstrate that DSL achieves substantial improvements in topic coherence and assignment accuracy over existing baselines. Additionally, we also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents, highlighting its effectiveness for retrieval-oriented applications.
Raymond Li, Amirhossein Abaskohi, Chuyuan Li +2
University of British Columbia, Vancouver, Canada.