Disentangling Similarity and Relatedness in Topic Models

Hanlin Xiao; Yang Wang; Mauricio A. Álvarez; Rainer Breitling

cs.CLMar 11, 2026

Disentangling Similarity and Relatedness in Topic Models

Authors: Hanlin Xiao, Yang Wang, Mauricio A. Álvarez, Rainer Breitling

Organizations: Manchester Institute of Biotechnology, The University of Manchester, UK · Department of Computer Science, The University of Manchester, UK · Department of Chemistry, The University of Manchester, UK

Abstract

The recent success of large pre-trained language models (PLMs) has motivated their integration into topic modeling. However, PLM-augmented topic models differ from classical co-occurrence models such as Latent Dirichlet Allocation (LDA) not only in performance, but also in the type of semantic structure they capture. We formalize this distinction along two psycholinguistic axes: thematic relatedness (dog/bone) and taxonomic similarity (dog/wolf). To measure both axes over topic words, we construct a large synthetic benchmark of word pairs using LLM-based annotation and train a neural scorer on it. Across multiple corpora and model families, the scorer places different topic-model families at distinct positions within the joint similarity-relatedness space. The two scores further predict downstream task performance: tasks requiring similarity benefit from similarity-rich topics, whereas tasks requiring relatedness benefit from the converse, and excessive emphasis on either axis degrades performance on tasks aligned with the opposing semantic structure. Neither axis is uniformly beneficial. Measuring both therefore provides a practical, model-agnostic diagnostic for evaluating the semantic structure captured by topic models.

Figures & tables

Figure 1: Topic model atlas pooled across all nine corpora. Each violin shows the distribution of per-topic scores over 9 corpora

\times

10 seeds

\times

20 topics for one of 17 models, colored by model family (Appendix B.1 for the full model inventory). Left: shifted gap, defined as normalized similarity minus normalized relatedness, shifted so a random-word baseline sits at zero; positive values lean similarity, negative lean relatedness. Centre and right: per-corpus Z-scores on each axis. Per-corpus breakdowns in Appendix A.6 .

Model	Sim. ( $\rho_{s}$ )	Rel. ( $\rho_{s}$ )
Neural Scorer	0.667	0.506
BERT ( Devlin et al., 2019 )	0.229	0.239
OpenAI ( OpenAI, 2024 )	0.347	0.182
WordNet W–P ( Wu and Palmer, 1994 )	0.620	$-0.374$

Table 1: Spearman rank correlation (

\rho_{s}

) with human similarity (Sim.) and relatedness (Rel.) ratings on the 659-pair TxThmNorms benchmark. Baselines: cosine similarity over single-word BERT and OpenAI text-embedding-3-large embeddings, and WordNet Wu–Palmer similarity.

Evaluation Axis	Kendall’s $W$
Similarity	0.768
Relatedness	0.743
Shifted Normalized Gap	0.732

Table 2: Cross-corpus rank agreement of topic models across nine datasets, quantified by Kendall’s coefficient of concordance

W

; all three axes attain

p<0.001

Figure 2: Per-task rank (1

=

best, 5

=

worst) of five models on three downstream tasks, grouped by class; cell color encodes rank (blue

=

best, red

=

worst). The double dissociation is visible: co-occurrence-based models lead on Event Monitoring and Category Retrieval but trail on Synonym-Aware Retrieval; PLM-augmented models reverse the pattern.

Single-factor partial $R^{2}$
	Relatedness		Similarity
Predictor	Event Mon.	Cat. Retr.	Syn-Aware Retr.
Similarity	.00	.06	.53
Relatedness	.24	.42	.03
Gap	.64	.64	.77
$C_{V}$ Coherence	.49	.41	.08

Table 3: Cross-corpus pooled regression of downstream performance. Top: single-factor partial

R^{2}

. Bottom: joint two-factor coefficients (

z

-scored within corpus).

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Word Pair	Score Type	Score 1	Score 2	Datasource 1	Datasource 2
breakfast / dinner	similarity	0.941	0.149	TxThmNorms	SimLex-999
camel / desert	relatedness	0.903	0.580	TxThmNorms	MEN

Appendix

Table 4: Conflicting scores across different datasets (normalized).

	LDA (Reuters)	BERT-KT (Reuters)
Topic 1	dollar, rate, exchange, currency, market	rise, increase, fall, decline, drop
Topic 2	growth, government, economic, economy, inflation	president, chairman, minister, secretary, chief
Topic 3	gold, reserves, production, ounces, mine	make, continue, reduce, support, remain

Appendix

Table 5: Representative topics generated by LDA and BERT-KT on Reuters ( Sia et al., 2020 ) . Each topic is shown using its top 5 words.

		Task A		Task B	Metric Scores
Model	Nature	Hit@3	Recall@3	SCR	Sim.	Rel.	Gap	Coh.	Div.
lda	Non-PLM	0.585	0.492	0.688	0.246	0.582	$-$ 0.336	0.510	0.660
bertopic	PLM-Aug.	0.573	0.482	0.679	0.228	0.558	$-$ 0.330	0.612	0.824
nmf	Non-PLM	0.526	0.456	0.642	0.251	0.572	$-$ 0.321	0.547	0.618
neurallda	Non-PLM	0.523	0.441	0.624	0.224	0.537	$-$ 0.313	0.417	0.998
prodlda	Non-PLM	0.520	0.445	0.662	0.233	0.532	$-$ 0.299	0.629	0.868

Appendix

Table 6: Event Monitoring and Category Retrieval: model performance on the Reuters corpus.

PLM Augmented
Model	Type	Reut.	20NG	BBC	DBLP	ACL	M10	NewsC	MN-DS	arXiv
TNTM	PLM Aug.	0.94	—	0.96	—	—	0.70	0.94	0.98	0.97
BERT-KT	PLM Aug.	0.76	0.48	0.40	0.60	0.68	0.54	0.74	0.74	0.56
T&M	PLM Aug.	0.45	0.25	0.38	0.39	—	0.33	—	—	—
Non-PLM Augmented
LDA	Non-PLM Aug.	0.23	0.16	0.12	0.06	0.20	0.13	0.04	0.12	0.17

Appendix

Table 7: Synonym-Aware Retrieval: Jaccard@10 across the nine corpora. “—” indicates the model was excluded for that corpus (genuine topic collapse) or not run.

Figure 3: Synonym-Aware Retrieval regression with corpus fixed effects: similarity score predicts synonym-aware retrieval consistency (Jaccard@10) across the nine corpora (partial

R^{2}=0.53

p<0.001

). Parallel lines show per-corpus intercepts with a shared slope.

approach/method	evaluate/assess	evaluation/assessment	identify/detect
identification/detection	discover/find	analysis/study	model/framework
method/technique	procedure/process	improve/enhance	develop/design
combine/integrate	predict/estimate	prediction/estimation	approximate/estimate
approximation/estimation	accuracy/precision	robust/stable	stability/consistency
verification/validation	goal/objective	goal/target	objective/target
task/work	handle/manage	assist/support	build/construct

Appendix

Table 8: Technical/scientific synonym list (50 pairs). Shared by 20NewsGroups, ACL, DBLP, and M10; arXiv uses a four-lemma variant of this list.

increase/rise	decline/fall	reduce/lower	stop/halt
purchase/acquire	agreement/accord	company/firm	stock/share
lawsuit/suit	announce/declare	begin/start	benefit/advantage
change/alter	choose/select	close/shut	complete/finish
concern/worry	create/build	deal/agreement	delay/postpone
demand/request	deny/reject	dismiss/reject	estimate/forecast
expand/grow	fast/quick	final/ultimate	goal/objective

Appendix

Table 9: Business/news synonym list (50 pairs). Shared by Reuters and BBC News; MN-DS and News Category use variants of this list with

\sim

15 substitutions each for lemmas absent from the corpus vocabulary.

Task	$n$	$\beta_{\text{sim}}$	$\beta_{\text{rel}}$	$R^{2}$
Event Monitoring	46	$\mathbf{-1.01}$	$\mathbf{+1.31}$	.58
Category Retrieval	46	$\mathbf{-0.88}$	$\mathbf{+1.37}$	.68
Synonym-Aware Retrieval	38	$\mathbf{+0.34}$	$\mathbf{-0.19}$	.79

Appendix

Table 10: Joint cross-corpus fixed-effects regression outcome

\sim

sim

+

rel

+

C

(corpus), both predictors

z

-scored within corpus. Each coefficient is the partial effect of one axis controlling for the other;

R^{2}

is the full-model value (includes the corpus fixed effects). Bold :

p<.001

Figure 4: Per-corpus topic model atlas. Each row is one corpus; columns give the per-model similarity, relatedness, and shifted-gap distributions (per-corpus standardised, $\approx$ 10 runs per (model, corpus) cell; excluded cells are listed in Table 12 ). Models share a fixed spectrum order across all panels, so the broadly stable left-to-right color pattern reflects the cross-corpus rank concordance reported in Table 2 . Models are identified on the horizontal axis by numeric id, assigned in that shared spectrum order (relatedness-leaning to similarity-leaning): 1 = bertopic , 2 = lsi , 3 = lda , 4 = fastopic , 5 = etm-fresh , 6 = ctm , 7 = CT2V , 8 = etm-bert , 9 = prodlda , 10 = nmf , 11 = neurallda , 12 = hdp , 13 = cwtm , 14 = ecrtm , 15 = random , 16 = T&M , 17 = bertkt , 18 = tntm . See Appendix B.1 for the full model inventory.

Model	Approach	Reference
LDA	Hierarchical Bayesian model; documents as mixtures of topics over vocabulary words	Blei et al. (2003)
LSI	Truncated SVD on the term–document matrix to extract latent semantic factors	Deerwester et al. (1990)
HDP	Nonparametric Bayesian extension of LDA; infers the number of topics automatically	Teh et al. (2006)
NMF	Factorises the term–document matrix under non-negativity constraints into additive topic components	Lee and Seung (1999)
NeuralLDA	VAE with logistic-normal prior for amortised LDA inference (AVITM, LDA variant)	Srivastava and Sutton (2017)
ProdLDA	Product-of-experts formulation in a VAE framework (AVITM, ProdLDA variant)	Srivastava and Sutton (2017)

Appendix

Table 11: Topic models included in the atlas. The 14 models above the mid-rule appear on all nine corpora; the four below appear on a subset only.

Model	Corpus	Failure mode
CWTM	BBC News	Diversity collapse: topics near-identical across runs
ETM ( etm_fresh )	DBLP	Diversity collapse: topic diversity $\leq 0.1$ on all runs
TNTM	DBLP	Word-count collapse: runs yield only 1–5 topics with $\leq$ 3 words
TNTM	ACL	Word-count collapse: runs yield only 1–4 topics with $\leq$ 3 words

Appendix

Table 12: Model–corpus pairs omitted from the atlas. An audit of the earlier, longer exclusion list found most of its entries stale—those model–corpus pairs do produce usable topics under the current pipeline and have been restored. Only the four pairs below exhibit genuine, reproducible collapse.

Source Family	Sub-sources	Pairs
WordNet ( Miller, 1995 )	synonyms (9.9K), cohyponyms (18.6K), hypernyms (4.6K), antonyms (1.3K)	34,346
ConceptNet 5.7 ( Speer et al., 2017 )	RelatedTo (5.3K), AtLocation (2.1K), PartOf (1.7K), UsedFor (0.3K), Causes (0.2K)	9,616
Embedding neighbors	BERT top- $k$ cosine nearest neighbours (max 3/cluster, 20/word)	14,961
Controlled negatives	far-distance (5.0K), cross-domain (3.7K), cross-POS (3.7K)	12,423
Total		71,346

Appendix

Table 13: Composition of the 71,346-pair training dataset by source family.

Prompt design (DeepSeek-V4)
Configuration	$r_{\text{sim}}$	$r_{\text{rel}}$	mean $r$
Answer-first (scores, then reason)	0.828	0.809	0.818
Reason-first, 1 sentence	0.873	0.835	0.854
Reason-first, 2 sentences	0.879	0.837	0.858
Reason-first, 2 sent, simple prompt	0.869	0.809	0.839
Single annotator

Appendix

Table 14: LLM annotator selection on TxThmNorms (659 human-rated pairs).

r

is the Pearson correlation between LLM and human scores. All single-annotator and ensemble rows use reason-first, two-sentence reasoning. Best mean

r

per block in bold.

Block	Setting	Value
Input	Word embeddings	OpenAI text-embedding-3-large , prompt-templated static, 3072D
	Pair features (embedding)	$[\mathbf{e}_{1};\,\mathbf{e}_{2};\,\|\mathbf{e}_{1}{-}\mathbf{e}_{2}\|;\,\mathbf{e}_{1}{\odot}\mathbf{e}_{2}]$ ( $4\times 3072=12{,}288$ D)
	Pair features (WordNet)	$8$ D: path sim, LCH, WuP, Lin, JCN, shortest path, same category, is-synonym (binary)
	Total input dimension	$12{,}296$
Architecture	Shared trunk	$12{,}296\rightarrow 3{,}072\rightarrow 768$ (Linear + GELU + Dropout)
	Task heads	Similarity: $768\rightarrow 1$ (Sigmoid); Relatedness: $768\rightarrow 1$ (Sigmoid)

Appendix

Table 15: Neural scorer architecture and training configuration.

Figure 8: Training curves for the production neural scorer ( ensemble3_rf2 , OpenAI text-embedding-3-large prompt embeddings + WordNet features, no gap loss). Clockwise from top-left: MSE loss (train/val/test) over epochs; validation similarity Spearman; held-out test Spearman for similarity and relatedness; validation relatedness Spearman. Validation and test Spearman correlations rise quickly and plateau by epoch 20, while train loss continues to decline—the expected train/val gap on a noisy regression label, with no degradation in held-out ranking.

Subset	$\rho_{\text{sim}}$	$\rho_{\text{rel}}$
Noun–noun ( $n=5{,}645$ )	0.897	0.827
Other ( $n=4{,}911$ )	0.925	0.836
All ( $n=10{,}556$ )	0.916	0.836

Appendix

Table 16: Production scorer ( ensemble3_rf2 ) performance on noun–noun vs. not-noun–noun test pairs of the internal 70/15/15 split. Primary POS is determined by the most frequent part of speech among each word’s WordNet synsets; pairs whose Ref or Pair word has no WordNet entry are dropped.

Figure 9: Scaling analysis for the production scorer (OpenAI text-embedding-3-large prompt embeddings + WordNet features): test-set Spearman correlation as a function of training-set size, averaged over two seeds; error bars show

\pm 1

standard deviation. Both axes continue to improve from 500 to 64,000 pairs (

\rho_{\text{sim}}

: 0.78

\rightarrow

0.92;

\rho_{\text{rel}}

: 0.62

\rightarrow

0.83), with progressively smaller marginal gains beyond 16,000 pairs.

Embedding	Dim	MLP trunk	Internal (S/R)	Holdout (S/R)	External (S/R)	Mean
Count-based word embedding
GloVe 840B ( Pennington et al., 2014 )	300	256-64	0.885 / 0.811	0.841 / 0.811	0.520 / 0.437	0.718
Large-model static embeddings
X2Static (RoBERTa-large) ( Gupta and Jaggi, 2021 )	1024	1024-512-128	0.877 / 0.808	0.841 / 0.781	0.532 / 0.421	0.710
Model2Vec (potion-base)	512	256-64	0.751 / 0.656	0.754 / 0.697	0.466 / 0.291	0.603
WordLlama	256	256-64	0.697 / 0.565	0.588 / 0.577	0.270 / 0.302	0.500

Appendix

Table 17: Input-embedding ablation. Each row is one scorer trained on the 71k LLM-labeled dataset with the listed input embedding, all else fixed—a controlled comparison of embedding choices, not the deployed-scorer numbers. Spearman

\rho

(similarity / relatedness) on the internal test split, the word-level holdout, and the external TxThmNorms benchmark; the MLP trunk is scaled to the embedding dimension. The adopted embedding and the best mean are in bold. The deployed production scorer further retrains the bold ( OpenAI te3-large, prompt ) configuration on the final reason-first 3-model ensemble labels, reaching

0.905/0.822

(internal),

0.869/0.798

(word-holdout),

0.667/0.506

(TxThmNorms).

Component	Setting	Pair-heldout (S,R)	Word-heldout (S,R)	External (Tx,Thm)
Input	$[\mathbf{e}_{1};\mathbf{e}_{2}]$ (no interaction)	.736, .632	.712, .684	.606, -.006
WordNet	OFF	.887, .814	.843, .783	.496, .420
Architecture	Cross-attn hybrid	.648, .578	.631, .627	.202, -.010

Appendix

Table 18: Ablation of the remaining scorer design axes—input representation, WordNet features, and architecture—relative to the adopted configuration. Spearman

\rho

(S,R) on the pair-heldout and word-heldout splits and the external TxThmNorms benchmark. The WordNet row is the production OpenAI-embedding scorer retrained with the 8-dimensional WordNet feature vector disabled; the input-representation and architecture rows report development-time results from the scorer design search. Each ablation isolates one axis against an otherwise-fixed configuration.

Corpus	Domain	Documents	Avg Tokens	Vocab Size
20NewsGroups ( Lang, 1995 )	Online forum posts (20 topics)	18,248	69.4	24,381
BBC News ( Greene and Cunningham, 2006 )	News articles (5 categories)	2,225	111.0	2,814
Reuters-21578 ( Lewis, 1997 )	Newswire (financial/economic)	7,717	54.4	5,699
DBLP ( Ley, 2002 )	CS paper titles	54,595	5.3	1,480
M10 ( Lim and Buntine, 2015 )	Scientific paper titles (10 areas)	8,355	5.8	1,591
ACL ( Bird et al., 2008 )	NLP paper full texts	10,551	1,661.0	8,868

Appendix

Table 19: Summary Statistics of Text Corpora

Component	Wall-clock	Hardware / notes
LLM annotation (71,346 pairs $\times$ 3 providers)	${\approx}1$ h per provider, run in parallel	1 CPU, 5 GB RAM; API-bound. Total API spend ${\approx}\$ 50 $–$ $90$ .
Neural-scorer training	17 min	1 ${\times}$ A100 80GB, 100 epochs, batch 32
Scorer ablations + scaling sweep	${\approx}9$ GPU-hours total	Same hardware as above
Atlas topic-model training (1,567 runs across 17 topic models and a random baseline $\times$ 9 corpora $\times$ 10 seeds)	${\approx}200$ – $250$ GPU-hours total	GH200 single-GPU; CPU-only for co-occurrence models. Per-model wall time varies from $<1$ min (Random, BERT-KT, LDA) to ${\approx}1$ h (CombinedTM); Contextual Top2Vec additionally requires ${\approx}110$ GB host RAM.
Atlas scoring (162 cells)	48 min	1 ${\times}$ A100 80GB, 12 CPUs, 64 GB RAM
Downstream evaluation (regressions, retrieval tasks, traditional metrics)	${\leq}10$ GPU-hours	4 CPUs, 32 GB RAM per task

Appendix

Table 20: Compute budget for the main pipeline components. Atlas training totals are approximate because per-model wall time varies by an order of magnitude across the seventeen model families and nine corpora; the slowest contributors per corpus are CombinedTM, CWTM, ECRTM, Thompson & Mimno, and Contextual Top2Vec.

Topic Modeling

Topic modeling discovers recurring themes and document-topic structure in text, helping organize, compare, and track large collections.

See all 57 papers

When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics
Aug 13, 2026Charu Karakkaparambil JamesTopic Modeling
RPTU University Kaiserslautern-Landau Kaiserslautern, Germany
Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders
Jul 31, 2025Carolina Zheng, Nicolas Beltran-Velez, Sweta Karlekar +5Topic Modeling
Columbia University · Independent · Google Research
Improving Topic Modeling by Distilling Soft Labels from Language Models
Feb 20, 2026Raymond Li, Amirhossein Abaskohi, Chuyuan Li +2Topic Modeling
University of British Columbia, Vancouver, Canada.

Aug 13, 2026J/K move · Enter open · S save

When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics

Charu Karakkaparambil James

RPTU University Kaiserslautern-Landau Kaiserslautern, Germany

Topic Modeling

Open paper AskPDF arXiv

Dynamic topic models capture evolving word distributions, but traditional coherence metrics may fail when vocabulary changes while semantic meaning persists. We evaluate 120 topics from CoNTM and DLDA across NYT, DBLP, and arXiv, using three human annotators and Low, Medium, and High lexical-change categories. Traditional temporal coherence shows highly variable agreement with human judgments (

ρ

=-0.256 to 0.614). In contrast, LLM-based semantic similarity agrees strongly with human semantic judgments for CoNTM on NYT (

ρ

=0.609), DBLP (

ρ

=0.721), and arXiv (

ρ

=0.502), but is less consistent for DLDA. Lexical-change stratification reveals variation hidden by aggregate evaluation. We therefore advocate lexical-change-aware evaluation, jointly reporting traditional coherence and LLM-based semantic measures as complementary rather than interchangeable signals.

Abstract

Topic Modeling

Explore similar work

When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics

Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders

Improving Topic Modeling by Distilling Soft Labels from Language Models