When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics

Charu Karakkaparambil James

cs.CLAug 13, 2026

When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics

Authors: Charu Karakkaparambil James

Organizations: RPTU University Kaiserslautern-Landau Kaiserslautern, Germany

Abstract

Dynamic topic models capture evolving word distributions, but traditional coherence metrics may fail when vocabulary changes while semantic meaning persists. We evaluate 120 topics from CoNTM and DLDA across NYT, DBLP, and arXiv, using three human annotators and Low, Medium, and High lexical-change categories. Traditional temporal coherence shows highly variable agreement with human judgments ( $ρ$ =-0.256 to 0.614). In contrast, LLM-based semantic similarity agrees strongly with human semantic judgments for CoNTM on NYT ( $ρ$ =0.609), DBLP ( $ρ$ =0.721), and arXiv ( $ρ$ =0.502), but is less consistent for DLDA. Lexical-change stratification reveals variation hidden by aggregate evaluation. We therefore advocate lexical-change-aware evaluation, jointly reporting traditional coherence and LLM-based semantic measures as complementary rather than interchangeable signals.

Figures & tables

Figure 1: Relationship between lexical temporal change and traditional NPMI coherence for CoNTM and DLDA across NYT, arXiv, and DBLP. Panels (a–b) show a negative association between lexical change and coherence across datasets. Panels (c–d) illustrate that topics with relatively low coherence can nevertheless exhibit meaningful and interpretable semantic evolution, motivating evaluation beyond lexical coherence alone.

Model	Dataset	$C_{\mathrm{trad.}}\leftrightarrow H_{\mathrm{TTC}}$	$S_{\mathrm{LLM}}\leftrightarrow H_{\mathrm{SS}}$
CoNTM	NYT	$-0.256$	$0.609$
CoNTM	DBLP	$0.425$	$0.721$
CoNTM	arXiv	$0.500$	$0.502$
DLDA	NYT	$0.531$	$0.445$
DLDA	DBLP	$0.614$	$-0.086$
DLDA	arXiv	$0.007$	$-0.190$

Table 1: Spearman’s

\rho

between automatic metrics and corresponding three-annotator human judgments.

C_{\mathrm{trad.}}

denotes traditional temporal coherence,

H_{\mathrm{TTC}}

denotes human Temporal Topic Coherence, and

H_{\mathrm{SS}}

denotes human Semantic Similarity. Bold values indicate statistically significant correlations (

p<0.05

Setting	Low	Medium	High
CoNTM–NYT, $C_{\mathrm{trad.}}\leftrightarrow H_{\mathrm{TTC}}$	$-0.786$	$0.116$	$0.516$
DLDA–NYT, $C_{\mathrm{trad.}}\leftrightarrow H_{\mathrm{TTC}}$	$0.094$	$0.873$	$0.278$
DLDA–DBLP, $C_{\mathrm{trad.}}\leftrightarrow H_{\mathrm{TTC}}$	$0.177$	$0.898$	$0.359$
CoNTM–NYT, $S_{\mathrm{LLM}}$ $\leftrightarrow H_{\mathrm{SS}}$	$0.801$	$0.750$	$-0.016$
CoNTM–DBLP, $S_{\mathrm{LLM}}$ $\leftrightarrow H_{\mathrm{SS}}$	$0.600$	$0.786$	$0.626$
CoNTM–arXiv, $S_{\mathrm{LLM}}$ $\leftrightarrow H_{\mathrm{SS}}$	$0.730$	$0.311$	$0.117$

Table 2: Selected lexical-change-stratified Spearman correlations (

\rho

). Blue rows represent traditional coherence versus human Temporal Topic Coherence (TTC), while orange rows represent LLM semantic similarity versus human Semantic Similarity (SS). Green cells highlight the statistically significant correlations (

p<0.05

). Category sizes are

N=7

7

, and

6

for Low, Medium, and High lexical change, respectively.

Topic Modeling

Topic modeling discovers recurring themes and document-topic structure in text, helping organize, compare, and track large collections.

See all 57 papers

Disentangling Similarity and Relatedness in Topic Models
Mar 11, 2026Hanlin Xiao, Yang Wang, Mauricio A. Álvarez +1Topic Modeling Semantic Textual Similarity
Manchester Institute of Biotechnology, The University of Manchester, UK · Department of Computer Science, The University of Manchester, UK · Department of Chemistry, The University of Manchester, UK
When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation
Mar 2, 2026Thibault Prouteau, Francis Lareau, Nicolas Dugué +2Topic Modeling Automated Evaluation
Université de Lorraine, CNRS, LORIA, Nancy (France) · Université de Sherbrooke, Dept of Philosophy and Applied Ethics, Sherbrooke (Canada) · Université du Québec à Montréal, Dept of Philosophy & CIRST, Montréal (Canada) +2
Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders
Jul 31, 2025Carolina Zheng, Nicolas Beltran-Velez, Sweta Karlekar +5Topic Modeling
Columbia University · Independent · Google Research

Mar 11, 2026J/K move · Enter open · S save

Disentangling Similarity and Relatedness in Topic Models

Hanlin Xiao, Yang Wang, Mauricio A. Álvarez, Rainer Breitling

Manchester Institute of Biotechnology, The University of Manchester, UK · Department of Computer Science, The University of Manchester, UK · Department of Chemistry, The University of Manchester, UK

Topic Modeling Semantic Textual Similarity

Open paper AskPDF arXiv

The recent success of large pre-trained language models (PLMs) has motivated their integration into topic modeling. However, PLM-augmented topic models differ from classical co-occurrence models such as Latent Dirichlet Allocation (LDA) not only in performance, but also in the type of semantic structure they capture. We formalize this distinction along two psycholinguistic axes: thematic relatedness (dog/bone) and taxonomic similarity (dog/wolf). To measure both axes over topic words, we construct a large synthetic benchmark of word pairs using LLM-based annotation and train a neural scorer on it. Across multiple corpora and model families, the scorer places different topic-model families at distinct positions within the joint similarity-relatedness space. The two scores further predict downstream task performance: tasks requiring similarity benefit from similarity-rich topics, whereas tasks requiring relatedness benefit from the converse, and excessive emphasis on either axis degrades performance on tasks aligned with the opposing semantic structure. Neither axis is uniformly beneficial. Measuring both therefore provides a practical, model-agnostic diagnostic for evaluating the semantic structure captured by topic models.

Abstract

Topic Modeling

Explore similar work

Disentangling Similarity and Relatedness in Topic Models

When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation

Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders