When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation
Authors: Thibault Prouteau, Francis Lareau, Nicolas Dugué, Jean-Charles Lamirel, Christophe Malaterre
Organizations: Université de Lorraine, CNRS, LORIA, Nancy (France) · Université de Sherbrooke, Dept of Philosophy and Applied Ethics, Sherbrooke (Canada) · Université du Québec à Montréal, Dept of Philosophy & CIRST, Montréal (Canada) · Le Mans Université, LIUM, Le Mans (France) · Université de Strasbourg, Strasbourg (France)
Topic models uncover latent thematic structures in text corpora, yet evaluating their quality remains challenging, particularly in specialized domains. Existing methods often rely on automated metrics like topic coherence and diversity, which may not fully align with human judgment. Human evaluation tasks, such as word intrusion, provide valuable insights but are costly and primarily validated on general-domain corpora. This paper introduces Topic Word Mixing (TWM), a novel human evaluation task assessing inter-topic distinctness by testing whether annotators can distinguish between word sets from single or mixed topics. TWM complements word intrusion's focus on intra-topic coherence and provides a human-grounded counterpart to diversity metrics. We evaluate six topic models - both statistical and embedding-based (LDA, NMF, Top2Vec, BERTopic, CFMF, CFMF-emb) - comparing automated metrics with human evaluation methods based on nearly 4,000 annotations from a domain-specific corpus of philosophy of science publications. Our findings reveal that word intrusion and coherence metrics do not always align, particularly in specialized domains, and that TWM captures human-perceived distinctness while appearing to align with diversity metrics. We release the annotated dataset and task generation code. This work highlights the need for evaluation frameworks bridging automated and human assessments, particularly for domain-specific corpora.
Figures & tables
Figure 1: Example of a two-topic Word Mixing task in LabelStudio .
Coherence-10
Diversity-10
K≈
10
25
50
10
25
50
Top2Vec
0,72
0,79
0,80
1,00
1,00
0,98
CFMF
0,69
0,72
0,72
0,96
0,85
0,81
CFMF-emb
0,67
0,69
0,70
0,90
0,79
0,73
NMF
0,59
0,65
0,67
0,90
0,88
0,78
BERTopic
0,52
0,58
0,62
0,79
0,70
0,66
Table 1: Coherence CV and diversity (10 words) of each topic model.
Figure 2: Macro average annotation accuracy for TWI per model (95% confidence intervals).
Figure 3: Macro average annotation accuracy for TWI per model family.
Figure 4: Adjusting results to account for task difficulty (intruder-top-words similarity).
Figure 5: Macro average task F1 Score for TWM per model (with 95% confidence intervals).
Figure 6: Macro average task F1 Score for TWM per model family.
Dynamic topic models capture evolving word distributions, but traditional coherence metrics may fail when vocabulary changes while semantic meaning persists. We evaluate 120 topics from CoNTM and DLDA across NYT, DBLP, and arXiv, using three human annotators and Low, Medium, and High lexical-change categories. Traditional temporal coherence shows highly variable agreement with human judgments (ρ=-0.256 to 0.614). In contrast, LLM-based semantic similarity agrees strongly with human semantic judgments for CoNTM on NYT (ρ=0.609), DBLP (ρ=0.721), and arXiv (ρ=0.502), but is less consistent for DLDA. Lexical-change stratification reveals variation hidden by aggregate evaluation. We therefore advocate lexical-change-aware evaluation, jointly reporting traditional coherence and LLM-based semantic measures as complementary rather than interchangeable signals.
Charu Karakkaparambil James
RPTU University Kaiserslautern-Landau Kaiserslautern, Germany
Existing topic modeling methods, from LDA to recent neural and LLM-based approaches, which focus mainly on statistical coherence, often produce redundant or off-target topics that miss the user's underlying intent. We introduce Human-centric Topic Modeling, Human-TM), a novel task formulation that integrates a human-provided goal directly into the topic modeling process to produce interpretable, diverse and goal-oriented topics. To tackle this challenge, we propose the Goal-prompted Contrastive Topic Model with Optimal Transport (GCTM-OT), which first uses LLM-based prompting to extract goal candidates from documents, then incorporates these into semantic-aware contrastive learning via optimal transport for topic discovery. Experimental results on three public subreddit datasets show that GCTM-OT outperforms state-of-the-art baselines in topic coherence and diversity while significantly improving alignment with human-provided goals, paving the way for more human-centric topic discovery systems.
Rui Wang, Yi Zheng, Dongxin Wang +5
School of Computer Science, Nanjing University of Posts and Telecommunications · School of Computer Science and Information Engineering, Hefei University of Technology · School of Electronic Engineering and Computer Science, Queen Mary University of London +1
Traditional topic models are effective at uncovering latent themes in large text collections. However, due to their reliance on bag-of-words representations, they struggle to capture semantically abstract features. While some neural variants use richer representations, they are similarly constrained by expressing topics as word lists, which limits their ability to articulate complex topics. We introduce Mechanistic Topic Models (MTMs), a class of topic models that operate on interpretable features learned by sparse autoencoders (SAEs). By defining topics over this semantically rich space, MTMs can reveal deeper conceptual themes with expressive feature descriptions. Moreover, uniquely among topic models, MTMs enable controllable text generation using topic steering vectors. To properly evaluate MTM topics against word list approaches, we propose topic judge, an LLM-based pairwise comparison evaluation framework. Across eight datasets, MTMs match or exceed traditional and neural baselines on coherence metrics, are consistently preferred by topic judge, and enable effective LLM steering.
Carolina Zheng, Nicolas Beltran-Velez, Sweta Karlekar +5
Columbia University · Independent · Google Research