Document Topic Alignment Metrics for Evaluating Topic Models of Short-Text Public Health Communications on Social Media

Wangjiaxuan Xin; Shuhua Yin; Yaorong Ge; Shi Chen

cs.CLSep 13, 2026

Document Topic Alignment Metrics for Evaluating Topic Models of Short-Text Public Health Communications on Social Media

Authors: Wangjiaxuan Xin, Shuhua Yin, Yaorong Ge, Shi Chen

Organizations: College of Computing and Informatics The University of North Carolina at Charlotte · Department of Public Health and Health Administration The University of North Carolina at Charlotte

Abstract

Topic models are widely used to analyze public health-related social media short texts, yet their evaluation remains dominated by metrics that focus entirely on generated topics alone. There is a lack of metrics that quantitatively assess whether assigned topics meaningfully represent the corresponding short-text posts. We propose Document-Topic Alignment metrics (DoTA), an assignment-aware evaluation framework comprising metrics that measure semantic alignment between documents (posts) and their assigned topics. We also introduce margin-based and discriminative variants that capture topic assignment confidence and distinguishability. We evaluate DoTA across five topic models on three public health-related social media datasets from X and compare DoTA metrics with conventional topic-based metrics. Results show that DoTA provides complementary evaluation cues and aligns meaningfully with human evaluations. These findings establish the need for assignment-aware evaluation and demonstrate that the addition of DoTA enables a more comprehensive and practically meaningful evaluation for assessing short-text topic modeling performance.

Figures & tables

Figure 1: Topic modeling for social media short texts. A corpus of $N$ short posts is used to generate $K$ topics, each represented by $T$ keywords. Each post is then assigned to exactly one topic.

Metric	Focus	Competition	Interpretation
$DoTA_{\mathrm{base}}$	Absolute Alignment	None	Semantic Fit
$DoTA_{\mathrm{max}}$	Confidence Margin	Hardest Topic	Ambiguity
$DoTA_{\mathrm{disc}}$	Discriminative Separation	All Topics	Distinguishability

Table 1: Comparison of DoTA variants

Figure 2: Illustration of three DoTA metrics for a document $d_{i}$ with assigned topic $t_{z_{i}}$ . (a) $DoTA_{\text{base}}$ measures absolute similarity to the assigned topic. (b) $DoTA_{\text{max}}$ considers the margin between the assigned topic and the most similar competing topic. (c) $DoTA_{\text{disc}}$ compares the similarity to the assigned topic with the average similarity to all other topics.

	COVID-19 ( $K=5$ )					DoxyPEP ( $K=5$ )					Mpox ( $K=5$ )
Model	$C_{v}$	$C_{\text{UMass}}$	$TU$	$TR$	$TD$	$C_{v}$	$C_{\text{UMass}}$	$TU$	$TR$	$TD$	$C_{v}$	$C_{\text{UMass}}$	$TU$	$TR$	$TD$
FASTopic	.403	-6.245	1	0	1	.233	-7.146	1	0	.96	.282	-5.784	1	0	1
BERTopic	.395	-1.803	.6	.386	.6	.228	-5.447	.693	.24	.693	.358	-2.162	.573	.38	.573
TSCTM	.301	-4.597	1	0	1	.279	-6.584	1	0	1	.293	-5.524	1	0	1
LDA	.357	-2.604	.666	.273	.666	.317	-3.797	.906	.046	.906	.399	-2.429	.72	.18	.72
BTM	.373	-1.739	.613	.306	.613	.258	-4.259	.92	.046	.92	.377	-3.7	.773	.173	.773

Table 2: Topic-Based metrics across datasets and topic numbers A, B

	COVID-19 ( $K=5$ )			DoxyPEP ( $K=5$ )			Mpox ( $K=5$ )
Model	$DoTA_{\text{base}}$	$DoTA_{\text{max}}$	$DoTA_{\text{disc}}$	$DoTA_{\text{base}}$	$DoTA_{\text{max}}$	$DoTA_{\text{disc}}$	$DoTA_{\text{base}}$	$DoTA_{\text{max}}$	$DoTA_{\text{disc}}$
FASTopic	.17	-.054	.074	.292	.052	.152	.206	.001	.099
BERTopic	.385	-.017	.06	.466	.014	.103	.54	.043	.099
TSCTM	.21	-.064	.076	.328	.037	.157	.215	-.018	.077
LDA	.365	-.035	.029	.464	.126	.229	.381	-.065	.098
BTM	.34	-.057	.017	.433	.015	.191	.469	-.037	.032

Table 3: DoTA metrics across datasets and topic numbers A, B

Figure 3: Sensitivity of DoTA metrics across different topic numbers based on the COVID-19 dataset.

Figure 4: Spearman’s $\rho$ and Kendall’s $\tau_{b}$ correlations between metrics. The lower triangle shows Spearman’s $\rho$ as filled squares, and the upper triangle shows Kendall’s $\tau_{b}$ as circles, with circle size indicating absolute correlation magnitude.

Metric	Spearman’s $\rho$	Kendall’s $\tau_{b}$
$DoTA_{\text{base}}$	.82	.73
$DoTA_{\text{max}}$	.63	.52
$DoTA_{\text{disc}}$	.67	.53

Table 4: Correlations between DoTA metrics and human ratings (Cohen’s

\kappa

= 0.82)

Topic Modeling

Topic modeling discovers recurring themes and document-topic structure in text, helping organize, compare, and track large collections.

See all 58 papers

Improving Topic Modeling of Social Media Short Texts with Rephrasing: A Case Study of COVID-19 Related Tweets
Oct 21, 2025Wangjiaxuan Xin, Shuhua Yin, Shi Chen +1Social Media Analysis Topic Modeling
College of Computing and Informatics, University of North Carolina at Charlotte, Charlotte, North Carolina, NC, United States · Department of Epidemiology and Community Health, University of North Carolina at Charlotte, Charlotte, NC, United States
When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation
Mar 2, 2026Thibault Prouteau, Francis Lareau, Nicolas Dugué +2Topic Modeling Automated Evaluation
Université de Lorraine, CNRS, LORIA, Nancy (France) · Université de Sherbrooke, Dept of Philosophy and Applied Ethics, Sherbrooke (Canada) · Université du Québec à Montréal, Dept of Philosophy & CIRST, Montréal (Canada) +2
From Documents to Segments: A Contextual Reformulation for Topic Assignment
May 18, 2026Hoonsang Yoon, Takyoung Kim, Wonkee Lee +3Topic Modeling
1LG AI Research · University of Illinois Urbana-Champaign

Oct 21, 2025J/K move · Enter open · S save

Improving Topic Modeling of Social Media Short Texts with Rephrasing: A Case Study of COVID-19 Related Tweets

Wangjiaxuan Xin, Shuhua Yin, Shi Chen, Yaorong Ge

College of Computing and Informatics, University of North Carolina at Charlotte, Charlotte, North Carolina, NC, United States · Department of Epidemiology and Community Health, University of North Carolina at Charlotte, Charlotte, NC, United States

Social Media Analysis Topic Modeling

Open paper AskPDF arXiv

Social media platforms such as Twitter (now X) provide rich data for analyzing public discourse, especially during crises such as the COVID-19 pandemic. However, the brevity, informality, and noise of social media short texts often hinder the effectiveness of traditional topic modeling, producing incoherent or redundant topics that are often difficult to interpret. To address these challenges, we have developed TM-Rephrase, a model-agnostic framework that leverages large language models (LLMs) to rephrase raw tweets into more standardized and formal language prior to topic modeling. Using a dataset of 25,027 COVID-19-related Twitter posts, we investigate the effects of two rephrasing strategies, general- and colloquial-to-formal-rephrasing, on multiple topic modeling methods. Results demonstrate that TM-Rephrase improves three metrics measuring topic modeling performance (i.e., topic coherence, topic uniqueness, and topic diversity) while reducing topic redundancy of most topic modeling algorithms, with the colloquial-to-formal strategy yielding the greatest performance gains and especially for the Latent Dirichlet Allocation (LDA) algorithm. This study contributes to a model-agnostic approach to enhancing topic modeling in public health related social media analysis, with broad implications for improved understanding of public discourse in health crisis as well as other important domains.

Abstract

Topic Modeling

Explore similar work

Improving Topic Modeling of Social Media Short Texts with Rephrasing: A Case Study of COVID-19 Related Tweets

When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation

From Documents to Segments: A Contextual Reformulation for Topic Assignment