Reasoning-Based Refinement of Unsupervised Text Clusters with LLMs

Tunazzina Islam

cs.CLApr 8, 2026

Reasoning-Based Refinement of Unsupervised Text Clusters with LLMs

Authors: Tunazzina Islam

Organizations: Department of Computer Science, Purdue University, West Lafayette, IN 47907

Abstract

Unsupervised methods are widely used to induce latent semantic structure from large text collections, yet their outputs often contain incoherent, redundant, or poorly grounded clusters that are difficult to validate without labeled data. We propose a reasoning-based refinement framework that leverages large language models (LLMs) not as embedding generators, but as semantic judges that validate and restructure the outputs of arbitrary unsupervised clustering algorithms. Our framework introduces three reasoning stages: (i) coherence verification, where LLMs assess whether cluster summaries are supported by their member texts; (ii) redundancy adjudication, where candidate clusters are merged or rejected based on semantic overlap; and (iii) label grounding, where clusters are assigned interpretable labels through a two-stage process that generates and consolidates semantically similar labels in a fully unsupervised manner. This design decouples representation learning from structural validation and mitigates the common failure modes of embedding-only approaches. We evaluate the framework in real-world social media corpora from two platforms with distinct interaction models, demonstrating consistent improvements in cluster coherence and human-aligned labeling quality over classical topic models and recent representation-based baselines. Human evaluation shows strong agreement with LLM-generated labels, despite the absence of gold-standard annotations. We further conduct robustness analysis under matched temporal and volume conditions to assess cross-platform stability. Beyond empirical gains, our results suggest that LLM-based reasoning can serve as a general mechanism for validating and refining unsupervised semantic structure, enabling more reliable and interpretable analysis of large text collections without supervision.

Figures & tables

Figure 1: Overview of our framework. Unsupervised clustering generates initial cluster proposals that are often noisy. We treat these clusters as hypotheses and use LLMs as semantic judges to (1) verify coherence, (2) adjudicate redundancy, and (3) generate interpretable labels via a two-stage grounding process, producing refined, coherent, and distinct clusters.

Figure 2: Example of the incoherent cluster from X.

Panel of Figure 3 — Figure 3: Example of merged cluster summaries.

Dataset	Metric	HDBSCAN	SBERT-rf	LLM-rf
X	$C$	359	250	232
	$S$ ( $\uparrow$ )	0.122	0.156	0.674
	$DB_{i}$ ( $\downarrow$ )	2.322	0.569	0.635
Bluesky	$C$	37	34	36
	$S$ ( $\uparrow$ )	-0.017	0.052	0.979
	$DB_{i}$ ( $\downarrow$ )	2.739	0.282	0.227

Table 1: Cluster quality comparison. rf: refinement.

Panel of Figure 4 — Figure 4: Comparing intra-cluster similarity across HDBSCAN, SBERT-refinement, and LLM-refinement.

Model	X Acc. (%)	Bluesky Acc.(%)
LDA	30.4	36.2
BERTopic	38.7	42.4
SBERT	56.2	53.6
TopicGPT	72.8	68.4
Llama 3.2	66.6	60.0
Mistral Large 2	71.6	71.8

Table 2: Assignment comparison w.r.t. human judgment.

Figure 5: UMAP projection of vegan discourse from X (green) and Bluesky (blue) with broader themes.

Broader Theme	Platform	Example Posts
Animal Rights	X	…Stop this cruelty! #AnimalsLivesMatter #animalcruelty #vegan #animalrights PETITION: Justice for Monkey Hanged from Tree …..
	Bluesky	Wool industry is infested with violence & documented cruelty in nearly every shearing shed that investigators entered. One of the shearers is seen punching sheep in the nose and face. Another is seen jabbing a sheep………..
Food & Recipes	X	…and I are taco lovers & all things Mexico so we can’t wait to try Veggie Lad’s #vegan recipe…
	Bluesky	If you want some next-level delicious tofu, try out this vegan orange chicken! #vegan #vegansky #veganfood #tofu…..
Vegan & Sustainable Products	X	Why you should be using shampoo bars to have healthier hair and protect the environment #ecofriendly #CrueltyFree #vegan…..
	Bluesky	Delve into the world of Amazon’s organic beauty with eco-friendly, cruelty-free products that redefine skincare #OrganicBeauty #SustainableBeauty…..

Table 3: Examples of posts by broader theme.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

vegandiet, veganfood, veganlife, veganlover,

veganlifestyle, vegancommunity,

veganfoodshare, vegans, veganfoodlove,

veganjourney, plant-based, cruelty-free,

dairyfree, crueltyfree, meat-free,

govegan, veganfoodporn, animal rights.

Appendix

Table 4: List of the keywords for data collection.

Dataset	$\tau$ $(C,S,DB_{i})$
X	0.75 (126, 0.378, 1.037), 0.80 (174, 0.519, 0.838),
	0.85 (232, 0.674, 0.635) , 0.90 (292, 0.825, 0.493)
Bluesky	0.75 (22, 0.589, 0.852), 0.80 (32, 0.859, 0.498),
	0.85 (36, 0.979, 0.227) , 0.90 (37, 1.0, $6.2e-08$ )

Appendix

Table 5: Threshold selection using grid search.

Figure 6: Prompt templates (shown as zero-shot).

Figure 7: Prompt example of mapping text $\rightarrow$ theme (Bluesky dataset). The black colored segment is the input prompt and the blue colored segment is the generated output by LLMs.

Dataset	Labels
X	Daily Motivation and Inspiration, Gluten-Free and Vegan Publication Updates, Veganism Advocacy and Lifestyle Promotion, Vegan and Vegetarian Recipes and Cookbook, Veganism and Plant-Based Ethical Lifestyle, Veganism Advocacy and Animal Rights, Vegan Food and Lifestyle Celebrations, Vegan Desserts Recipe, Handmade Vegan Soaps Promotion, Veganism Impacts, Challenges, and Discussions, Vegan and Vegetarian Dining Experiences, Vegan and Gluten-Free Food Promotions, Promotion of Low-Sugar Vegan Tea Products, Promotion of Vegan Haircare Products.
Bluesky	Sustainable and Ethical Beauty Products, Veganism and Ethical Living, Vegan Cooking, Recipes and Cuisine, Humorous Perspectives on Vegan Lifestyles and Food, Vegan Snacks and Treats, Nutrition and Healthy Living, Pet Products and Vegan Options on Amazon, Animal Rights and Advocacy, Vegan Birthday Celebrations, Vegan Products and Community, Critiques and Humor in Social and Political Contexts, Vegan and Plant-Based Trends, Tour Announcements, New Music Releases, The Benefits and Innovations in Plant-Based Solutions, Natural Health Supplements, Vegan Lifestyle and Digital Projects, Social and Ethical Commentary, Plant-based lifestyle and acceptance, Vegan Lifestyle and Events, Plant-Based Living and Community Advocacy, Vegan Lifestyle and Media.

Appendix

Table 6: Generated labels by LLMs from X and Bluesky data.

Figure 8: Distribution of assigned labels/themes using GPT-4o on X (green) and Bluesky (blue) datasets.

Dataset	Test / Comparison	Statistic	p-value	Significance
X	Kruskal–Wallis H-statistic	16.187	0.0003	$p<0.001$
	Mann–Whitney U: HDBSCAN vs LLM	42706.0	0.4808	n.s.
	Mann–Whitney U: HDBSCAN vs SBERT	52922.5	0.000165	$p<0.001$
	Mann–Whitney U: SBERT vs LLM	24018.0	0.001829	$p<0.01$
Bluesky	Kruskal-Wallis H-statistic	0.2809	0.8690	n.s.
	Mann–Whitney U: HDBSCAN vs LLM	680.0	0.8816	n.s.

Appendix

Table 7: Statistical test results for cluster metric comparisons. n.s. = statistically not significant (p-value

>0.05

Platform	Unique Theme	Example Posts
X	Daily Motivation and Inspiration	5 BEST EXERCISES FOR PEOPLE w/ BACK PAIN #fitness #nutrition #vegan #mindfulness #motivation #inspiration #SaturdayMotivation…..
X	Gluten-Free and Vegan Publication Updates	@TheHecticVegan: The Hectic Vegan Magazine Issue 5 #glutenfree #vegan
Bluesky	Humorous Perspectives on Vegan Lifestyles and Food	Vegan chefs consider it a huge insult when you cut yourself & drip your own blood on to their burgers to enhance the flavor.
Bluesky	Critiques and Humor in Social and Political Contexts	The funny thing about vegans is that they think they’re morally superior for not eating meat because of deforestation when their entire diet is based on exploiting farmers get f****d (moral) vegans.

Appendix

Table 8: Examples of platform-exclusive themes and representative posts.

Platform	Theme	Example Posts
X	veganism impacts, challenges, and discussions	…. y’all all the f**king vegan stables are gone. no rice, no beans, no bread, no chips, no oat milk, no kombucha, no peanut.
		I keep striking out in the meat sections of my local grocery stores. This pandemic just might force me into veganism. https://t.co/EwsFYindAn
X	vegan and vegetarian dining experiences	… I’d prob cry if anyone ever cooked vegan for me.
		We look forward to hosting our vegetarian cooking class tonight! See you at 18h00! #FreshEarth #CookingClasses http://t.co/WpSz5Uqfki
X	veganism advocacy and lifestyle promotion	People who act like they can’t enjoy vegetarian meals because they eat meat are so annoying.
		@Sydney843 @QUBFoodProf @DiscoStew66 Organic vegetables almost always use animal byproducts in fertilisers and therefore are not vegan. Vegans should only ever eat non-organic unless from explicitly from a vegan farm.

Appendix

Table 9: Example posts from X and Bluesky where GPT-4o’s assigned themes do not align with human judgment.

Theme	pearson_corr_with_month
Promotion of vegan haircare products	0.771
Veganism advocacy and animal rights	0.560
Veganism advocacy and lifestyle promotion	0.495
Vegan and glutenfree food promotions	0.459
Veganism impacts challenges and discussions	0.440
Vegan and vegetarian dining experiences	0.352

Appendix

Table 10: Theme–time correlation on X.

Corpus	Platform	#Posts	#Clusters	Silhouette ( $\uparrow$ )	Davies–Bouldin ( $\downarrow$ )
Full	X	939	14	0.40	3.12
Full	Bluesky	2479	21	0.15	3.06
Balanced 28-day snapshot	X	40	7	0.60	0.61
Balanced 28-day snapshot	Bluesky	40	17	0.08	1.14

Appendix

Table 11: Cluster quality metrics by platform and corpus.

Figure 9: Theme distribution in balanced corpora.

Clustering

Clustering groups data points by shared structure, developing methods to discover, validate, and interpret groups across diverse data and constraints.

See all 345 papers

ClusterFusion: Hybrid Clustering with Embedding Guidance and LLM Adaptation
Dec 4, 2025Yiming Xu, Yuan Yuan, Vijay Viswanathan +1Clustering Text Clustering
Adobe · Carnegie Mellon University
Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents
Mar 6, 2026Peijun Qing, Puneet Mathur, Nedim Lipka +5Clustering Unsupervised Clustering
Dartmouth College, USA · Adobe Research, San Jose, USA
PRISM: LLM-Guided Semantic Clustering for High-Precision Topics
Apr 3, 2026Connor Douglas, Utkucan Balci, Joseph Aylett-BullockSemantic Clustering Topic Modeling
New York University New York, NY, USA · Binghamton University Binghamton, NY, USA · United Nations New York, NY, USA

Dec 4, 2025J/K move · Enter open · S save

ClusterFusion: Hybrid Clustering with Embedding Guidance and LLM Adaptation

Yiming Xu, Yuan Yuan, Vijay Viswanathan, Graham Neubig

Adobe · Carnegie Mellon University

Clustering Text Clustering Semantic Clustering

Open paper AskPDF arXiv

Text clustering is a fundamental task in natural language processing, yet traditional clustering algorithms with pre-trained embeddings often struggle in domain-specific contexts without costly fine-tuning. Large language models (LLMs) provide strong contextual reasoning, yet prior work mainly uses them as auxiliary modules to refine embeddings or adjust cluster boundaries. We propose ClusterFusion, a hybrid framework that instead treats the LLM as the clustering core, guided by lightweight embedding methods. The framework proceeds in three stages: embedding-guided subset partition, LLM-driven topic summarization, and LLM-based topic assignment. This design enables direct incorporation of domain knowledge and user preferences, fully leveraging the contextual adaptability of LLMs. Experiments on three public benchmarks and two new domain-specific datasets demonstrate that ClusterFusion not only achieves state-of-the-art performance on standard tasks but also delivers substantial gains in specialized domains. To support future work, we release our newly constructed dataset and results on all benchmarks.

Abstract

Clustering

Explore similar work

ClusterFusion: Hybrid Clustering with Embedding Guidance and LLM Adaptation

Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents

PRISM: LLM-Guided Semantic Clustering for High-Precision Topics