Unsupervised methods are widely used to induce latent semantic structure from large text collections, yet their outputs often contain incoherent, redundant, or poorly grounded clusters that are difficult to validate without labeled data. We propose a reasoning-based refinement framework that leverages large language models (LLMs) not as embedding generators, but as semantic judges that validate and restructure the outputs of arbitrary unsupervised clustering algorithms. Our framework introduces three reasoning stages: (i) coherence verification, where LLMs assess whether cluster summaries are supported by their member texts; (ii) redundancy adjudication, where candidate clusters are merged or rejected based on semantic overlap; and (iii) label grounding, where clusters are assigned interpretable labels through a two-stage process that generates and consolidates semantically similar labels in a fully unsupervised manner. This design decouples representation learning from structural validation and mitigates the common failure modes of embedding-only approaches. We evaluate the framework in real-world social media corpora from two platforms with distinct interaction models, demonstrating consistent improvements in cluster coherence and human-aligned labeling quality over classical topic models and recent representation-based baselines. Human evaluation shows strong agreement with LLM-generated labels, despite the absence of gold-standard annotations. We further conduct robustness analysis under matched temporal and volume conditions to assess cross-platform stability. Beyond empirical gains, our results suggest that LLM-based reasoning can serve as a general mechanism for validating and refining unsupervised semantic structure, enabling more reliable and interpretable analysis of large text collections without supervision.
Figures & tables
Figure 1: Overview of our framework. Unsupervised clustering generates initial cluster proposals that are often noisy. We treat these clusters as hypotheses and use LLMs as semantic judges to (1) verify coherence, (2) adjudicate redundancy, and (3) generate interpretable labels via a two-stage grounding process, producing refined, coherent, and distinct clusters.
Figure 2: Example of the incoherent cluster from X.
Figure 4: Comparing intra-cluster similarity across HDBSCAN, SBERT-refinement, and LLM-refinement.
Model
X Acc. (%)
Bluesky Acc.(%)
LDA
30.4
36.2
BERTopic
38.7
42.4
SBERT
56.2
53.6
TopicGPT
72.8
68.4
Llama 3.2
66.6
60.0
Mistral Large 2
71.6
71.8
Table 2: Assignment comparison w.r.t. human judgment.
Figure 5: UMAP projection of vegan discourse from X (green) and Bluesky (blue) with broader themes.
Broader Theme
Platform
Example Posts
Animal Rights
X
…Stop this cruelty! #AnimalsLivesMatter #animalcruelty #vegan #animalrights PETITION: Justice for Monkey Hanged from Tree …..
Bluesky
Wool industry is infested with violence & documented cruelty in nearly every shearing shed that investigators entered. One of the shearers is seen punching sheep in the nose and face. Another is seen jabbing a sheep………..
Food & Recipes
X
…and I are taco lovers & all things Mexico so we can’t wait to try Veggie Lad’s #vegan recipe…
Bluesky
If you want some next-level delicious tofu, try out this vegan orange chicken! #vegan #vegansky #veganfood #tofu…..
Vegan & Sustainable Products
X
Why you should be using shampoo bars to have healthier hair and protect the environment #ecofriendly #CrueltyFree #vegan…..
Bluesky
Delve into the world of Amazon’s organic beauty with eco-friendly, cruelty-free products that redefine skincare #OrganicBeauty #SustainableBeauty…..
Table 3: Examples of posts by broader theme.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
vegandiet, veganfood, veganlife, veganlover,
veganlifestyle, vegancommunity,
veganfoodshare, vegans, veganfoodlove,
veganjourney, plant-based, cruelty-free,
dairyfree, crueltyfree, meat-free,
govegan, veganfoodporn, animal rights.
Appendix
Table 4: List of the keywords for data collection.
Figure 7: Prompt example of mapping text → theme (Bluesky dataset). The black colored segment is the input prompt and the blue colored segment is the generated output by LLMs.
Dataset
Labels
X
Daily Motivation and Inspiration, Gluten-Free and Vegan Publication Updates, Veganism Advocacy and Lifestyle Promotion, Vegan and Vegetarian Recipes and Cookbook, Veganism and Plant-Based Ethical Lifestyle, Veganism Advocacy and Animal Rights, Vegan Food and Lifestyle Celebrations, Vegan Desserts Recipe, Handmade Vegan Soaps Promotion, Veganism Impacts, Challenges, and Discussions, Vegan and Vegetarian Dining Experiences, Vegan and Gluten-Free Food Promotions, Promotion of Low-Sugar Vegan Tea Products, Promotion of Vegan Haircare Products.
Bluesky
Sustainable and Ethical Beauty Products, Veganism and Ethical Living, Vegan Cooking, Recipes and Cuisine, Humorous Perspectives on Vegan Lifestyles and Food, Vegan Snacks and Treats, Nutrition and Healthy Living, Pet Products and Vegan Options on Amazon, Animal Rights and Advocacy, Vegan Birthday Celebrations, Vegan Products and Community, Critiques and Humor in Social and Political Contexts, Vegan and Plant-Based Trends, Tour Announcements, New Music Releases, The Benefits and Innovations in Plant-Based Solutions, Natural Health Supplements, Vegan Lifestyle and Digital Projects, Social and Ethical Commentary, Plant-based lifestyle and acceptance, Vegan Lifestyle and Events, Plant-Based Living and Community Advocacy, Vegan Lifestyle and Media.
Appendix
Table 6: Generated labels by LLMs from X and Bluesky data.
Figure 8: Distribution of assigned labels/themes using GPT-4o on X (green) and Bluesky (blue) datasets.
Dataset
Test / Comparison
Statistic
p-value
Significance
X
Kruskal–Wallis H-statistic
16.187
0.0003
p<0.001
Mann–Whitney U: HDBSCAN vs LLM
42706.0
0.4808
n.s.
Mann–Whitney U: HDBSCAN vs SBERT
52922.5
0.000165
p<0.001
Mann–Whitney U: SBERT vs LLM
24018.0
0.001829
p<0.01
Bluesky
Kruskal-Wallis H-statistic
0.2809
0.8690
n.s.
Mann–Whitney U: HDBSCAN vs LLM
680.0
0.8816
n.s.
Appendix
Table 7: Statistical test results for cluster metric comparisons. n.s. = statistically not significant (p-value >0.05 ).
Platform
Unique Theme
Example Posts
X
Daily Motivation and Inspiration
5 BEST EXERCISES FOR PEOPLE w/ BACK PAIN #fitness #nutrition #vegan #mindfulness #motivation #inspiration #SaturdayMotivation…..
X
Gluten-Free and Vegan Publication Updates
@TheHecticVegan: The Hectic Vegan Magazine Issue 5 #glutenfree #vegan
Bluesky
Humorous Perspectives on Vegan Lifestyles and Food
Vegan chefs consider it a huge insult when you cut yourself & drip your own blood on to their burgers to enhance the flavor.
Bluesky
Critiques and Humor in Social and Political Contexts
The funny thing about vegans is that they think they’re morally superior for not eating meat because of deforestation when their entire diet is based on exploiting farmers get f****d (moral) vegans.
Appendix
Table 8: Examples of platform-exclusive themes and representative posts.
Platform
Theme
Example Posts
X
veganism impacts, challenges, and discussions
…. y’all all the f**king vegan stables are gone. no rice, no beans, no bread, no chips, no oat milk, no kombucha, no peanut.
I keep striking out in the meat sections of my local grocery stores. This pandemic just might force me into veganism. https://t.co/EwsFYindAn
X
vegan and vegetarian dining experiences
… I’d prob cry if anyone ever cooked vegan for me.
We look forward to hosting our vegetarian cooking class tonight! See you at 18h00! #FreshEarth #CookingClasses http://t.co/WpSz5Uqfki
X
veganism advocacy and lifestyle promotion
People who act like they can’t enjoy vegetarian meals because they eat meat are so annoying.
@Sydney843 @QUBFoodProf @DiscoStew66 Organic vegetables almost always use animal byproducts in fertilisers and therefore are not vegan. Vegans should only ever eat non-organic unless from explicitly from a vegan farm.
Appendix
Table 9: Example posts from X and Bluesky where GPT-4o’s assigned themes do not align with human judgment.
Theme
pearson_corr_with_month
Promotion of vegan haircare products
0.771
Veganism advocacy and animal rights
0.560
Veganism advocacy and lifestyle promotion
0.495
Vegan and glutenfree food promotions
0.459
Veganism impacts challenges and discussions
0.440
Vegan and vegetarian dining experiences
0.352
Appendix
Table 10: Theme–time correlation on X.
Corpus
Platform
#Posts
#Clusters
Silhouette ( ↑ )
Davies–Bouldin ( ↓ )
Full
X
939
14
0.40
3.12
Bluesky
2479
21
0.15
3.06
Balanced 28-day snapshot
X
40
7
0.60
0.61
Bluesky
40
17
0.08
1.14
Appendix
Table 11: Cluster quality metrics by platform and corpus.
Text clustering is a fundamental task in natural language processing, yet traditional clustering algorithms with pre-trained embeddings often struggle in domain-specific contexts without costly fine-tuning. Large language models (LLMs) provide strong contextual reasoning, yet prior work mainly uses them as auxiliary modules to refine embeddings or adjust cluster boundaries. We propose ClusterFusion, a hybrid framework that instead treats the LLM as the clustering core, guided by lightweight embedding methods. The framework proceeds in three stages: embedding-guided subset partition, LLM-driven topic summarization, and LLM-based topic assignment. This design enables direct incorporation of domain knowledge and user preferences, fully leveraging the contextual adaptability of LLMs. Experiments on three public benchmarks and two new domain-specific datasets demonstrate that ClusterFusion not only achieves state-of-the-art performance on standard tasks but also delivers substantial gains in specialized domains. To support future work, we release our newly constructed dataset and results on all benchmarks.
General-purpose embedding models excel at recognizing semantic similarities but fail to capture the characteristics of texts specified by user instructions. In contrast, instruction-tuned embedders can align embeddings with textual instructions yet cannot autonomously infer latent corpus structures, such as determining the optimal number of clusters. To address both limitations, we reframe instruction-following clustering as a generative task and train large reasoning models (LRMs) as autonomous clustering agents. Our reasoning-driven training pipeline enables LRMs to interpret high-level clustering instructions and then infer the corresponding latent groupings. To evaluate this paradigm, we introduce ReasonCluster, a comprehensive benchmark comprising 28 diverse tasks spanning daily dialogue, legal cases, and financial reports. Experiments across diverse datasets and clustering scenarios show that our approach consistently outperforms strong embedding-based methods and LRM baselines, demonstrating that explicit reasoning fosters more faithful and interpretable instruction-based clustering.
Peijun Qing, Puneet Mathur, Nedim Lipka +5
Dartmouth College, USA · Adobe Research, San Jose, USA
In this paper, we propose Precision-Informed Semantic Modeling (PRISM), a structured topic modeling framework combining the benefits of rich representations captured by LLMs with the low cost and interpretability of latent semantic clustering methods. PRISM fine-tunes a sentence encoding model using a sparse set of LLM- provided labels on samples drawn from some corpus of interest. We segment this embedding space with thresholded clustering, yielding clusters that separate closely related topics within some narrow domain. Across multiple corpora, PRISM improves topic separability over state-of-the-art local topic models and even over clustering on large, frontier embedding models while requiring only a small number of LLM queries to train. This work contributes to several research streams by providing (i) a student-teacher pipeline to distill sparse LLM supervision into a lightweight model for topic discovery; (ii) an analysis of the efficacy of sampling strategies to improve local geometry for cluster separability; and (iii) an effective approach for web-scale text analysis, enabling researchers and practitioners to track nuanced claims and subtopics online with an interpretable, locally deployable framework.
Connor Douglas, Utkucan Balci, Joseph Aylett-Bullock
New York University New York, NY, USA · Binghamton University Binghamton, NY, USA · United Nations New York, NY, USA