Behavioral clusters learned over user or item interactions drift as behavior evolves: some fragment into sub-behaviors, some converge, some disappear, and new patterns emerge. Two paradigms dominate the maintenance of evolving clusterings. Threshold triggers fire a pre-specified action when a hardcoded statistic crosses a cutoff; refit-and-track (MONIC-style) re-clusters every window and infers transitions from how the new clusters align with the previous window's. Both hardwire the decision into a threshold or a re-clustering rule. We study an alternative in which a large language model (LLM) makes the lifecycle decision in place. Given a threshold-free, whole-population view of the current cluster set (statistical health, labels, member texts, residual mass), the LLM decides per cluster whether and how to act (split, merge, retire, create, or no-action) without re-clustering. Because the useful signal channel depends on how informative the labels are, we fix a richness-conditioned policy in advance: statistics when label distinctiveness is low, fused statistics+semantics when it is high. On three public datasets spanning label richness (YooChoose 0.17, MOOC 0.50, News-Category 0.70), the policy-selected LLM mode obtains the highest F1 on all three, with far fewer interventions and stronger restraint than tuned threshold heuristics and a MONIC steelman (paired Wilcoxon p <= 0.003, bootstrap 95% CIs excluding zero). The gain is concentrated more in precision than recall. The useful modality shifts from statistics to semantics/fusion as richness rises. Semantics hurt on YooChoose, fusion is roughly neutral on MOOC, and the semantic channel is required on News. One base prompt template and one fixed policy serve all three datasets, whereas the tuned baseline collapses when moved between them. We treat this shift as a practitioner diagnostic observed on three datasets, not a validated law.
Figures & tables
Figure 1. The per-window LLM-arbiter loop. New items are assigned to the current registry (no refit). A threshold-free, whole-population report is built (statistical health + labels + representative member texts + residual mass). The LLM decides all actions in place with written rationale, and decisions are executed deterministically on the registry. Flow diagram of the per-window arbiter loop: new items are assigned to the existing cluster registry with no re-clustering; a threshold-free whole-population report of statistical health, labels, representative member texts, and residual mass is built; a single LLM decides an action for every cluster with a written rationale; and the decisions are executed deterministically on the registry.
Table 2. Main results over 15 seeds (mean ± 95% CI on every column). Best F1 per dataset in bold , and π marks the mode chosen by the richness policy. The Actions column (intervention economy) is also bolded on the π row, showing the policy-selected mode reaches the best F1 with among the fewest interventions of any detecting method. Reading precision and recall separately shows the LLM’s advantage is concentrated in precision: on MOOC the tuned baseline has higher recall ( 0.381 vs. 0.324 ) but roughly half the precision.
Dataset
LLM
Baseline
Δ F1
95% CI
p
MOOC
stats-only
Smart-Trig.
+0.075
[0.038,0.117]
0.0008
News
combined
MONIC
+0.111
[0.068,0.156]
0.0009
YooChoose
stats-only
Smart-Trig.
+0.113
[0.055,0.170]
0.0027
Table 3. Focal significance: best LLM vs. best baseline per dataset (paired, 15 seeds). Δ F1 with 104 -sample bootstrap 95% CI and one-sided paired Wilcoxon p .
Figure 2. Semantic-richness spectrum: LLM channel F1 vs. label richness across the three datasets. Line plot of decision F1 against cluster-label richness for the three LLM channels across the datasets. The statistics-only channel is strongest at low richness while the combined channel becomes strongest at high richness, so the winning channel shifts from statistics to fusion as richness rises.
Dataset
Mode
split
merge
retire
create
YooChoose
stats-only
0.46
–
0.44
–
semantics-only
–
–
–
0.31
combined
0.38
0.04
0.18
–
MOOC
stats-only
0.48
–
0.48
–
semantics-only
–
0.18
0.35
0.49
combined
0.36
–
0.54
–
Table 4. Per-action F1 for the LLM modes (strict scorer, 15 seeds). “–” =0 .
Dataset
Method
split
merge
retire
create
YooChoose
Smart-Triggers (best baseline)
0.566
0.060
–
–
LLM stats-only ( π )
0.460
–
0.439
–
MOOC
Smart-Triggers (best baseline)
0.443
–
0.394
0.210
LLM stats-only ( π )
0.475
–
0.476
–
News
MONIC (best baseline)
0.182
–
0.175
–
LLM combined ( π )
–
0.485
–
–
Table 5. Per-action F1: best baseline vs. the policy-selected LLM mode on each dataset (strict scorer, 15 seeds). Baselines are Smart-Triggers (YooChoose, MOOC) and MONIC (News), each the strongest baseline by aggregate F1 on that dataset. Bold marks the winner of each row, and “–” =0 .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Mode
Real
Shuffle
Generic
News
combined
0.228
0.066
0.000
News
semantics-only
0.155
0.091
0.000
YooChoose
combined
0.176
0.197
0.249
YooChoose
semantics-only
0.083
0.082
0.393
Appendix
Table 6. Label-causality ablation. F1 (mean over 15 seeds) with real, shuffled, and generic labels.
Dataset
LOSO (per-dataset)
Cross-dataset
LLM
MOOC
0.297
0.064
0.348
News
0.242
0.146
0.228
YooChoose
0.308
0.062
0.336
Appendix
Table 7. Same-numeric-feature learned arbiter (F1, 15 seeds). LOSO grants per-dataset supervision the LLM never receives. Cross-dataset matches the LLM’s zero-tuning transfer setting.
Untuned
Oracle
Oracle smart
Held-out
LLM
Dataset
smart
stat
(overfit)
oracle smart
MOOC
0.272
0.133
0.406
0.381
0.348
News
0.000
0.087
0.000
0.000
0.228
YooChoose
0.223
0.105
0.401
0.356
0.336
Appendix
Table 8. Trigger upper bounds vs the LLM (F1, 15 seeds). The oracle columns select thresholds using the evaluation ground truth and are unattainable upper bounds, not deployable methods.
Dataset
Contrast
±1
±2
±3
MOOC
LLM (stats-only)
0.192
0.348
0.348
Best baseline (Smart-Triggers)
0.252
0.272
0.272
News
LLM (combined)
0.173
0.228
0.257
Best baseline (MONIC)
0.075
0.116
0.143
YooChoose
LLM (stats-only)
0.259
0.336
0.336
Best baseline (Smart-Triggers)
0.119
0.223
0.223
Appendix
Table 9. Best-LLM vs. best-baseline decision F1 (mean over 15 seeds) as the matching tolerance varies. The winning method is unchanged on News and YooChoose at every tolerance. On MOOC the LLM wins at ±2 and ±3 , while Smart-Triggers edges it at the tightest ±1 , where the LLM’s ramp-latent detections are clipped most.
Figure 3. Decision F1 by method on each dataset (15 seeds, ± 95% CI). An LLM mode is best on every dataset. Grouped bar chart of decision F1 by method on each of the three datasets, with 95 percent confidence interval bars. An LLM mode has the highest F1 on every dataset.
Figure 4. Paired F1 improvement (LLM − best baseline) with 95% bootstrap CIs, all excluding zero. Bar chart of the paired F1 improvement of the best LLM mode over the best baseline on each dataset, with 95 percent bootstrap confidence interval bars, all lying above zero.
Figure 5. Per-action decision F1 by LLM channel and dataset. Grouped bar chart of per-action decision F1 for split, merge, retire, and create, broken out by LLM channel and dataset, showing which actions each channel detects on each dataset.
Figure 6. Decision F1 vs. number of actions per run. LLM arbiter: accurate and economical. Scatter plot of decision F1 against the number of actions taken per run. The LLM arbiter points fall in the high-F1, low-action region, indicating accurate and economical maintenance.
Figure 7. Conceptual spectrum: as label richness rises, the channel that wins moves from statistics to semantics/fusion. The LLM arbiter beats mechanical baselines throughout with one prompt. Conceptual spectrum showing that as cluster-label richness increases, the best-performing signal channel shifts from statistics-only to fused statistics-plus-semantics, while the LLM arbiter stays ahead of the mechanical baselines throughout using a single prompt.
Figure 8. Three paradigms and where each makes the decision: (a) threshold trigger, (b) MONIC refit-and-track, (c) LLM decides in place (this work). Side-by-side comparison of three lifecycle-maintenance paradigms and where each makes its decision: a threshold trigger fires on a hardcoded statistic, MONIC re-clusters every window and reads transitions off the alignment, and the proposed method places the decision in an LLM acting in place over the whole population with no re-clustering.
Recent text-clustering methods use large language models to propose a cluster taxonomy from a corpus and then assign each text to it. These pipelines are fundamentally programmatic: the sequence of LLM calls and the rules for stopping, merging, and splitting clusters are fixed in code in advance, so they generalise poorly across corpora of different structure and cannot easily incorporate user-supplied constraints such as a target cluster count or a clustering intent. We propose an agentic alternative in which an orchestrator LLM inspects the state of the discovery process at each step and dispatches one of a small set of specialised agents -- proposer, synthesizer, auditor, investigator, and critic -- adapting the pipeline to the corpus rather than executing a fixed one. On seven public text-clustering benchmarks the method achieves state-of-the-art performance, beating the strongest prior baseline by 47% in mean ARI.
Using a Large Language Model (LLM) as the clusterer at production scale is hard: prompts cannot hold the entire label space, and per-document serial processing does not deliver the throughput real workloads require. We present RAILS, a retrieval-augmented incremental LLM clusterer that turns clustering into a simple loop over a growing label pool and scales through document batching with bounded concurrency. On six public benchmarks RAILS exceeds the strongest prior LLM-clustering method on average, lifting accuracy from 51.2% to 59.3%, NMI from 67.2% to 74.8%, and ARI from 45.4% to 54.7%. We further report production-deployment evidence from a SaaS ticket-topic-discovery pipeline, where RAILS has replaced a traditional HDBSCAN stage with higher clustering quality, transparent prompt-driven control, and stateful incremental operation.
Armin Oliya, Aleksandra Sawczuk, Radosław Białobrzeski
Scaling LLM-based applications to millions of users is bottlenecked by the inference cost and latency of modern foundation models. A natural fix is to cluster the inputs and call the LLM only on cluster representatives, letting other members inherit the output -- but this is only safe if each member is measurably close to its representative. Existing clustering methods do not offer such per-sample quality control at scale: none jointly guarantee a minimal within-cluster similarity, exact matching of categorical attributes, and scalability to tens of millions of samples. We propose a two-stage algorithm that generates initial clusters with Mini-batch K-Means, then greedily selects representatives within each initial cluster -- a step equivalent to the Johnson-Chvatal heuristic for Set Cover over alpha-balls in embedding space. The algorithm enforces the similarity and attribute guardrails exactly by construction, and runs in O(nd+n2d/K) time and O(nd+n2/K2) memory for n samples, feature dimension d, and K initial clusters -- linear in n when K grows proportionally with n. We provide benchmarks against common clustering methods on internal and public datasets: our method not only delivers per-sample guardrails but also runs 10-1000x faster and scales to data sizes where most standard methods become intractable. Deployed on 38 million customers for a persona-based recommender, the clustering method cut downstream cost and latency by 50-fold while preserving personalization and unblocked the production launch.