Challenges and Solutions for Bandits in the Wild: Warm-Started Mixture Bandits for Cross-Cohort Slate Recommendation

Serafima Lebedeva; Sumantrak Mukherjee; Ali Arshad Sadal; Ilias Ekşi; Rahul Sharma; Julia Mueller; Theresa Dombrowski; Jakob Karolus

cs.LGSep 29, 2026

Challenges and Solutions for Bandits in the Wild: Warm-Started Mixture Bandits for Cross-Cohort Slate Recommendation

Authors: Serafima Lebedeva, Sumantrak Mukherjee, Ali Arshad Sadal, Ilias Ekşi, Rahul Sharma, Julia Mueller, Theresa Dombrowski, Jakob Karolus, +3 more

Organizations: German Research Center for Artificial Intelligence (DFKI), Kaiserslautern · RPTU, Kaiserslautern-Landau · LMU, Munich

Abstract

Many recommender services repeatedly encounter cold-start cohorts, where new users arrive with little or no interaction history. This creates two challenges: learning user preferences quickly from limited feedback and sustaining useful recommendations when each user has a finite catalog that can become repetitive or depleted over time. We propose CohortMix-TS, a warm-started mixture bandit that learns latent user groups from earlier cohorts and uses available metadata to construct group-informed priors for new users. Starting from these fixed priors, the model personalizes independently as feedback from each user becomes available. Session slates combine Thompson sampling with diversity and inventory-depletion controls. We evaluate CohortMix-TS through simulation, semi-synthetic experiments, and a 25-day randomized in-the-wild deployment with 713 registered participants in a Campus Games quiz application. Our evaluations show that cross-cohort transfer improves early recommendation quality and user-level regret, while inventory-aware slate construction helps prevent premature exhaustion of preferred items. In the field deployment, treatment users also showed a larger early-to-late change in correctness than users receiving random recommendations. Together, these results show how warm-start transfer and inventory-aware recommendations can support personalization for short-lived, repeatedly cold-starting cohorts.

Figures & tables

Figure 1. One diagnostic for each generated environment. (a) Transfer expected reward at alignment

\chi=0.75

for CohortMix-TS , Cold-start TS, Metadata LinUCB, and Metadata LinTS; higher

\chi

means closer agreement between historical and target preferences. (b) Preferred-arm inventory use when that arm contains

20\%

of the

TK

item opportunities for No controls, Diversity only, Depletion only, and Full selector. (c) Real-catalog scarcity stress in the 2024–2025 year-split environment. Each line joins five consecutive five-session blocks. Both coordinates average over users whose generated optimal arm is the actual 98-item arm: expected 10-item slate reward (horizontal) and the percentage with an unseen item remaining in that arm before the session slate is selected (vertical). Open markers denote sessions 1–5 and filled terminal markers denote sessions 21–25. Shaded bands in (a)–(b) are percentile bootstrap 95% intervals across 160 cohorts; (c) reports cohort-average trajectories across 96 cohorts. Three panels. The first plots expected reward over 25 sessions for full CohortMix-TS, Cold-start TS, Metadata LinUCB, and Metadata LinTS. The full CohortMix-TS line is highest by the end. The second plots preferred-arm inventory consumed over 25 sessions for four slate-control variants; no controls and diversity only exhaust the arm early, while depletion-aware variants use it gradually. The third plots five connected positions for full CohortMix-TS, Cold-start TS, No depletion, and Random in a real-catalog scarcity stress. Its horizontal axis is expected reward of a ten-item slate for users whose generated optimal arm is a 98-item arm. Its vertical axis is the percentage of those users with an unseen item remaining in that arm before their session slate is selected. An open marker is the first block and a larger filled terminal marker is the fifth.

Policy	Early $\uparrow$	Campaign $\uparrow$	Minority $\uparrow$	$P90$ regret $\downarrow$
Full CohortMix-TS	0.622	0.669	0.653	19.18
Warm TS (fixed mixture)	0.579	0.642	0.628	24.39
Hard-cluster TS	0.592	0.609	0.506	62.77
Cold-start TS	0.576	0.639	0.638	24.73
Static source mixture	0.585	0.585	0.400	87.47
Metadata LinUCB	0.541	0.596	0.505	65.23

Table 1. Parametric transfer results at alignment

\chi=0.75

, averaged over 160 generated cohorts. Rewards are expected per displayed item; Minority is campaign reward for the lower-probability hidden subtype. P90 pseudo-regret is the 90th percentile of per-user pseudo-regret at the final session. Bold marks the best result and underline the second-best result in each column.

Policy	Early $\uparrow$	Campaign $\uparrow$	Late $\uparrow$	Pseudo-regret $\downarrow$
CohortMix-TS	0.597	0.627	0.632	37.71
Cold-start TS	0.584	0.618	0.628	39.94
Metadata LinUCB	0.580	0.590	0.601	46.84
Metadata LinTS	0.580	0.594	0.598	45.99
Hard membership	0.583	0.615	0.625	40.59
Global prior	0.584	0.615	0.624	40.67

Table 2. Year-split data-calibrated semi-synthetic results over 96 generated cohorts. Source-only 2024 structure initializes each policy; 2025 records calibrate hidden target preferences. Reward entries are expected per displayed item; pseudo-regret is per user over 250 displayed items. The main benchmark turns slate penalties off to isolate transfer; Figure 1 (c) reports the separate real-catalog scarcity stress. Bold marks the best result and underline the second-best result in each column.

Figure 2. In-the-wild deployment outcomes. (a) Participant-level early–late daily-correctness change

\Delta_{u}

in the restricted, complete-window analysis (

n=52

; treatment

=23

, control

=29

). Points are users; diamonds and bars are group means and bootstrap 95% intervals. The treatment–control difference is

0.062

(95% interval

[0.005,0.119]

;

p=0.043

). (b) Daily mean correct/answered rate, averaged within each responding user-day; (c) daily active users; and (d) active-day survival. Panels (b)–(d) are descriptive all-registrant traces (

n=713

) after removing technical-failure dates and have changing daily denominators. Blue denotes random control and orange denotes CohortMix-TS . Four panels. The first uses violin plots, points, and mean confidence intervals to show early–late daily correctness change for 29 random-control and 23 CohortMix-TS users. The treatment distribution and mean are above the control distribution and mean. The second panel shows descriptive participant-day correct-answer rates, the third the number of daily active users, and the fourth the share of all registrants with at least a given number of active quiz days. Blue represents control and orange represents CohortMix-TS throughout.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Policy	Early $\uparrow$	Campaign $\uparrow$	Late $\uparrow$	Early exhaust $\downarrow$
No controls	.4386	.4333	.4320	1.00
Diversity only	.5244	.5002	.4506	1.00
Depletion only	.4756	.4790	.4834	.00
No adaptive controller	.5177	.5081	.4983	.00
Full selector	.5168	.5083	.4996	.00

Appendix

Table 3. Ablation and sensitivity details. (Left) finite-inventory component ablation at

m/(TK)=.20

; rewards are expected per displayed item and Early exhaust is the fraction of users whose preferred arm is exhausted before the final week. “No adaptive controller” retains

\gamma=.5

and

\delta=1

but sets

\phi=0

. (Right) the early-reward difference between CohortMix-TS and Cold-start TS in the transfer environment; positive values favor CohortMix-TS . Bold marks the best and underline the second-best result within each left-hand reward column.

Figure 3. Source-only group–arm variation conditional on metadata. Within the three highest-support metadata categories that split into multiple inferred groups, each cell is a group’s leave-group-out correctness residual (percentage points) relative to other historical users with the same metadata category and item arm. A–C denote anonymized metadata categories; rows are ordered by source answer support. The gray cell has fewer than 30 observations. A five-column heatmap for historical item arms one through five. Nine rows are grouped into three anonymized metadata categories A, B, and C. Cells show positive or negative leave-group-out correctness residuals in percentage points relative to users with the same metadata category; colors range from blue negative values through white near zero to red positive values. One low-support cell is gray.

Explore similar work

LLM-Derived Priors for Thompson Sampling in Cold-Start Comment Recommendation
Aug 4, 2026Eugene Lee, Oseong Choi, Byungsoo Kang +1Thompson Sampling Large Language Model Personalization
NAVER WEBTOON South Korea
Denoising Implicit Feedback for Cold-start Recommendation
Jun 17, 2026Gaode Chen, Shicheng Wang, Shikun Li +8Cold-Start Noisy Labels
Kuaishou Technology Beijing, China · Hong Kong Baptist University Hong Kong, China · Institute of Information Engineering, Chinese Academy of Sciences Beijing, China
Impatient Bandits: Optimizing for the Long-Term Without Delay
Jan 14, 2025Kelly W. Zhang, Thomas Baldwin-McDonald, Kamil Ciosek +2Bandits Long-Term
Imperial College London · University of Manchester · Spotify +2

Aug 4, 2026, cs.IRJ/K move · Enter open · S save

LLM-Derived Priors for Thompson Sampling in Cold-Start Comment Recommendation

Eugene Lee, Oseong Choi, Byungsoo Kang, Taeyeong Jang

NAVER WEBTOON South Korea

Thompson Sampling Large Language Model Personalization Bandits Feedback Gender

Open paper AskPDF arXiv

Multi-armed bandit algorithms, especially Thompson sampling, are widely used in online recommendation. Despite their ability to adapt from online feedback, these methods often suffer from cold-start limitations when newly introduced arms have little or no interaction history. In our setting, the candidate arms are user-generated textual comments, whose semantic content can reveal a title's appeal before sufficient interaction feedback is available. We therefore use large language models (LLMs) to extract semantic signals from comment text and convert them into informative Bayesian priors that warm-start Thompson sampling under sparse early-stage feedback. To account for aggregate segment-level differences in response patterns, we maintain and update posteriors separately for each gender-age segment. In a real-world online A/B/C test, we compare a uniform prior with two LLM-based designs: a Gender Prior for demographic-affinity cues and a Content Prior for title-specific identity cues. The results show that LLM-based priors are most beneficial in sparse-feedback regimes -- with the largest gains emerging once a small amount of interaction evidence has accumulated -- and that prior design leads to distinct funnel-level effects. We further analyze prior-reward alignment and demographic heterogeneity, finding that click-oriented alignment is strongest for the Gender Prior and that treatment effects vary substantially across demographic segments. These findings suggest that LLM-derived priors can serve as a practical warm-start mechanism for text-rich bandit recommendation, while also revealing deployment trade-offs.

Abstract

Explore similar work

LLM-Derived Priors for Thompson Sampling in Cold-Start Comment Recommendation

Denoising Implicit Feedback for Cold-start Recommendation

Impatient Bandits: Optimizing for the Long-Term Without Delay