Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Bandits

Adam Bayley; Xiaodan Zhu; Raquel Aoki; Yanshuai Cao; Kevin H. Wilson

cs.LGApr 2, 2026

Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Bandits

Authors: Adam Bayley, Xiaodan Zhu, Raquel Aoki, Yanshuai Cao, Kevin H. Wilson

Organizations: Department of Electrical and Computer Engineering, Queen’s University · RBC Borealis

Abstract

The recent advancement of Large Language Models (LLMs) offers new opportunities to generate user preference data to warm-start bandits. Recent studies on contextual bandits with LLM initialization (CBLI) have shown that these synthetic priors can significantly lower early regret. However, these findings assume that LLM-generated choices are reasonably aligned with actual user preferences. In this paper, we systematically examine how LLM-generated preferences perform when random and label-flipping noise is injected into the synthetic training data. For aligned domains, we find that warm-starting remains effective up to 30% corruption, loses its advantage around 40%, and degrades performance beyond 50%. When there is systematic misalignment, even without added noise, LLM-generated priors can lead to higher regret than a cold-start bandit. To explain these behaviors, we develop a theoretical analysis that decomposes the effect of random label noise and systematic misalignment on the prior error driving the bandit's regret, and derive a sufficient condition under which LLM-based warm starts are provably better than a cold-start bandit. We validate these results across multiple conjoint datasets and LLMs, showing that estimated alignment reliably tracks when warm-starting improves or degrades recommendation quality.

Figures & tables

Figure 1: Overview of the CBLI evaluation framework (Noisy-CBLI). An LLM generates synthetic preference data, which is optionally corrupted with random or label-flipping noise at rate $p$ , and used to warm-start a LinUCB bandit that is then fine-tuned on real user data.

Figure 2: Cumulative regret on the COVID-19 Vaccine dataset under preference-flipping noise. “10k_fX” indicates X times 10% of LLM-generated labels flipped. Shaded regions are 95 percent CI over $G=10$ runs.

	Pre-training Size ( $N$ )
Noise ( $p$ )	1k	3k	10k
0%	7.81 $\pm$ 1.31	10.91 $\pm$ 0.81	11.52 $\pm$ 0.75
10%	6.50 $\pm$ 1.09	10.48 $\pm$ 1.19	10.41 $\pm$ 0.90
20%	5.10 $\pm$ 1.50	8.43 $\pm$ 1.15	9.46 $\pm$ 0.92
30%	4.19 $\pm$ 1.71	6.60 $\pm$ 1.11	7.49 $\pm$ 0.93
40%	2.18 $\pm$ 1.75	2.88 $\pm$ 2.01	2.98 $\pm$ 1.62

Table 1: Percentage reduction in cumulative regret (%

\Delta

Regret) for the COVID-19 Vaccine dataset using GPT-4o priors. Reported across three synthetic dataset sizes (

N

). Mean over

G=10

seeds

\pm

95% CI.

Figure 3: Cumulative regret on the COVID-19 Vaccine dataset under random-response noise.

Model	Dataset	Noise (%)	N = 1k	N = 3k	N = 10k
GPT-3.5 Turbo	COVID-19	0	7.20 $\pm$ 1.33	9.06 $\pm$ 1.06	9.45 $\pm$ 1.17
		30	2.81 $\pm$ 3.66	7.06 $\pm$ 1.61	7.44 $\pm$ 1.34
		50	-0.37 $\pm$ 4.16	-3.76 $\pm$ 2.68	-8.07 $\pm$ 3.45
	Immigration	0	0.91 $\pm$ 0.75	0.36 $\pm$ 2.18	0.04 $\pm$ 2.22
		30	0.40 $\pm$ 0.89	-4.72 $\pm$ 3.38	-2.63 $\pm$ 2.54
		50	-0.22 $\pm$ 1.36	-5.09 $\pm$ 4.41	-4.60 $\pm$ 2.73

Table 2: Percentage reduction in cumulative regret (%

\Delta

Regret) compared to cold-start. Reported across 4 models, 3 datasets, and 3 representative noise levels (

p\in\{0,0.3,0.5\}

). Mean over

G=10

seeds.

Model	COVID-19	Immigration	Travel
GPT-3.5 Turbo	51.0	57.1	28.1
GPT-4o	44.4	56.2	26.9
Llama 3.1	46.2	59.2	28.2
Qwen 3	62.3	56.9	28.9

Table 3: Estimated prior error

\widehat{\mathcal{B}}_{0}=\|\theta_{0}-\theta_{\mathrm{real}}\|_{A_{0}}

for each model and dataset. Lower values indicate closer alignment between LLM-synthetic and human preferences in the

A_{0}

-geometry.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Variant	Checkpoint Path	Revision (Hash)
Llama 3.1	8B Instruct	meta-llama/llama-3.1-8B-Instruct	0e9e39f
Qwen 3	8B Instruct	Qwen/Qwen3-8B	b968826
GPT-3.5	Turbo	Proprietary API	Sept–Oct 2025
GPT-4o	Omni	Proprietary API	Sept–Oct 2025

Appendix

Table 4: Model specifications. OpenAI models are listed by access window; open-weights models include their Git revision ID.

Parameter	OpenAI Models	Open-Weights Models
	(GPT-3.5, GPT-4o)	(Llama 3.1, Qwen 3)
Temperature	0.5	0.5
Max Output Tokens	Model Maximum	4,000
Top_p	1.0	1.0
Frequency Penalty	0.0	0.0
Presence Penalty	0.0	0.0

Appendix

Table 5: Inference parameters.

Dataset	Noise (%)	N = 1k	N = 3k	N = 10k
COVID-19	0	17.57 ± 4.29	11.23 ± 2.61	9.45 ± 1.17
COVID-19	30	11.27 ± 3.78	5.91 ± 4.48	7.44 ± 1.34
COVID-19	50	7.74 ± 2.65	1.81 ± 4.20	-8.07 ± 3.45
Immigration	0	13.03 ± 2.37	6.22 ± 1.41	4.03 ± 1.22
Immigration	30	6.49 ± 4.78	1.32 ± 2.83	0.56 ± 1.25
Immigration	50	-2.05 ± 4.16	-15.15 ± 4.12	-17.33 ± 3.50

Appendix

Table 6: GPT-3.5 Turbo results from older snapshot under preference flipping; percentage reduction in cumulative regret ( %

\Delta

Regret ) versus a cold-start LinUCB baseline. Mean over

G=10

seeds

\pm

95 % CI.

Contextual Bandits

Contextual bandits learn which action to take from features of each situation when only the chosen action’s reward is observed.

See all 149 papers

LLM-Derived Priors for Thompson Sampling in Cold-Start Comment Recommendation
Aug 4, 2026Eugene Lee, Oseong Choi, Byungsoo Kang +1Thompson Sampling Cold-Start Recommendation
NAVER WEBTOON South Korea
Agentic-Priors: Using Large Language Models to Address the Cold Start Problem in Preference Elicitation
Oct 15, 2026Bruno Arcanjo, Zhaoxing Li, Sebastian SteinPreference Elicitation User Preference Modeling
University of Southampton, Southampton, United Kingdom
Calibration-Gated LLM Pseudo-Observations for Online Contextual Bandits
Apr 16, 2026Maksim Pershin, Ivan Golovanov, Pavel Baltabaev +1Contextual Bandits Linear Contextual Bandits

Aug 4, 2026J/K move · Enter open · S save

LLM-Derived Priors for Thompson Sampling in Cold-Start Comment Recommendation

Eugene Lee, Oseong Choi, Byungsoo Kang, Taeyeong Jang

NAVER WEBTOON South Korea

Thompson Sampling Cold-Start Recommendation Bayesian Inference Large Language Model-Based Recommendation User Preference Modeling

Open paper AskPDF arXiv

Multi-armed bandit algorithms, especially Thompson sampling, are widely used in online recommendation. Despite their ability to adapt from online feedback, these methods often suffer from cold-start limitations when newly introduced arms have little or no interaction history. In our setting, the candidate arms are user-generated textual comments, whose semantic content can reveal a title's appeal before sufficient interaction feedback is available. We therefore use large language models (LLMs) to extract semantic signals from comment text and convert them into informative Bayesian priors that warm-start Thompson sampling under sparse early-stage feedback. To account for aggregate segment-level differences in response patterns, we maintain and update posteriors separately for each gender-age segment. In a real-world online A/B/C test, we compare a uniform prior with two LLM-based designs: a Gender Prior for demographic-affinity cues and a Content Prior for title-specific identity cues. The results show that LLM-based priors are most beneficial in sparse-feedback regimes -- with the largest gains emerging once a small amount of interaction evidence has accumulated -- and that prior design leads to distinct funnel-level effects. We further analyze prior-reward alignment and demographic heterogeneity, finding that click-oriented alignment is strongest for the Gender Prior and that treatment effects vary substantially across demographic segments. These findings suggest that LLM-derived priors can serve as a practical warm-start mechanism for text-rich bandit recommendation, while also revealing deployment trade-offs.

Abstract

Contextual Bandits

Explore similar work

LLM-Derived Priors for Thompson Sampling in Cold-Start Comment Recommendation

Agentic-Priors: Using Large Language Models to Address the Cold Start Problem in Preference Elicitation

Calibration-Gated LLM Pseudo-Observations for Online Contextual Bandits