User preferences shift as needs, context, and tastes evolve. This matters for personalized recommendation, LLM response generation, and RLHF-trained reward models, which can become miscalibrated as preferences drift. Existing non-stationary bandit methods forget uniformly across all dimensions, but real preference shifts are typically partial: some dimensions go stale while others remain valid. Neural contextual bandits accumulate confidence monotonically via the Gram matrix, with no mechanism to direct uncertainty where predictions have gone wrong. We propose PEARL (Prediction Error-Adaptive Regularized Learning), which reverses the Gram update along feature directions where predictions are surprisingly wrong, while leaving confidence intact elsewhere. We prove that (i) any non-adaptive policy suffers linear regret after a preference shift, (ii) no fixed exploration rate achieves low regret in both stationary and non-stationary regimes, and (iii) PEARL's cumulative regret scales with the number of shifted users N_s, not the total population N. On PRISM, a personalized LLM response selection benchmark (1,500 users, 21 LLMs), PEARL beats greedy by +4.0 points and the best neural bandit baseline by +2.5 points (p<0.001), while performing on par with greedy when preferences are stable. It beats sliding-window and discounted approaches by 3 to 10 points, showing that selective forgetting outperforms uniform forgetting. The selection policy is a ~100K-parameter MLP that runs in milliseconds on CPU, leaves LLM weights frozen, and works with any API-based model.
Figures & tables
Figure 1: (a) Real preference data exhibits partial stationarity: dimensions go stale at different rates. (b) PEARL exploits this structure, recovering faster than Greedy on shifted users.
Dimension
Spearman ρ
p -value
Signal
Helpfulness
+ 0.178
4.3×10−11
Strong
Fluency
+ 0.154
1.2×10−8
Strong
Factuality
+ 0.134
6.8×10−7
Strong
Diversity
+ 0.100
2.1×10−4
Moderate
Safety
+ 0.086
1.5×10−3
Moderate
Personalisation
+ 0.012
0.668
None
Table 1: Per-dimension correlation between stated preference strength and mean prediction error for flipped users (flip 0.5, 20 seeds, n=1,355 flipped users). Positive ρ means stronger stated preference predicts higher post-flip error. Five of eight dimensions are significant.
Dataset
User ID
User Features
Rating Type
Multi-Model
Users
PRISM
✓
8-dim survey
Cardinal 1–100
21 LLMs
1,500
Chatbot Arena
—
—
Pairwise
2 per battle
∼ 200K anon
Anthropic HH
—
—
Pairwise
1 chosen/rej.
Unknown
UltraFeedback
—
—
Cardinal 1–5
4 models
Crowdworkers
OpenAssistant
✓
— (inferred)
Ordinal
Tree-struct.
∼ 10K
Table 2: Preference datasets compared on the four properties needed for personalized bandit evaluation.
Figure 2: Preference heterogeneity in PRISM. (a) Distribution of mean ratings across 1,395 users (std = 11.5), showing substantial inter-user disagreement. (b) Cross-user standard deviation of mean ratings for each of the 21 LLMs; even the most agreed-upon model has std > 13, confirming that the same model output elicits widely divergent evaluations.
Method
Selection
Pers.
Onl.
Tests
PEARL ( γ , η )
Error-adaptive UCB + bidir. Gram
✓
✓
Full method
NeuralUCB-A ( η=0 )
Error-adaptive UCB, no injection
✓
✓
Bidir. Gram abl.
NeuralUCB-F
Fixed- α UCB + Gram
✓
✓
Fixed exploration
No-Online
UCB ( α=1.0 )
✓
—
Online learning
NeuralTS ( Zhang et al., 2020 )
Thompson sampling
✓
✓
Alt. exploration
Greedy
argmaxμk
✓
✓
No exploration
Table 3: Experimental methods and the component each ablates.
Parameter
Value
Dataset
PRISM ( Kirk et al., 2024 )
Users
1,500
Candidate filtering
K=4 , reward spread >0.3
Interactions (after filtering)
5,011
Hidden dimension ( d )
128
Dropout
0.1
Table 4: Full experimental configuration.
Flip
Oracle
PEARL
NUCB-A
NUCB-F
NeuralTS
Greedy
No-Onl.
Random
γ=4
η=0
0.25
.875
.651
.649
.639
.615
.631
.604
.548
0.50
.875
.641
.616
.610
.593
.601
.541
.542
0.75
.875
.667
.640
.632
.598
.621
.480
.537
Table 5: PRISM flip: average reward at epoch 10 (20 seeds). PEARL leads all baselines across flip severities, with the advantage over NeuralUCB-A widening from marginal ( + 0.2 pts) at f=0.25 to + 2.5–2.7 pts at f≥0.5 . Stationary results (flip =0.0 ) in Section 14 .
Configuration
What it adds
Reward
Δ
No-Online
(static model)
.541
Greedy
+ online updates
.601
+ .060
NeuralUCB-F
+ fixed exploration
.610
+ .009
NeuralUCB-A
+ error-adaptive expl.
.616
+ .006
PEARL
+ bidirectional Gram
.641
+ .025
Table 6: Ablation cascade on PRISM (flip 0.5, epoch 10, 20 seeds). Each row adds one mechanism; the Δ column shows its marginal contribution.
Method
Reward
Δ vs. Greedy
95% CI
p
Greedy
.684
—
—
—
PEARL ( γ=4 , η=0.01 )
.681
− .003
[ − .005, + .000]
.066
No-Online
.664
− .020
[ − .021, − .018]
< .001
Random
.554
− .130
[ − .131, − .128]
< .001
Table 7: Stationary performance (flip 0.0, epoch 10, 20 seeds, paired t -test vs. Greedy). PEARL’s exploration does not significantly hurt performance: the 95% CI for PEARL − Greedy spans zero.
Method
Reward
vs. PEARL
PEARL ( γ=100 , η=0.01 )
.848
—
NeuralUCB ( α=20 )
.795
− .053
Random
.776
− .072
SW-UCB ( W=50 )
.754
− .094
NeuralTS
.734
− .114
Greedy
.711
− .137
Table 8: Synthetic simulation under complete preference inversion ( f=1.0 ), epoch 3. PEARL recovers fastest from worst-case staleness, leading all baselines. Config: 10 users, K=10 fixed arms, d=5 , γ=100 , η=0.01 .
Figure 3: Learning dynamics under preference flip (20 seeds, shaded regions show ± 1 standard error). Top row: average reward; bottom row: average regret. At flip 0.5 (left), PEARL ( γ=4.0 ) separates from Greedy mid-training. At flip 0.75 (right), the gap widens further, No-Online collapses below Random, and the adaptive advantage is clear by epoch 3.
Figure 4: Drift recovery trajectories under flip f=0.5 (20 seeds, rolling window of 500 interactions). Left: all users. Center: flipped users only. Right: non-flipped users only. Flipped users start near 0.42 and recover to 0.75; non-flipped users maintain stable performance. PEARL ( γ=4.0 ) separates from Greedy mid-recovery. No-Online flatlines in all conditions.
Pretrain
Flip
Greedy
PEARL ( γ=4.0 )
A − G
25%
0.5
.634
.664
+ .030
50% (main)
0.5
.601
.641
+ .040
75%
0.5
.592
.627
+ .035
Table 9: Pretrain fraction ablation (epoch 10, 20 seeds). PEARL’s advantage over Greedy (A − G) is substantial across conditions. The gap grows with pretrain fraction (the “stale prior” effect): stronger pretraining creates more misleading priors, amplifying the benefit of error-adaptive exploration and bidirectional Gram updates.
Method
Spread > 0.1
Spread > 0.2
Spread > 0.3
Adaptive ( γ=4.0 )
.649 ± .011
.639 ± .019
.641 ± .015
Greedy
.625 ± .008
.618 ± .014
.613 ± .017
Gap (A − G)
+ .024
+ .021
+ .028
Table 10: Filtering sensitivity under flip 0.5 (20 seeds). Method rankings and the adaptive-greedy gap are preserved across filtering thresholds.
Method
Flip 0.25
Flip 0.5
Flip 0.75
PEARL γ=4.0 ( du=40 )
.651
.641
.667
PEARL γ=4.0 ( du=0 )
.642
.612
.646
Greedy
.631
.601
.621
du=40 vs. du=0
+ .009
+ .029
+ .021
Table 11: No-Pers ablation: PEARL γ=4.0 with and without user features (20 seeds, epoch 10). User features help at partial flip (0.25–0.75) by identifying which users need re-exploration.
Method
Low div.
Mid div.
High div.
PEARL ( γ=4.0 )
.635
.617
.601
Greedy
.625
.600
.578
Gap
+ .010***
+ .018***
+ .023***
Table 12: Candidate diversity stratification under flip 0.5 (20 seeds, epoch 10). PEARL’s advantage is present across all diversity levels but amplified by higher candidate diversity. Low-diversity interactions simulate the reduced distinctiveness of single-model Best-of-N candidates. Tercile averages differ from the population average in Table 5 due to unequal interaction counts across groups.
Flip
Method
Epoch 1
Epoch 3
Epoch 5
Epoch 10
0.50
PEARL ( γ=4.0 )
.544
.554
.574
.641
NeuralUCB-A ( η=0 )
.547
.558
.571
.616
Greedy
.541
.545
.556
.601
No-Online
.541
.541
.541
.541
Random
.541
.542
.542
.542
Table 13: Average reward at each epoch checkpoint under preference flips (20 seeds). By epoch 10, PEARL leads at flip 0.5. No-Online is flat across epochs; at flip ≥0.5 it falls below Random.
Flip
Comparison
Δ
95% CI
p
Sig.
0.25
PEARL − Greedy
− .013
[ − .019, − .008]
<.001
Greedy ↑
0.50
PEARL − Greedy
+ .004
[ − .001, + .009]
.118
n.s.
0.75
PEARL − Greedy
+ .047
[ + .039, + .055]
<.001
PEARL ↑
Table 14: Epoch-1 significance tests (20 seeds, paired t -test). PEARL’s epoch-1 advantage is significant only at flip ≥0.75 ; at flip 0.5 the + 0.4-point lead is not significant ( p=0.118 ). At flip 0.25, Greedy leads at epoch 1 ( p<0.001 ). This confirms that PEARL’s value emerges through cumulative online learning, not from a single-pass advantage.
Figure 5: Reward model calibration (20 seeds). Top row: reliability diagrams showing predicted vs. observed reward. Bottom row: prediction distributions. All online-adapted methods (PEARL, Greedy, NeuralTS) achieve excellent calibration (ECE < 0.03) across both flip conditions. No-Online’s calibration degrades sharply (ECE 0.37 at flip 0.5 to 0.42 at flip 0.75) as its stale predictions diverge from shifted preferences.
Method
Flip 0.25
Flip 0.5
Flip 0.75
PEARL ( γ=4 )
.652 [.647,.655]
.642 [.638,.646]
.667 [.660,.671]
NUCB-A ( η=0 )
.648 [.645,.653]
.615 [.610,.620]
.640 [.635,.645]
NUCB-F
.639 [.635,.643]
.609 [.604,.615]
.632 [.626,.637]
NeuralTS
.615 [.613,.616]
.592 [.590,.595]
.596 [.593,.602]
Greedy
.631 [.628,.636]
.600 [.591,.609]
.623 [.618,.626]
No-Online
.604 [.601,.607]
.542 [.539,.545]
.480 [.478,.483]
Table 15: Interquartile Mean (IQM) of average reward at epoch 10 with 95% BCa bootstrap confidence intervals (10,000 resamples), following Agarwal et al. (2021) . IQM averages the middle 50% of runs, reducing sensitivity to outliers.
Figure 6: Performance stratified by user preference divergence (20 seeds, tercile split by EMA prediction error). At flip 0.5, PEARL’s advantage over Greedy is largest for low-divergence users (+3.9 pts), who were well-modeled before the flip and thus most disrupted by it. No-Online collapses across all groups under flip.
Method
Reward
vs. PEARL
PEARL ( γ=4 , η=0.01 )
.641
—
NeuralUCB-A ( η=0 )
.616
− .025
NeuralUCB-F
.610
− .031
Greedy
.601
− .040
NeuralTS
.593
− .048
SW-UCB ( W=1000 )
.560
− .081
Table 16: PRISM comparison under flip 0.5 at epoch 10 (20 seeds). PEARL dominates SW-UCB by 8+ points and alternative reward models by 9–10 points. Setting η=0 (NeuralUCB-A, no bidirectional injection) ablates the Gram update, costing 2.5 points.
Variant
Reward
η=0 (NeuralUCB-A)
.616
η=0.001
.645
η=0.01 (default)
.641
η=0.1
.634
Auto ηbase=0.01
.647
Auto ηbase=0.05
.635
Table 17: Injection rate ablations (20 seeds, epoch 10, flip 0.5). All rows are PEARL variants; η=0 recovers NeuralUCB-A (no injection). The bidirectional update provides + 2.5 pts over η=0 and is robust across an order of magnitude. PEARL-auto scales ηeff=ηbase⋅εˉ/0.5 . Always-inject fires unconditionally, without the error condition.
Method
Flip 0.5
PEARL ( γ=4 , η=0.01 )
.641
NeuralUCB-A ( η=0 )
.616
D-NeuralUCB ( γgram=0.999 )
.614
D-NeuralUCB ( γgram=0.99 )
.581
Table 18: Discounted NeuralUCB vs. PEARL (flip 0.5, epoch 10, 20 seeds). γgram=0.999 matches NeuralUCB-A (uniform mild discount ≈ no discount). γgram=0.99 is too aggressive. PEARL’s directional forgetting dominates.
Method
Ep. 1
Ep. 3
Ep. 5
Ep. 7
Ep. 8
Ep. 9
Ep. 10
PEARL ( γ=4.0 )
.648
.595
.589
.608
.620
.631
.641
NeuralUCB-A ( η=0 )
.627
.589
.583
.607
.618
.629
.639
Greedy
.664
.603
.570
.576
.584
.594
.605
Table 19: Decelerating drift ( Edrift=3 , f=0.75 , 20 seeds): average reward by epoch. PEARL’s exploration investment during the high-drift phase (epochs 1–3) pays off once preferences settle, leading Greedy by + 3.6 pts at epoch 10.
du
Flip 0.25
0.5
0.75
32 (default)
.652
.642
.667
8
.631
.610
.640
0 (NeuralUCB-A)
.644
.612
.636
Table 20: Embedding dimension ablation (epoch 10, IQM, 20 seeds). du=32 (default) significantly outperforms du=8 at partial flip, indicating the embeddings learn useful signal despite data sparsity. du=0 removes all user features.
Non-Stationary Bandits study sequential choices when rewards or constraints change over time, requiring learners to balance exploration with adaptation.
Large language models can answer the same question in different styles, but users' preferred styles are hidden and vary across users and questions. We study inference-time personalization through online selection from a fixed set of prompts, each representing a persona. Users arrive sequentially, each asks a short sequence of questions, and the system displays two responses per question and observes one pairwise choice. We formulate this interaction as a variant of the contextual dueling bandit problem in which the learner starts without preference data, adapts within each session, and exploits information shared across users. The objective is to minimize cumulative strong regret, summing the utility gaps of both displayed responses relative to the user's best response among the available options. We make two contributions. First, we introduce the problem and release a reproducible preference benchmark and dataset built around a tutoring case study, containing educational questions, LLM-generated answers in six teaching styles, and pairwise judgments from LLM-simulated students with hidden preference profiles. We fit Bradley-Terry models to these judgments to obtain reference utilities for regret evaluation. Second, we propose a baseline that learns a mixture prior over users from completed sessions and adapts to each new user within the session. On students and topics never used in development, it removes 44-48% of random pairs' excess regret and achieves over half the performance of a per-user linear oracle fitted in hindsight. We release code, prompts, and the benchmark.
Reinforcement learning from human feedback (RLHF) aligns large language models (LLMs) with human preferences, yet most pipelines learn a single reward model that overlooks individual differences in preferences. Personalized reward models (PRMs) address this by conditioning rewards on user-specific feedback, most commonly through in-context learning (ICL), where a user's historical comparisons are supplied as contextual preference pairs. However, we identify a key limitation of ICL-based PRMs: they fail to capture the preference relations conveyed by contextual pairs. To address this, we propose Preference-Aligned Test-Time Training (P-TTT), which explicitly encodes these relations into user-specific fast weights for personalized reward prediction. P-TTT introduces sequence-level update and apply operations to match the response-level granularity of preference feedback, together with a preference-aligned objective that directly uses pairwise preference relations to guide fast-weight adaptation. Notably, P-TTT is simple to implement and computationally efficient, updating fast weights within a single forward pass without inference-time backpropagation. Extensive experiments show that P-TTT more effectively captures historical preference relations and outperforms state-of-the-art methods by a large margin.
Bohao Wang, Xiaoyan Zhao, Yang Zhang +4
Zhejiang University · National University of Singapore
The recent advancement of Large Language Models (LLMs) offers new opportunities to generate user preference data to warm-start bandits. Recent studies on contextual bandits with LLM initialization (CBLI) have shown that these synthetic priors can significantly lower early regret. However, these findings assume that LLM-generated choices are reasonably aligned with actual user preferences. In this paper, we systematically examine how LLM-generated preferences perform when random and label-flipping noise is injected into the synthetic training data. For aligned domains, we find that warm-starting remains effective up to 30% corruption, loses its advantage around 40%, and degrades performance beyond 50%. When there is systematic misalignment, even without added noise, LLM-generated priors can lead to higher regret than a cold-start bandit. To explain these behaviors, we develop a theoretical analysis that decomposes the effect of random label noise and systematic misalignment on the prior error driving the bandit's regret, and derive a sufficient condition under which LLM-based warm starts are provably better than a cold-start bandit. We validate these results across multiple conjoint datasets and LLMs, showing that estimated alignment reliably tracks when warm-starting improves or degrades recommendation quality.
Adam Bayley, Xiaodan Zhu, Raquel Aoki +2
Department of Electrical and Computer Engineering, Queen’s University · RBC Borealis