PEARL: Prediction Error-Adaptive Regularized Learning for Selective Forgetting under Partial Preference Drift
Organizations: AWS
Abstract
User preferences shift as needs, context, and tastes evolve. This matters for personalized recommendation, LLM response generation, and RLHF-trained reward models, which can become miscalibrated as preferences drift. Existing non-stationary bandit methods forget uniformly across all dimensions, but real preference shifts are typically partial: some dimensions go stale while others remain valid. Neural contextual bandits accumulate confidence monotonically via the Gram matrix, with no mechanism to direct uncertainty where predictions have gone wrong. We propose PEARL (Prediction Error-Adaptive Regularized Learning), which reverses the Gram update along feature directions where predictions are surprisingly wrong, while leaving confidence intact elsewhere. We prove that (i) any non-adaptive policy suffers linear regret after a preference shift, (ii) no fixed exploration rate achieves low regret in both stationary and non-stationary regimes, and (iii) PEARL's cumulative regret scales with the number of shifted users N_s, not the total population N. On PRISM, a personalized LLM response selection benchmark (1,500 users, 21 LLMs), PEARL beats greedy by +4.0 points and the best neural bandit baseline by +2.5 points (p<0.001), while performing on par with greedy when preferences are stable. It beats sliding-window and discounted approaches by 3 to 10 points, showing that selective forgetting outperforms uniform forgetting. The selection policy is a ~100K-parameter MLP that runs in milliseconds on CPU, leaves LLM weights frozen, and works with any API-based model.
Figures & tables
| Dimension | Spearman | -value | Signal |
|---|---|---|---|
| Helpfulness | 0.178 | Strong | |
| Fluency | 0.154 | Strong | |
| Factuality | 0.134 | Strong | |
| Diversity | 0.100 | Moderate | |
| Safety | 0.086 | Moderate | |
| Personalisation | 0.012 | None |
| Dataset | User ID | User Features | Rating Type | Multi-Model | Users |
|---|---|---|---|---|---|
| PRISM | ✓ | 8-dim survey | Cardinal 1–100 | 21 LLMs | 1,500 |
| Chatbot Arena | — | — | Pairwise | 2 per battle | 200K anon |
| Anthropic HH | — | — | Pairwise | 1 chosen/rej. | Unknown |
| UltraFeedback | — | — | Cardinal 1–5 | 4 models | Crowdworkers |
| OpenAssistant | ✓ | — (inferred) | Ordinal | Tree-struct. | 10K |
| Method | Selection | Pers. | Onl. | Tests |
| PEARL ( , ) | Error-adaptive UCB + bidir. Gram | ✓ | ✓ | Full method |
| NeuralUCB-A ( ) | Error-adaptive UCB, no injection | ✓ | ✓ | Bidir. Gram abl. |
| NeuralUCB-F | Fixed- UCB + Gram | ✓ | ✓ | Fixed exploration |
| No-Online | UCB ( ) | ✓ | — | Online learning |
| NeuralTS ( Zhang et al., 2020 ) | Thompson sampling | ✓ | ✓ | Alt. exploration |
| Greedy | ✓ | ✓ | No exploration |
| Parameter | Value |
|---|---|
| Dataset | PRISM ( Kirk et al., 2024 ) |
| Users | 1,500 |
| Candidate filtering | , reward spread |
| Interactions (after filtering) | 5,011 |
| Hidden dimension ( ) | 128 |
| Dropout | 0.1 |
| Flip | Oracle | PEARL | NUCB-A | NUCB-F | NeuralTS | Greedy | No-Onl. | Random |
|---|---|---|---|---|---|---|---|---|
| 0.25 | .875 | .651 | .649 | .639 | .615 | .631 | .604 | .548 |
| 0.50 | .875 | .641 | .616 | .610 | .593 | .601 | .541 | .542 |
| 0.75 | .875 | .667 | .640 | .632 | .598 | .621 | .480 | .537 |
| Configuration | What it adds | Reward | |
|---|---|---|---|
| No-Online | (static model) | .541 | |
| Greedy | online updates | .601 | .060 |
| NeuralUCB-F | fixed exploration | .610 | .009 |
| NeuralUCB-A | error-adaptive expl. | .616 | .006 |
| PEARL | bidirectional Gram | .641 | .025 |
| Method | Reward | vs. Greedy | 95% CI | |
|---|---|---|---|---|
| Greedy | .684 | — | — | — |
| PEARL ( , ) | .681 | .003 | [ .005, .000] | .066 |
| No-Online | .664 | .020 | [ .021, .018] | .001 |
| Random | .554 | .130 | [ .131, .128] | .001 |
| Method | Reward | vs. PEARL |
|---|---|---|
| PEARL ( , ) | .848 | — |
| NeuralUCB ( ) | .795 | .053 |
| Random | .776 | .072 |
| SW-UCB ( ) | .754 | .094 |
| NeuralTS | .734 | .114 |
| Greedy | .711 | .137 |
| Pretrain | Flip | Greedy | PEARL ( ) | A G |
|---|---|---|---|---|
| 25% | 0.5 | .634 | .664 | .030 |
| 50% (main) | 0.5 | .601 | .641 | .040 |
| 75% | 0.5 | .592 | .627 | .035 |
| Method | Spread 0.1 | Spread 0.2 | Spread 0.3 |
|---|---|---|---|
| Adaptive ( ) | .649 .011 | .639 .019 | .641 .015 |
| Greedy | .625 .008 | .618 .014 | .613 .017 |
| Gap (A G) | .024 | .021 | .028 |
| Method | Flip 0.25 | Flip 0.5 | Flip 0.75 |
|---|---|---|---|
| PEARL ( ) | .651 | .641 | .667 |
| PEARL ( ) | .642 | .612 | .646 |
| Greedy | .631 | .601 | .621 |
| vs. | .009 | .029 | .021 |
| Method | Low div. | Mid div. | High div. |
|---|---|---|---|
| PEARL ( ) | .635 | .617 | .601 |
| Greedy | .625 | .600 | .578 |
| Gap | .010*** | .018*** | .023*** |
| Flip | Method | Epoch 1 | Epoch 3 | Epoch 5 | Epoch 10 |
|---|---|---|---|---|---|
| 0.50 | PEARL ( ) | .544 | .554 | .574 | .641 |
| NeuralUCB-A ( ) | .547 | .558 | .571 | .616 | |
| Greedy | .541 | .545 | .556 | .601 | |
| No-Online | .541 | .541 | .541 | .541 | |
| Random | .541 | .542 | .542 | .542 |
| Flip | Comparison | 95% CI | Sig. | ||
|---|---|---|---|---|---|
| 0.25 | PEARL Greedy | .013 | [ .019, .008] | Greedy | |
| 0.50 | PEARL Greedy | .004 | [ .001, .009] | .118 | n.s. |
| 0.75 | PEARL Greedy | .047 | [ .039, .055] | PEARL |
| Method | Flip 0.25 | Flip 0.5 | Flip 0.75 |
|---|---|---|---|
| PEARL ( ) | .652 [.647,.655] | .642 [.638,.646] | .667 [.660,.671] |
| NUCB-A ( ) | .648 [.645,.653] | .615 [.610,.620] | .640 [.635,.645] |
| NUCB-F | .639 [.635,.643] | .609 [.604,.615] | .632 [.626,.637] |
| NeuralTS | .615 [.613,.616] | .592 [.590,.595] | .596 [.593,.602] |
| Greedy | .631 [.628,.636] | .600 [.591,.609] | .623 [.618,.626] |
| No-Online | .604 [.601,.607] | .542 [.539,.545] | .480 [.478,.483] |
| Method | Reward | vs. PEARL |
|---|---|---|
| PEARL ( , ) | .641 | — |
| NeuralUCB-A ( ) | .616 | .025 |
| NeuralUCB-F | .610 | .031 |
| Greedy | .601 | .040 |
| NeuralTS | .593 | .048 |
| SW-UCB ( ) | .560 | .081 |
| Variant | Reward |
|---|---|
| (NeuralUCB-A) | .616 |
| .645 | |
| (default) | .641 |
| .634 | |
| Auto | .647 |
| Auto | .635 |
| Method | Flip 0.5 |
|---|---|
| PEARL ( , ) | .641 |
| NeuralUCB-A ( ) | .616 |
| D-NeuralUCB ( ) | .614 |
| D-NeuralUCB ( ) | .581 |
| Method | Ep. 1 | Ep. 3 | Ep. 5 | Ep. 7 | Ep. 8 | Ep. 9 | Ep. 10 |
|---|---|---|---|---|---|---|---|
| PEARL ( ) | .648 | .595 | .589 | .608 | .620 | .631 | .641 |
| NeuralUCB-A ( ) | .627 | .589 | .583 | .607 | .618 | .629 | .639 |
| Greedy | .664 | .603 | .570 | .576 | .584 | .594 | .605 |
| Flip 0.25 | 0.5 | 0.75 | |
|---|---|---|---|
| 32 (default) | .652 | .642 | .667 |
| 8 | .631 | .610 | .640 |
| 0 (NeuralUCB-A) | .644 | .612 | .636 |
Non-Stationary Bandits
Non-Stationary Bandits study sequential choices when rewards or constraints change over time, requiring learners to balance exploration with adaptation.
See all 42 papers