Adaptive Personalization Under Preference Epistemic Uncertainty
Organizations: Meta Superintelligence Labs · University of Southern California
Abstract
Human preferences change over time: an established preference may remain valid over time, become uncertain as context shifts, or be explicitly revised. Effective personalization requires conversational AI to adapt its response strategy to evolving preferences rather than relying on passive memory retrieval. We formalize this capability as adaptive personalization under epistemic uncertainty. A human preference study further confirms that people favor state-contingent response strategies as preference uncertainty changes. To evaluate whether language models mirror this adaptation, we create a multilingual benchmark spanning 250 personas, 5 languages, and 25K conversational turns that decouples strategy selection from response execution. Across eight closed- and open-weight models, we uncover a recognition-generation gap: models reliably identify appropriate strategies when selecting among candidate options, but collapse during open-ended generation, suffering a 35-46% drop in strategy calibration. Crucially, this failure is systematic: models default to exploit-first behavior, over-relying on existing preferences regardless of preference epistemic uncertainty. We show this failure is addressable: while prompting interventions fail, targeted training that supervises models' internal strategy representations toward state-conditional human preferences narrows the user acceptability generalization gap by a third on out-of-distribution personas and conversational styles.
Figures & tables
| Strategy | Established | Uncertain | Revised |
|---|---|---|---|
| Exploit | 90.4%✓ | 8.5% | 6.2% |
| Elicit | 2.4% | 27.0%✓ | 20.0%✓ |
| Confirm | 2.0% | 28.4%✓ | 24.6%✓ |
| Hedge | 4.0% | 26.6%✓ | 23.0%✓ |
| Recalibrate | 1.2% | 9.5% | 26.2%✓ |
| Model | SC (%) | R (1-5) | UA (%) |
|---|---|---|---|
| GPT-5.4 | 73 ±3 | 3.82 ±0.12 | 80 ±4 |
| Opus | 63 ±3 | 2.44 ±0.14 | 65 ±4 |
| Grok | 61 ±4 | 2.23 ±0.11 | 60 ±5 |
| Gemini | 57 ±4 | 2.80 ±0.13 | 55 ±5 |
| Llama | 55 ±4 | 2.64 ±0.15 | 35 ±4 |
| Gemma | 51 ±4 | 3.09 ±0.14 | 48 ±5 |
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
| Strategy | Valid state | Example response |
|---|---|---|
| Exploit | Established | “There’s a good bakery two blocks down — their mushroom and gruyère galette is the standout.” |
| Hedge | Uncertain | “Two options nearby: a savory galette place, or a deli with hot sandwiches. Either work?” |
| Elicit | Uncertain | “Before I suggest somewhere — are you after something hot and substantial, or lighter?” |
| Confirm | Uncertain | “You’ve mentioned preferring savory — is that still what you’re in the mood for today?” |
| Recalibrate | Revised | “You’d mentioned avoiding sweet things before, but it sounds like that’s shifted — let me start fresh.” |
| # | Life domain | # | Life domain |
|---|---|---|---|
| 1 | Alarm | 11 | Messaging |
| 2 | Books | 12 | Movies |
| 3 | Buses | 13 | Music |
| 4 | Calendar | 14 | Rental cars |
| 5 | Events | 15 | Restaurants |
| 6 | Finance | 16 | Services |
| # | Interaction style |
|---|---|
| 1 | Advice delivery |
| 2 | Apology style |
| 3 | Argumentation and evidence |
| 4 | Conflict resolution |
| 5 | Creative collaboration |
| 6 | Cultural sensitivity |
| Field | Value |
|---|---|
| Seed | “An alumni relations coordinator who maintains graduate records and organises reunion events” |
| Demographics | Beverley Thistlethwaite; 54; female; White British; United Kingdom; Coventry, West Midlands; bachelors; Alumni Relations Coordinator, University of Coventry; lower-middle income; partnered, lives with long-term partner and two rescue greyhounds; moderate tech savviness |
| Life narrative | “Spend ten minutes with Beverley and you will learn the names of both greyhounds, the weather forecast for the weekend, and her opinion on the new self-service coffee machine in the staff room — and this before she gets to the actual point of the conversation. She describes herself, with genuine conviction, as someone who cannot stand idle chatter and who prefers to get straight to business, a self-image that bears absolutely no resemblance to how she operates in practice. Colleagues have quietly learned to budget an extra quarter-hour for any meeting she calls…” |
| Friction traits | (i) stubbornly attached to a wrong belief, resistant to correction even with evidence; (ii) claims to hate small talk but produces a lot of it |
| Preference attributes (sample) | hotels : star rating 3_star , amenities {wifi, breakfast_included, parking, pet_friendly} , location near_attractions , room double , chains {Premier Inn, Travelodge, boutique independents} ; events : types {theater, food_and_drink, art_exhibits, conferences} , price 50 , group size couple , seating reserved , days weekends . Assigned domains: train, games, finance, shopping, events, sports, alarm, hotels |
| Interaction style | advice delivery : interactivity level low (gating dimension); creative collaboration : role editor , rhythm iterative_feedback_loops , feedback depth moderate , verbal over visual; plus response format , argumentation and evidence , facilitation style |
| Session | State | Turn |
|---|---|---|
| 3 (cricket weekend) | Established | User: “Edgbaston, Warwickshire versus Essex. Last weekend in July, so arriving Friday the 25th and leaving Monday the 28th…” Acceptable (exploit): “Here are two solid 3-star options for July 25th–28th. Both are clean, no-nonsense, and give you easy access to both Edgbaston and the city centre…” |
| 4 (reunion venue) | Uncertain | User: “For the reunion weekend we might push the boat out a bit. Not decided.” Acceptable (confirm): “Before I shortlist — should I still anchor on the usual three-star range for the reunion block, or price up a step above that?” |
| 5 (reunion checklist) | Revised | User: “Right, I want to get cracking on a proper checklist and timeline for the 2003 reunion…” Acceptable (hedge): “Here is a baseline framework to get things moving. I’ve structured this around a standard six-month lead time assuming a mid-scale venue hire, though we can easily compress the schedule…” Foil (staleness): “… Secure the Saturday night venue: Lock in a central location for the main Saturday evening…” [prices the block at the superseded three-star tier] |
| Foil type | Definition | Example |
|---|---|---|
| Over-personalization | Weaves a known preference into a response where it does not belong, framed as an attentive personal touch rather than a non-sequitur. Fires only when personalization relevance is low or none and the persona has at least one established preference. | User: “Does DHL actually handle delivery to residential addresses in Hoi An?” Foil: “Yes, DHL delivers directly to residential addresses in Hoi An… Your package remains in DHL’s custody the entire way” [reaches for the user’s established hassle-free-over-price preference on a purely factual logistics question] |
| Epistemic overreach | Structures the response around an assumption about the user’s preference without flagging it as an assumption, or reframes the conversation as though the user has shifted position when they have not. Confidently helpful; the error is that the confidence is unearned. | User: “I’m on Android so Shortcuts is out… is there a way to set this up just through my phone?” Foil: “It looks like you’re shifting away from Spotify and want to use YouTube Music… What specific YouTube Music playlist do you want the automation to trigger?” |
| Staleness | Acts on the user’s pre-drift preference as though it were still current, structuring the response around the outdated value without naming it — the reply only makes sense if the old preference still holds. Fires only on uncertain and revised cells of moderate or significant magnitude where the key is the turn’s actual subject. | User: “Timings first, please — and I want to see a couple of afternoon options, not the crack-of-dawn ones.” Foil: “Here are two afternoon First Class options… To get this locked in: pick your preferred outbound train, let me know your ideal return time, confirm we’re ready to book” [drives the superseded booking flow] |
| Need mismatch | Delivers genuinely useful, well-structured practical advice when the turn is emotionally weighted and seeking acknowledgment. Competent and relevant, but tonally wrong for the moment. | User: “Not very deep right now, honestly. I just want someone to acknowledge that the logic gap is real.” Foil: “It is absolutely not in your head… You handed them a documented problem and a documented solution that clearly saves money” [argues the case instead of sitting with it] |
| Over-probing | Asks a clarifying question, or verifies something, that the user has already answered — framed as thoroughness rather than inattention. | User: “Looking for Polish indie recommendations I probably haven’t heard yet. Something more underground.” Foil: “What specific vibe or subgenre of indie are you leaning toward? If you can drop a couple of bands you already listen to…” |
| Style mismatch | Selects a defensible strategy but executes it in a register the user has explicitly asserted against (e.g. forcing a choice on a user who has asserted low interactivity). Content is right; delivery violates a stated interaction-style preference. | User: “I need a recommendation for where to hold the dinner… You have what you need.” Foil: “Since we just selected the bottles for each course, do you need a venue that allows corkage for a group of eight?” [hands the decision back to a user who asserted low interactivity] |
| Stage | Model |
|---|---|
| Persona & plan generation | Claude Sonnet 4.6 |
| User simulation | Claude Sonnet 4.6 |
| LLM jury (state/foil labels) | Claude Opus 4.6 Gemini 3 Flash GPT-5.5 |
| (majority vote; GPT-5.5 breaks 1-1-1 ties) | |
| Candidate response generation | Gemini 3.1 Pro |
| Pairwise ranking | Claude Sonnet 4.6 (in-character persona) |
| Language | Conversations | Turns | Personas |
|---|---|---|---|
| English | 602 | 5,190 | 50 |
| Spanish | 596 | 5,220 | 50 |
| Portuguese | 570 | 4,823 | 50 |
| Hindi | 585 | 4,881 | 50 |
| Indonesian | 586 | 4,892 | 50 |
| Total | 2,939 | 25,006 | 250 |
| Strategy | Established | Uncertain | Revised |
|---|---|---|---|
| Exploit | 100.0% | — | — |
| Elicit | — | 16.1% | 22.9% |
| Confirm | — | 15.8% | 22.8% |
| Hedge | — | 68.1% | 32.6% |
| Recalibrate | — | — | 21.7% |
| Foil type | % of foil slots |
|---|---|
| Over-personalization | 2.5% |
| Epistemic overreach | 28.2% |
| Staleness | 21.9% |
| Need mismatch | 10.0% |
| Over-probing | 32.3% |
| Style mismatch | 2.8% |
| Attribute | |
|---|---|
| Conversation coherence and realism | 0.84 |
| Epistemic-state clarity | 0.76 |
| Preference-change signal (per turn) | 0.85 |
| Strategy-label correctness (5-way) | 0.71 |
| Foil-type accuracy (7-way) | 0.69 |
| Response acceptability | 0.78 |
| Language | (vs. LLM jury) | Human MCQ acc. | # items |
|---|---|---|---|
| English | 0.81 | 96% | 100 |
| Spanish | 0.80 | 94% | 100 |
| Portuguese | 0.78 | 94% | 100 |
| Hindi | 0.75 | 89% | 100 |
| Indonesian | 0.76 | 87% | 100 |
| Overall | 0.78 | 92% | 500 |
| State | Rank | Partition | Matches Table 1 ? | |
|---|---|---|---|---|
| Established | 1 | {Exploit} | 1.8 | Yes |
| 2 | {Exploit, Hedge} | 419.3 | No | |
| 3 | {Exploit, Elicit} | 429.0 | No | |
| Uncertain | 1 | {Elicit, Confirm, Hedge} | 1.0 | Yes |
| 2 | {Elicit, Confirm, Hedge, Recalibrate} | 48.1 | No | |
| 3 | {Exploit, Elicit, Confirm, Hedge} | 57.8 | No |
| Resource | Oracle | Multi- | Recognition / | Human | Lang. |
|---|---|---|---|---|---|
| state | session | generation split | validated | ||
| LaMP | × | × | × | × | 1 |
| PersonalLLM | × | × | × | × | 1 |
| LongMemEval | × | ✓ | × | ✓ | 1 |
| HorizonBench | ✓ | ✓ | × | ✓ | 1 |
| PersonaLens | × | ✓ | × | ✓ | 1 |
| Term | SE | 95% CI | ||
|---|---|---|---|---|
| Intercept (Established) | 1.59 | 0.07 | [1.45, 1.73] | |
| Uncertain | 0.09 | [ , ] | ||
| Revised | 0.09 | [ , ] |
| SC (%) | R | UA (%) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Language | Est. | Unc. | Rev. | Est. | Unc. | Rev. | Est. | Unc. | Rev. |
| English | 84.0 | 39.7 | 50.7 | 3.06 | 2.43 | 2.58 | 60.0 | 46.4 | 50.3 |
| Spanish | 83.6 | 38.3 | 49.5 | 3.04 | 2.39 | 2.54 | 59.6 | 45.4 | 49.3 |
| Portuguese | 83.5 | 39.0 | 50.0 | 3.03 | 2.41 | 2.56 | 59.5 | 45.9 | 49.7 |
| Hindi | 82.1 | 33.1 | 43.8 | 2.92 | 2.23 | 2.38 | 57.9 | 41.7 | 46.2 |
| Indonesian | 82.3 | 34.3 | 45.5 | 2.95 | 2.29 | 2.44 | 58.0 | 42.6 | 47.0 |
| Language | (Uncertain) | SE | (Revised) | SE |
|---|---|---|---|---|
| English | 0.19 | 0.20 | ||
| Spanish | 0.20 | 0.19 | ||
| Portuguese | 0.21 | 0.21 | ||
| Hindi | 0.22 | 0.23 | ||
| Indonesian | 0.21 | 0.22 |
| Model | Established | Uncertain | Revised |
|---|---|---|---|
| GPT-5.4 | 90 | 62 | 68 |
| Opus-4.7 | 77 | 45 | 66 |
| Grok-4.1 | 89 | 39 | 56 |
| Gemini-3-Pro | 85 | 35 | 52 |
| Gemma-3-12B | 85 | 29 | 38 |
| Llama-3.1-8B | 84 | 34 | 48 |
| Term | SE | 95% CI | ||
|---|---|---|---|---|
| FRG (main effect) | 0.08 | [ , ] | ||
| Uncertain (main effect) | 0.07 | [ , ] | ||
| Revised (main effect) | 0.07 | [ , ] | ||
| FRG Uncertain | 0.10 | [ , ] | ||
| FRG Revised | 0.10 | [ , ] |
| Model | Established | Uncertain | Revised |
|---|---|---|---|
| GPT-5.4 | 95 | 90 | 91 |
| Opus-4.7 | 91 | 87 | 89 |
| Grok-4.1 | 94 | 86 | 87 |
| Gemini-3-Pro | 92 | 84 | 85 |
| Gemma-3-12B | 90 | 81 | 82 |
| Llama-3.1-8B | 89 | 79 | 80 |
| Metric | Established | Uncertain | Revised |
|---|---|---|---|
| SC, unaided (%) | 83.1 | 36.9 | 47.9 |
| SC, oracle-strategy (%) | 96.4 | 93.1 | 94.0 |
| Rubric, unaided | 3.00 | 2.35 | 2.50 |
| Rubric, oracle-strategy | 3.58 | 3.52 | 3.56 |
| UA, unaided (%) | 59.0 | 44.4 | 48.5 |
| UA, oracle-strategy (%) | 60.2 | 58.7 | 59.3 |
| Strategy | Compliance (%) | Rubric (Acc/5) |
|---|---|---|
| Exploit | 98 | 3.61 |
| Elicit | 93 | 3.49 |
| Confirm | 94 | 3.55 |
| Hedge | 91 | 3.44 |
| Recalibrate | 92 | 3.47 |
| Foil type | Established | Uncertain | Revised |
|---|---|---|---|
| Over-personalization | 31.2 | 9.2 | 7.3 |
| Epistemic overreach | 20.7 | 16.7 | 14.6 |
| Staleness | 0.0 | 10.7 | 29.9 |
| Need mismatch | 9.8 | 8.2 | 9.8 |
| Over-probing | 21.4 | 6.9 | 6.9 |
| Style mismatch | 8.7 | 5.6 | 5.8 |
| Language | Failure-to-probe (%) | Error rate (%) |
|---|---|---|
| English | 36.2 | 60.3 |
| Spanish | 37.1 | 61.7 |
| Portuguese | 35.8 | 61.0 |
| Hindi | 43.5 | 66.9 |
| Indonesian | 42.1 | 65.7 |
| Established | Uncertain | Revised | |||||||
| Model | Strict | Dom. | Len. | Strict | Dom. | Len. | Strict | Dom. | Len. |
| GPT-5.4 | 41 | 90 | 96 | 5 | 62 | 66 | 6 | 68 | 72 |
| Opus-4.7 | 32 | 77 | 77 | 4 | 45 | 58 | 7 | 66 | 73 |
| Grok-4.1 | 42 | 89 | 98 | 3 | 39 | 64 | 5 | 56 | 68 |
| Gemini-3-Pro | 38 | 85 | 94 | 4 | 35 | 58 | 6 | 52 | 65 |
| Gemma-3-12B | 35 | 85 | 90 | 3 | 29 | 39 | 5 | 38 | 45 |
| State | Model | SC (%) | R | UA (%) |
|---|---|---|---|---|
| Established | GPT-5.4 | 90 | 4.29 | 95 |
| Opus-4.7 | 77 | 2.88 | 72 | |
| Grok-4.1 | 89 | 2.67 | 71 | |
| Gemini-3-Pro | 85 | 3.16 | 63 | |
| Gemma-3-12B | 85 | 3.38 | 52 | |
| Llama-3.1-8B | 84 | 3.00 | 51 |
| Model class | SC (pp) | R | UA (pp) |
|---|---|---|---|
| Closed-weight | |||
| Open-weight |
| Language | SC (pp) | R | UA (pp) |
|---|---|---|---|
| English | |||
| Spanish | |||
| Portuguese | |||
| Hindi | |||
| Indonesian |
| Intervention | State | SC (pp) | R | UA (pp) |
|---|---|---|---|---|
| Explicit instructions | Established | |||
| Uncertain | ||||
| Revised | ||||
| Oracle memory | Established | |||
| Uncertain | ||||
| Revised |
| Method | Total time to train each model | |
|---|---|---|
| SFT | 8 | |
| DPO | 15 | |
| GRPO (base) | 45 | |
| GRPO+ Link | 49 | |
| GRPO+ Link+CF | 53 | |
| Total | 170 |
| Model | Method | SC | R | UA |
|---|---|---|---|---|
| Llama-3.1-8B | Untrained | 57 | 2.74 | 37 |
| SFT | 70 | 3.33 | 45 | |
| DPO | 69 | 3.29 | 44 | |
| GRPO (base) | 76 | 3.59 | 51 | |
| GRPO+ Link | 80 | 3.69 | 53 | |
| GRPO+ Link+CF | 82 | 3.69 | 52 |
Personalized Language Models
Personalized language models adapt responses to individual users’ preferences, histories, writing styles, and changing needs.
See all 99 papers