Persona Learning from Feedback for Personalized AI Assistants

Yiran Shen; Yu Xia; Liuyi Yao; Prithviraj Ammanabrolu

cs.LGOct 28, 2026

Persona Learning from Feedback for Personalized AI Assistants

Authors: Yiran Shen, Yu Xia, Liuyi Yao, Prithviraj Ammanabrolu

Organizations: UC San Diego · Alibaba Group

Abstract

Personalizing AI assistants requires adapting their behavior to users' preferences and task contexts. Existing work elicits assistant personas through persona prompting or finetuning on persona-conditioned generations. However, persona descriptions alone do not identify what to change or preserve in a particular response, or which behaviors suit the current request. We introduce $\textit{Persona Learning from Feedback (PLF)}$ , a framework that learns reusable, preference-grounded assistant personas as independent LoRA adapters on a shared frozen backbone. During training, a frozen self-teacher conditions on persona specifications and response-specific feedback to provide token-level supervision on each adapter's own generated prefixes. A feedback gate focuses corrective updates on responses judged to need improvement. At deployment, a supervised router uses the request or dialogue to select a single adapter or a fixed composition, without user-specific finetuning. We evaluate PLF on single-turn personalization with model-generated critiques and multi-turn tutoring with simulated students' reactions. Across both settings, PLF improves model-judged preference fit over fixed-persona policies and strong baselines including persona prompting and persona distillation, demonstrating how feedback-trained behaviors can be reused for adaptive assistant personalization without degrading general capabilities.

Figures & tables

Figure 1: Persona Learning from Feedback (PLF). (a) A frozen teacher uses critiques or conversational feedback to supervise persona adapters on student-generated prefixes. A feedback gate excludes satisfactory responses from updates. (b) A supervised router selects personas using the request or dialogue, activating a single adapter or a fixed signed composition.

Figure 2: Preference fit of feedback-trained personas. Percentages of PLF wins, ties, and alternative wins on (a) 499 single-turn requests and (b) 1,000 tutoring dialogue pairs from 500 openings evaluated with both student types. Persona prompting uses the same router as PLF.

Table 3

Figure 3: Student satisfaction under changing preferences. Share of replies without flagged objections, averaged across switch directions: (a) whole lessons, with 95% bootstrap intervals; (b) aligned to the switch at

j

. Only the oracle knows the new preference when choosing turn

j

(a) Persona composition: warm + concise
User request: “Hi! Could you tell me five benefits of working from home? Keep it friendly and short, please!”
Concise head	“1. Save time on commuting. 2. Work in your own rhythm. 3. No office stress. 4. More control over your day. 5. Save money on expenses.”
Warm head	“Absolutely! Here are five cozy, home-based wins-no jargon, just vibes: 1. No commute stress-You skip traffic, parking, and the daily grind. Just chill in your PJs and sip your coffee…”
PLF mixture	“Sure! Here’s a quick list: 1. No commute stress-just save time and energy. 2. Work in your favorite comfy spot. 3. More time for yourself and your family. 4. Flex your schedule to fit your energy. 5. Fewer office distractions-focus, no noise!”
(b) Tutoring: direct $\rightarrow$ interactive
Student feedback	“You’re just telling me facts again; ask me a real question instead. ”

Table 3: Examples of persona composition and adaptation. The warm-head response (129 words in full) is excerpted; emoji are omitted.

Panel of Figure 4 — Figure 4: Feedback gating during persona learning. (a) Critic no-change rate in single-turn personalization. (b) Satisfaction-judge acceptance in multi-turn tutoring before additional gating checks. Higher values indicate more responses passing the displayed feedback check. Thin lines are raw values and thick lines are centered nine-point means.

Single-turn QA/ Qwen3-4B-Instruct-2507
Method	IFEval $\uparrow$	GSM8K $\uparrow$	MMLU-Pro $\uparrow$	MT-Bench $\uparrow$
Base	86.5	93.2	67.2	6.87
Persona prompting	87.2 (+0.7)	92.8 ( $-$ 0.4)	67.4 (+0.2)	6.78 ( $-$ 0.09)
Persona distillation	86.1 ( $-$ 0.4)	93.4 (+0.2)	67.4 (+0.2)	6.68 ( $-$ 0.19)
Feedback imitation	85.6 ( $-$ 0.9)	92.3 ( $-$ 0.9)	66.3 ( $-$ 0.9)	6.69 ( $-$ 0.18)
PLF	85.4 ( $-$ 1.1)	93.0 ( $-$ 0.2)	66.9 ( $-$ 0.3)	6.79 ( $-$ 0.08)

Table 4: Scores on four standard LLM benchmarks. We compare the base models with PLF and three alternative baseline methods. Scores are percentages except MT-Bench (1–10).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Users	Original requests	Usable rewrites
Train	1,033	5,698	5,677
Validation	39	203	203
Test	89	500	499

Appendix

Table 5: Single-turn data splits. User counts refer to the original-request splits after filtering and deduplication. Rewritten requests are used for router training or evaluation, not for persona-head training. The final head-training corpus is separately expanded to 6,400 original requests from 1,242 users.

Setting	Value
LoRA	Rank 32, scale 64, dropout 0, all linear layers
Training pass	6,400 original requests, 100 batches of 64
Batch sizes	Training 64, training microbatch 4, rollout microbatch 8
Student sampling	Temperature 0.7, top- $p$ 1.0, one sample per head and request
Token limits	1,024 prompt tokens, 2,560 new response tokens
Teacher targets	Top-256 support, full-vocabulary student normalization

Appendix

Table 6: Single-turn persona-head training configuration. The backbone remains frozen, and the four heads are independent LoRA adapters.

PLF versus	$n$	Wins	Ties	Losses	Win rate [95% CI]
Single-turn personalization
Warm head	499	350	65	84	76.7 [73,80]
Impersonal head	499	344	72	83	76.2 [73,79]
Concise head	499	316	111	72	74.4 [71,78]
Thorough head	499	344	84	71	77.4 [74,81]
Base	499	332	78	89	74.3 [71,78]

Appendix

Table 7: Preference fit with full judgment counts. Wins, ties, and losses are from PLF’s perspective. Win rates and confidence intervals are percentages. The tutoring oracle knows the hidden preference.

Setting	$n$	PLF wins	Ties	Prompt wins	PLF win rate
Single-turn	96	43	22	31	56.3%
Multi-turn tutoring	100	47	18	35	56.0%

Appendix

Table 8: Human preference evaluation of PLF against persona prompting.

PLF versus	Wins	Ties	Losses	Win rate [95% CI]
Direct-instruction students
Direct head	173	98	229	44.4 [40.6,48.4]
Interactive head	432	32	36	89.6 [87.2,91.9]
Base	246	56	198	54.8 [50.7,58.8]
Persona prompting	244	67	189	55.5 [51.5,59.6]
Persona distillation	238	74	188	55.0 [51.2,58.8]

Appendix

Table 9: Tutoring preference fit by student type. Each comparison uses the same 500 test openings. Counts are PLF wins, ties, and losses against the indicated policy. Win rates and confidence intervals are percentages.

	Satisfaction $\uparrow$			Factual errors $\downarrow$
Policy	D	I	Mean	D	I	Mean
Base	77.8	73.5	75.6	9.2	6.5	7.9
Direct head	81.9	68.3	75.1	9.0	5.6	7.3
Interactive head	62.3	84.1	73.2	7.7	5.6	6.7
Persona prompting	77.1	82.5	79.8	10.0	6.2	8.1
PLF	80.6	83.5	82.1	9.3	4.9	7.1

Appendix

Table 10: Student satisfaction and factual errors. Each policy generates eight tutor turns on 500 openings for each student type. D and I denote direct-instruction and interactive students. Satisfaction is the percentage of replies not flagged as objections, and factual errors are the percentage of tutor turns flagged by the fact-checking judge. Means weight the student types equally. The oracle uses the matching head, with differences from the corresponding fixed-head run due to sampling.

Composition rule	Win rate	95% CI
Sum of the requested heads	39	[30,48]
Average of the requested heads	50	[42,59]
Axis-weighted composition	70	[62,78]

Appendix

Table 11: Validation comparison of composition rules. Win rates against prompting with both persona cards, with half credit for ties. Rates and confidence intervals are percentages.

Requested pair	$n$	Warm	Impersonal	Concise	Thorough
Warm + concise	48	89 [80,96]	78 [68,89]	72 [60,82]	98 [95,100]
Warm + thorough	59	81 [73,90]	97 [93,100]	99 [97,100]	47 [37,58]
Impersonal + concise	55	68 [57,79]	51 [39,63]	53 [43,64]	82 [72,91]
Impersonal + thorough	60	91 [84,97]	66 [55,76]	81 [71,89]	63 [54,72]

Appendix

Table 12: Composition on correctly routed test requests. Mixture win rates (%) against each fixed head, with 95% confidence intervals. Bold rates compare with a constituent of the requested pair. Results cover 222 correctly routed requests out of 247 paired-preference requests.

Requested pair	All $n$	Correct $n$	Voice head	Depth head
			PLF versus
Warm + concise	62	48	81 [73,90]	60 [50,71]
Warm + thorough	62	59	78 [69,86]	47 [37,56]
Impersonal + concise	61	55	55 [43,66]	54 [44,64]
Impersonal + thorough	62	60	65 [54,75]	63 [53,72]

Appendix

Table 13: Composition including routing errors. PLF win rates against the requested voice and depth heads on all requests assigned to each pair, including routing errors. Correct

n

gives the number whose selected pair matches the assigned pair. Win rates are percentages with half credit for ties and 95% bootstrap confidence intervals; all requests in the row form the denominator.

Switch	Direct head	Interactive head	Base	Oracle
Direct $\rightarrow$ interactive	55 [51,58]	81 [78,83]	54 [51,58]	47 [43,51]
Interactive $\rightarrow$ direct	67 [64,71]	56 [52,60]	59 [55,62]	40 [36,44]
Mean	61 [59,63]	68 [66,70]	56 [54,59]	43 [41,46]

Appendix

Table 14: Preference fit under switching. PLF win rates (%) with 95% confidence intervals. Each direction uses 500 test openings. Bold rates compare with the fixed head matching the preference before the switch. The oracle knows the new preference when choosing turn

j

		Satisfaction around the switch
Policy	Overall [95% CI]	Before $j$	$j$	$j+1$	$j+2$	$\geq j+3$
Base	73.0 [72.0,74.1]	77.4	64.1	73.7	69.0	70.3
Direct head	71.4 [70.4,72.5]	74.9	62.7	73.7	65.1	71.0
Interactive head	66.4 [65.3,67.6]	69.5	57.8	67.9	64.0	65.2
PLF	76.7 [75.7,77.6]	84.9	62.5	74.3	71.8	70.7
Oracle	79.2 [78.3,80.2]	86.4	70.8	74.7	73.4	74.0

Appendix

Table 15: Satisfaction under switching, averaged over directions. All satisfaction values are percentages. The overall column includes a 95% confidence interval over opening–direction pairs. The remaining columns align tutor turns to the switch at

j

and average the direction-specific bucket percentages.

Direct instruction $\rightarrow$ interactive guidance
Policy	Overall	Before $j$	$j$	$j+1$	$j+2$	$\geq j+3$
Base	77.1	81.0	69.4	77.8	75.9	73.3
Direct head	79.1	85.9	72.6	76.4	72.5	72.5
Interactive head	61.3	52.6	62.0	69.6	70.4	70.9
PLF	79.8	84.9	69.6	79.2	79.0	75.1
Oracle	81.3	87.0	73.2	78.6	77.6	77.3

Appendix

Table 16: Switching satisfaction by direction. Satisfaction percentages over 500 dialogues per direction, with buckets aligned to the switch turn

j

. The oracle anticipates the switch, while PLF first observes evidence of it after generating turn

j

Switch	First opportunity (%)	Additional delay	No detection
Direct $\rightarrow$ interactive	48.0	0.63	113/500
Interactive $\rightarrow$ direct	31.6	0.67	225/500
Both directions	39.8	0.65	338/1,000

Appendix

Table 17: PLF’s detection of a preference switch. First opportunity is the percentage of dialogues selecting the new head at

j+1

. Additional delay counts turns beyond

j+1

until the first selection of the new head, conditional on that selection occurring. No detection counts dialogues ending without such a selection.

	Direct-instruction head		Interactive head
Cycles	Satisfied	Masked	Satisfied	Masked
1–20	68.0	48.4	67.9	60.1
21–40	70.3	48.9	73.5	67.8
41–60	71.0	50.7	72.9	66.9
61–80	70.6	48.8	72.0	67.6
81–100	69.9	49.0	71.8	66.9

Appendix

Table 18: Tutoring feedback and masking during training. Percentages are averaged within blocks of 20 rollout cycles. Satisfied denotes acceptance by the training satisfaction judge. Masked denotes responses excluded from distillation after all gating checks. Each cycle contains 256 tutor turns per head.

Group	$n$	W/T/L	Win rate	95% CI
All requests	499	225/158/116	60.9	[57,64]
All single-persona	252	104/77/71	56.5	[52,62]
All paired	247	121/81/45	65.4	[61,70]
Warm	63	37/10/16	66.7	[56,77]
Impersonal	63	34/15/14	65.9	[56,76]
Concise	63	16/29/18	48.4	[40,57]

Appendix

Table 19: Single-turn gate ablation at step 60. Gated versus ungated PLF on test requests. Win rates (%) give ties half credit; intervals are 95% percentile bootstraps over requests. Groups are defined by requested preferences and include routing errors.

Student preference	$n$	W/T/L	Win rate	Sign-test $p$
Direct instruction	500	215/73/212	50.3	0.92
Interactive guidance	500	242/70/188	55.4	0.011
Both types	1,000	457/143/400	52.9	0.056

Appendix

Table 20: Tutoring gate ablation with matched heads. Gated versus ungated heads. Win rates (%) use half credit for ties. The two-sided sign test compares wins with losses, excluding ties; the pooled row combines both student types on the same 500 openings.

Measure	Gate	No gate	Difference [95% CI]
Interactive: strict satisfaction	74.5	70.5	$+4.0$ [1.7,6.4]
Interactive: no objection	85.1	82.7	$+2.4$ [0.9,4.0]
Interactive: asks a question	86.3	82.4	$+3.9$ [1.7,6.1]
Interactive: repetition complaints	2.7	4.2	$-1.5$ [ $-2.4$ , $-0.7$ ]
Direct: strict satisfaction	40.4	39.0	$+1.4$ [ $-1.3$ ,3.8]
Direct: no objection	81.9	81.1	$+0.8$ [ $-0.7$ ,2.4]

Appendix

Table 21: Tutoring behavior under gated and ungated training. Rates per 100 tutor turns; differences are gate minus no gate, with 95% paired bootstrap intervals over opening–student pairs. Strict satisfaction uses the full three-part gate rule; no objection is the evaluation satisfaction metric. Rates and differences are rounded independently.

Requested preference	$n$	Forward-KL win rate [95% CI]
Single-persona requests
Warm	63	49 [38,60]
Impersonal	63	68 [58,78]
Concise	63	40 [31,49]
Thorough	63	57 [48,67]
Paired-preference requests

Appendix

Table 22: Forward versus reverse KL at step 60. Forward-KL preference win rates (%) under the routed single-turn protocol, with 95% bootstrap intervals over requests. Request groups are defined by assigned preferences and include routing errors; values above 50% favor forward KL.

	Token entropy			Top-token agreement		Hedge-mass retention
Persona	T	F	R	F	R	F	R
Warm	0.78	1.08	0.82	0.79	0.80	0.91	0.90
Impersonal	0.61	0.98	0.64	0.80	0.80	0.81	0.76
Concise	0.32	0.93	0.31	0.73	0.73	0.57	0.45
Thorough	0.81	1.00	0.83	0.88	0.88	0.94	0.93

Appendix

Table 23: Diagnostics under the reconstructed training teacher. T, F, and R denote teacher, forward-KL head, and reverse-KL head. Top-token agreement compares the head’s most probable token with the teacher’s. Hedge-mass retention is the share of the teacher’s hedge-word probability retained by the head. Values are the reported averages.

Personalized Language Models

Personalized language models adapt responses to individual users’ preferences, histories, writing styles, and changing needs.

See all 91 papers

Learning Personalized Agents from Human Feedback
Feb 18, 2026Kaiqu Liang, Julia Kruk, Shengyi Qian +9Personal AI Agents User Preference Modeling
Meta Superintelligence Labs · Princeton University · Duke University
Learning Dynamic User Personas from Implicit Interaction Streams via Iterative Refinement
Jul 29, 2026Haifeng WuLLM Personalization User Modeling
PayPal
Learning What to Keep: Transfer-Aware Personal Skill Learning from User Feedback
Oct 14, 2026Li Wang, Chen Zhang, Yirong Yang +4Personalized Language Models Transfer Learning
Beihang University · Chengdu University of Information Technology · Nanyang Technological University +2

Feb 18, 2026J/K move · Enter open · S save

Learning Personalized Agents from Human Feedback

Kaiqu Liang, Julia Kruk, Shengyi Qian, Xianjun Yang+8

Meta Superintelligence Labs · Princeton University · Duke University

Personal AI Agents User Preference Modeling Memory-Augmented Agents AI Agent Benchmarks

Open paper AskPDF arXiv

Modern AI agents are powerful but often fail to align with the idiosyncratic, evolving preferences of individual users. Prior approaches typically rely on static datasets, either training implicit preference models on interaction history or encoding user profiles in external memory. However, these approaches struggle with new users and with preferences that change over time. We introduce Personalized Agents from Human Feedback (PAHF), a framework for continual personalization in which agents learn online from live interaction using explicit per-user memory. PAHF operationalizes a three-step loop: (1) seeking pre-action clarification to resolve ambiguity, (2) grounding actions in preferences retrieved from memory, and (3) integrating post-action feedback to update memory when preferences drift. To evaluate this capability, we develop a four-phase protocol and two benchmarks in embodied manipulation and online shopping. These benchmarks quantify an agent's ability to learn initial preferences from scratch and subsequently adapt to persona shifts. Our theoretical analysis and empirical results show that integrating explicit memory with dual feedback channels is critical: PAHF learns substantially faster and consistently outperforms both no-memory and single-channel baselines, reducing initial personalization error and enabling rapid adaptation to preference shifts.

Abstract

Personalized Language Models

Explore similar work

Learning Personalized Agents from Human Feedback

Learning Dynamic User Personas from Implicit Interaction Streams via Iterative Refinement

Learning What to Keep: Transfer-Aware Personal Skill Learning from User Feedback