Learning Personalized Agents from Human Feedback

Kaiqu Liang; Julia Kruk; Shengyi Qian; Xianjun Yang; Shengjie Bi; Yuanshun Yao; Shaoliang Nie; Mingyang Zhang

cs.AIFeb 18, 2026

Learning Personalized Agents from Human Feedback

Authors: Kaiqu Liang, Julia Kruk, Shengyi Qian, Xianjun Yang, Shengjie Bi, Yuanshun Yao, Shaoliang Nie, Mingyang Zhang,

Organizations: Meta Superintelligence Labs · Princeton University · Duke University

Abstract

Modern AI agents are powerful but often fail to align with the idiosyncratic, evolving preferences of individual users. Prior approaches typically rely on static datasets, either training implicit preference models on interaction history or encoding user profiles in external memory. However, these approaches struggle with new users and with preferences that change over time. We introduce Personalized Agents from Human Feedback (PAHF), a framework for continual personalization in which agents learn online from live interaction using explicit per-user memory. PAHF operationalizes a three-step loop: (1) seeking pre-action clarification to resolve ambiguity, (2) grounding actions in preferences retrieved from memory, and (3) integrating post-action feedback to update memory when preferences drift. To evaluate this capability, we develop a four-phase protocol and two benchmarks in embodied manipulation and online shopping. These benchmarks quantify an agent's ability to learn initial preferences from scratch and subsequently adapt to persona shifts. Our theoretical analysis and empirical results show that integrating explicit memory with dual feedback channels is critical: PAHF learns substantially faster and consistently outperforms both no-memory and single-channel baselines, reducing initial personalization error and enabling rapid adaptation to preference shifts.

Figures & tables

Figure 1: Static vs. continual personalization. Top: Static personalization, where offline human–AI logs are processed once to populate user profiles, and at deployment the agent only reads from this static memory. Bottom: Our continual personalization framework keeps the user in the loop during online interaction: the agent consults and updates explicit memory in the pre-action stage, and the user observes the outcome in the environment after the agent acts and provides post-action feedback, which is written back to memory to correct errors and adapt as preferences change.

Figure 2: Continual agent personalization using pre- and post-action feedback channels. On Day 1 (Initial Preference), the agent uses a pre-action interaction to learn Kate’s favorite drink (“Coke”) before acting. On Day 2 (Contextual Preference), the agent overgeneralizes this memory when Kate is sleepy; after serving Coke, it is corrected via post-action feedback (“I prefer tea when I am sleepy”). On Day 3 (Dynamic Preference), the agent acts consistently with its memory (serving tea), but Kate’s underlying preference has shifted to coffee. This error triggers another post-action update, illustrating how the framework supports continual personalization under preference drift.

Panel of Figure 3 — Figure 3: Results on embodied manipulation . Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for success rate (left), feedback frequency (middle), and ACPE (right). Shaded regions denote standard error.

Panel of Figure 4 — Figure 4: Results on online shopping . Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for success rate (left), feedback frequency (middle), and ACPE (right). Shaded regions denote standard error.

Method	Embodied		Shopping
Method	Phase 2	Phase 4	Phase 2	Phase 4
No memory	$32.3_{\pm 0.4}$	$44.8_{\pm 0.5}$	$27.8_{\pm 0.2}$	$27.0_{\pm 0.4}$
Pre-action only	$54.1_{\pm 1.1}$	$35.7_{\pm 1.0}$	$34.4_{\pm 0.5}$	$56.0_{\pm 0.7}$
Post-action only	$67.9_{\pm 1.5}$	$68.3_{\pm 1.2}$	$38.9_{\pm 0.5}$	$66.9_{\pm 0.8}$
PAHF (pre+post)	$\mathbf{70.5_{\pm 1.7}}$	$\mathbf{68.8_{\pm 1.3}}$	$\mathbf{41.3_{\pm 0.8}}$	$\mathbf{70.3_{\pm 1.1}}$

Table 1: Evaluation success rates. Phase-2 and Phase-4 success rates (%) for each method in embodied and shopping domains.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	GPT-4.1 Agent; GPT-4o Human				GPT-4o Agent; GPT-4.1 Human
	Embodied		Shopping		Embodied		Shopping
	Phase 2	Phase 4	Phase 2	Phase 4	Phase 2	Phase 4	Phase 2	Phase 4
No memory	$32.7$	$44.8$	$24.6$	$14.9$	$31.4$	$42.7$	$28.0$	$26.3$
Pre-action only	$52.8$	$35.1$	$35.8$	$59.0$	$57.8$	$30.4$	$37.2$	$55.0$
Post-action only	$81.3$	$77.1$	$36.0$	$66.6$	$66.3$	$67.8$	$39.1$	$66.2$
PAHF (pre+post)	$\mathbf{82.3}$	$\mathbf{78.8}$	$\mathbf{38.3}$	$\mathbf{67.4}$	$\mathbf{73.5}$	$\mathbf{68.3}$	$\mathbf{39.3}$	$\mathbf{67.9}$

Appendix

Table 2: Evaluation success rates (%). Phase 2 and 4 performance across different agent-simulator pairs (Memory: SQLite).

Method	SQLite Memory				FAISS Memory
	Embodied		Shopping		Embodied		Shopping
	Phase 2	Phase 4	Phase 2	Phase 4	Phase 2	Phase 4	Phase 2	Phase 4
No memory	$32.3_{\pm 0.4}$	$44.8_{\pm 0.5}$	$27.8_{\pm 0.2}$	$27.0_{\pm 0.4}$	$32.3_{\pm 0.4}$	$44.8_{\pm 0.5}$	$27.8_{\pm 0.2}$	$27.0_{\pm 0.4}$
Pre-action only	$54.1_{\pm 1.1}$	$35.7_{\pm 1.0}$	$34.4_{\pm 0.5}$	$56.0_{\pm 0.7}$	$51.8_{\pm 1.3}$	$36.2_{\pm 0.9}$	$35.9_{\pm 0.6}$	$57.2_{\pm 0.7}$
Post-action only	$67.9_{\pm 1.5}$	$68.3_{\pm 1.2}$	$38.9_{\pm 0.5}$	$66.9_{\pm 0.8}$	$66.9_{\pm 1.7}$	$66.6_{\pm 1.4}$	$39.2_{\pm 0.7}$	$65.9_{\pm 0.9}$
PAHF (pre+post)	$\mathbf{70.5_{\pm 1.7}}$	$\mathbf{68.8_{\pm 1.3}}$	$\mathbf{41.3_{\pm 0.8}}$	$\mathbf{70.3_{\pm 1.1}}$	$\mathbf{68.0_{\pm 1.8}}$	$\mathbf{68.2_{\pm 1.6}}$	$\mathbf{39.4_{\pm 1.2}}$	$\mathbf{70.8_{\pm 1.0}}$

Appendix

Table 3: Evaluation success rates (%). Phase 2 and 4 performance across memory backends and domains. Agent and Simulator: GPT-4o.

Panel of Figure 5 — Figure 5: Embodied manipulation results (Agent: GPT-4o, Human Simulator: GPT-4o, Memory: FAISS) . Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 6 — Figure 6: Online shopping result (Agent: GPT-4o, Human Simulator: GPT-4o, Memory: FAISS) . Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 7 — Figure 7: Embodied manipulation results (Agent: GPT-4.1, Human Simulator: GPT-4o, Memory: SQLite) . Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 8 — Figure 8: Online shopping results (Agent: GPT-4.1, Human Simulator: GPT-4o, Memory: SQLite). Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 9 — Figure 9: Embodied manipulation results (Agent: GPT-4o, Human Simulator: GPT-4.1, Memory: SQLite). Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 10 — Figure 10: Online shopping results (Agent: GPT-4o, Human Simulator: GPT-4.1, Memory: SQLite). Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 11 — Figure 11: Embodied manipulation results (Agent: GPT-4o, Human Simulator: Llama-4-Scout, Memory: SQLite). Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shading denote standard error.

Panel of Figure 12 — Figure 12: Online shopping results (Agent: GPT-4.1, Human Simulator: Llama-4-Scout, Memory: SQLite). Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Personal AI Agents

Personal AI agents use persistent user context and digital tools to carry out personalized, proactive tasks across apps, devices, and services.

See all 64 papers

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments
Mar 24, 2026Shuochen Liu, Junyi Zhu, Long Shu +11Memory Agent Benchmarks Long-Term Conversational Memory
University of Science and Technology of China · ESAT-PSI, KU Leuven · City University of Hong Kong +2
Personalizing Large Language Model Agents with Small Policy Models
Jul 31, 2026Dian Jin, Zhi Zhang, Huichao Li +3LLM Personalization LLM Agents
Department of Statistics and Data Science, National University of Singapore · Department of Statistics and Data Science, University of California, Los Angeles · Department of Data Science, Fudan University
Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences
Oct 21, 2026Jialin Li, Zhenhao Chen, Hanjun Luo +1Memory Agent Benchmarks Personal AI Agents
New York University Abu Dhabi, Abu Dhabi, UAE

Mar 24, 2026J/K move · Enter open · S save

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

Shuochen Liu, Junyi Zhu, Long Shu, Junda Lin+10

University of Science and Technology of China · ESAT-PSI, KU Leuven · City University of Hong Kong+2

Memory Agent Benchmarks Long-Term Conversational Memory Persona Consistency Personalized Language Models User Preference Modeling

Open paper AskPDF arXiv

Empowering large language models with long-term memory is crucial for building agents that adapt to users' evolving needs. Existing evaluations of this capability typically interleave preference-related dialogues with irrelevant conversations, reducing the task to needle-in-a-haystack retrieval while ignoring relationships between events driving user preference evolution. Such settings overlook a fundamental characteristic of real-world personalization: preferences emerge gradually and accumulate across interactions within noisy contexts. To bridge this gap, we introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall. Additionally, we incorporate (1) text variability and (2) linguistic alignment to simulate erratic user inputs and individual idiolects in real-world data. PERMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time. We design both multiple-choice and interactive tasks to probe the model's understanding of persona along the interaction timeline. Experiments demonstrate that by linking related interactions, advanced memory systems extract precise preferences and reduce token consumption, outperforming traditional semantic retrieval of raw dialogues. Nevertheless, they still struggle to maintain a coherent persona across temporal depth and cross-domain interference, highlighting the need for more robust personalized memory management in agents. Our code and data are open-sourced at https://github.com/PolarisLiu1/PERMA.

Abstract

Personal AI Agents

Explore similar work

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

Personalizing Large Language Model Agents with Small Policy Models

Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences