Learning Personalized Agents from Human Feedback

Kaiqu Liang; Julia Kruk; Shengyi Qian; Xianjun Yang; Shengjie Bi; Yuanshun Yao; Shaoliang Nie; Mingyang Zhang

cs.AIFeb 18, 2026

Learning Personalized Agents from Human Feedback

Authors: Kaiqu Liang, Julia Kruk, Shengyi Qian, Xianjun Yang, Shengjie Bi, Yuanshun Yao, Shaoliang Nie, Mingyang Zhang,

Organizations: Meta Superintelligence Labs · Princeton University · Duke University

Abstract

Modern AI agents are powerful but often fail to align with the idiosyncratic, evolving preferences of individual users. Prior approaches typically rely on static datasets, either training implicit preference models on interaction history or encoding user profiles in external memory. However, these approaches struggle with new users and with preferences that change over time. We introduce Personalized Agents from Human Feedback (PAHF), a framework for continual personalization in which agents learn online from live interaction using explicit per-user memory. PAHF operationalizes a three-step loop: (1) seeking pre-action clarification to resolve ambiguity, (2) grounding actions in preferences retrieved from memory, and (3) integrating post-action feedback to update memory when preferences drift. To evaluate this capability, we develop a four-phase protocol and two benchmarks in embodied manipulation and online shopping. These benchmarks quantify an agent's ability to learn initial preferences from scratch and subsequently adapt to persona shifts. Our theoretical analysis and empirical results show that integrating explicit memory with dual feedback channels is critical: PAHF learns substantially faster and consistently outperforms both no-memory and single-channel baselines, reducing initial personalization error and enabling rapid adaptation to preference shifts.

Figures & tables

Figure 1: Static vs. continual personalization. Top: Static personalization, where offline human–AI logs are processed once to populate user profiles, and at deployment the agent only reads from this static memory. Bottom: Our continual personalization framework keeps the user in the loop during online interaction: the agent consults and updates explicit memory in the pre-action stage, and the user observes the outcome in the environment after the agent acts and provides post-action feedback, which is written back to memory to correct errors and adapt as preferences change.

Figure 2: Continual agent personalization using pre- and post-action feedback channels. On Day 1 (Initial Preference), the agent uses a pre-action interaction to learn Kate’s favorite drink (“Coke”) before acting. On Day 2 (Contextual Preference), the agent overgeneralizes this memory when Kate is sleepy; after serving Coke, it is corrected via post-action feedback (“I prefer tea when I am sleepy”). On Day 3 (Dynamic Preference), the agent acts consistently with its memory (serving tea), but Kate’s underlying preference has shifted to coffee. This error triggers another post-action update, illustrating how the framework supports continual personalization under preference drift.

Panel of Figure 3 — Figure 3: Results on embodied manipulation . Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for success rate (left), feedback frequency (middle), and ACPE (right). Shaded regions denote standard error.

Panel of Figure 4 — Figure 4: Results on online shopping . Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for success rate (left), feedback frequency (middle), and ACPE (right). Shaded regions denote standard error.

Method	Embodied		Shopping
Method	Phase 2	Phase 4	Phase 2	Phase 4
No memory	$32.3_{\pm 0.4}$	$44.8_{\pm 0.5}$	$27.8_{\pm 0.2}$	$27.0_{\pm 0.4}$
Pre-action only	$54.1_{\pm 1.1}$	$35.7_{\pm 1.0}$	$34.4_{\pm 0.5}$	$56.0_{\pm 0.7}$
Post-action only	$67.9_{\pm 1.5}$	$68.3_{\pm 1.2}$	$38.9_{\pm 0.5}$	$66.9_{\pm 0.8}$
PAHF (pre+post)	$\mathbf{70.5_{\pm 1.7}}$	$\mathbf{68.8_{\pm 1.3}}$	$\mathbf{41.3_{\pm 0.8}}$	$\mathbf{70.3_{\pm 1.1}}$

Table 1: Evaluation success rates. Phase-2 and Phase-4 success rates (%) for each method in embodied and shopping domains.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	GPT-4.1 Agent; GPT-4o Human				GPT-4o Agent; GPT-4.1 Human
	Embodied		Shopping		Embodied		Shopping
	Phase 2	Phase 4	Phase 2	Phase 4	Phase 2	Phase 4	Phase 2	Phase 4
No memory	$32.7$	$44.8$	$24.6$	$14.9$	$31.4$	$42.7$	$28.0$	$26.3$
Pre-action only	$52.8$	$35.1$	$35.8$	$59.0$	$57.8$	$30.4$	$37.2$	$55.0$
Post-action only	$81.3$	$77.1$	$36.0$	$66.6$	$66.3$	$67.8$	$39.1$	$66.2$
PAHF (pre+post)	$\mathbf{82.3}$	$\mathbf{78.8}$	$\mathbf{38.3}$	$\mathbf{67.4}$	$\mathbf{73.5}$	$\mathbf{68.3}$	$\mathbf{39.3}$	$\mathbf{67.9}$

Appendix

Table 2: Evaluation success rates (%). Phase 2 and 4 performance across different agent-simulator pairs (Memory: SQLite).

Method	SQLite Memory				FAISS Memory
	Embodied		Shopping		Embodied		Shopping
	Phase 2	Phase 4	Phase 2	Phase 4	Phase 2	Phase 4	Phase 2	Phase 4
No memory	$32.3_{\pm 0.4}$	$44.8_{\pm 0.5}$	$27.8_{\pm 0.2}$	$27.0_{\pm 0.4}$	$32.3_{\pm 0.4}$	$44.8_{\pm 0.5}$	$27.8_{\pm 0.2}$	$27.0_{\pm 0.4}$
Pre-action only	$54.1_{\pm 1.1}$	$35.7_{\pm 1.0}$	$34.4_{\pm 0.5}$	$56.0_{\pm 0.7}$	$51.8_{\pm 1.3}$	$36.2_{\pm 0.9}$	$35.9_{\pm 0.6}$	$57.2_{\pm 0.7}$
Post-action only	$67.9_{\pm 1.5}$	$68.3_{\pm 1.2}$	$38.9_{\pm 0.5}$	$66.9_{\pm 0.8}$	$66.9_{\pm 1.7}$	$66.6_{\pm 1.4}$	$39.2_{\pm 0.7}$	$65.9_{\pm 0.9}$
PAHF (pre+post)	$\mathbf{70.5_{\pm 1.7}}$	$\mathbf{68.8_{\pm 1.3}}$	$\mathbf{41.3_{\pm 0.8}}$	$\mathbf{70.3_{\pm 1.1}}$	$\mathbf{68.0_{\pm 1.8}}$	$\mathbf{68.2_{\pm 1.6}}$	$\mathbf{39.4_{\pm 1.2}}$	$\mathbf{70.8_{\pm 1.0}}$

Appendix

Table 3: Evaluation success rates (%). Phase 2 and 4 performance across memory backends and domains. Agent and Simulator: GPT-4o.

Panel of Figure 5 — Figure 5: Embodied manipulation results (Agent: GPT-4o, Human Simulator: GPT-4o, Memory: FAISS) . Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 6 — Figure 6: Online shopping result (Agent: GPT-4o, Human Simulator: GPT-4o, Memory: FAISS) . Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 7 — Figure 7: Embodied manipulation results (Agent: GPT-4.1, Human Simulator: GPT-4o, Memory: SQLite) . Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 8 — Figure 8: Online shopping results (Agent: GPT-4.1, Human Simulator: GPT-4o, Memory: SQLite). Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 9 — Figure 9: Embodied manipulation results (Agent: GPT-4o, Human Simulator: GPT-4.1, Memory: SQLite). Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 10 — Figure 10: Online shopping results (Agent: GPT-4o, Human Simulator: GPT-4.1, Memory: SQLite). Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Panel of Figure 11 — Figure 11: Embodied manipulation results (Agent: GPT-4o, Human Simulator: Llama-4-Scout, Memory: SQLite). Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shading denote standard error.

Panel of Figure 12 — Figure 12: Online shopping results (Agent: GPT-4.1, Human Simulator: Llama-4-Scout, Memory: SQLite). Top row: Phase 1 learning curves for success rate (left), feedback frequency (middle), and average cumulative personalization error (ACPE) (right). Bottom row: Phase 3 learning curves for the same metrics. Shaded regions denote standard error.

Personal AI Agents

Personal AI agents use persistent user context and digital tools to carry out personalized, proactive tasks across apps, devices, and services.

See all 64 papers

Mar 24, 2026

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

Empowering large language models with long-term memory is crucial for building agents that adapt to users' evolving needs. Existing evaluations of this capability typically interleave preference-related dialogues with irrelevant conversations, reducing the task to needle-in-a-haystack retrieval while ignoring relationships between events driving user preference evolution. Such settings overlook a fundamental characteristic of real-world personalization: preferences emerge gradually and accumulate across interactions within noisy contexts. To bridge this gap, we introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall. Additionally, we incorporate (1) text variability and (2) linguistic alignment to simulate erratic user inputs and individual idiolects in real-world data. PERMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time. We design both multiple-choice and interactive tasks to probe the model's understanding of persona along the interaction timeline. Experiments demonstrate that by linking related interactions, advanced memory systems extract precise preferences and reduce token consumption, outperforming traditional semantic retrieval of raw dialogues. Nevertheless, they still struggle to maintain a coherent persona across temporal depth and cross-domain interference, highlighting the need for more robust personalized memory management in agents. Our code and data are open-sourced at https://github.com/PolarisLiu1/PERMA.

Shuochen Liu, Junyi Zhu, Long Shu +11

University of Science and Technology of China · ESAT-PSI, KU Leuven · City University of Hong Kong +2

Memory Agent Benchmarks Long-Term Conversational Memory Persona Consistency Personalized Language Models User Preference Modeling

Ask PDF

Jul 31, 2026

Personalizing Large Language Model Agents with Small Policy Models

Large language model (LLM) agents can retrieve memory, call tools, ask clarifying questions, and vary response style, yet adapting these execution decisions to an individual user remains difficult. Fine-tuning a separate LLM is costly or impossible for proprietary systems, while prompts and memory primarily expose user information to the agent rather than adapt its execution decisions from feedback. We formulate personalization of a frozen agent as online learning of a per-user execution policy from scalar feedback observed only for the executed action. We propose FABLE (Factorized Adaptive Bandit Layer for Execution), a lightweight policy layer outside a potentially black-box host agent. FABLE factorizes memory, information-acquisition, and response decisions so feedback updates related choices; filters actions through an externally specified feasible set before exploration; and learns user-specific residual preferences relative to a fixed default-and-cost score via Bayesian contextual Thompson sampling. Under a linear residual-reward model, a calibrated variant inherits an expected-regret bound against the best feasible action. We also characterize preferences unidentifiable under persistent feasibility constraints and provide anytime-valid false-promotion control. Across personalized-reasoning, controlled-feedback, and executable tool-use evaluations, FABLE improves several preference-sensitive behaviors relative to rule-only control while remaining competitive on end-to-end task performance.

Dian Jin, Zhi Zhang, Huichao Li +3

Department of Statistics and Data Science, National University of Singapore · Department of Statistics and Data Science, University of California, Los Angeles · Department of Data Science, Fudan University

LLM Personalization LLM Agents Contextual Bandits Policy Learning Online Learning

Ask PDF

Oct 21, 2026

Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences

Personal intelligence systems are taking on increasingly complex everyday tasks, yet repeated questions, unnecessary confirmations, and unwanted outreach can turn capable assistance into interaction burden. Reducing this burden requires agents to remember how users want to be assisted. However, existing memory benchmarks focus on factual recall and personalized answers and rarely test how agents learn and act on interaction preferences. We introduce MemPABench, an interactive environment for on-policy evaluation of memory in personal agents. MemPABench poses long-horizon tasks requiring agents to decide when to clarify, confirm, or act autonomously. Four grounded simulated users hold 14 preferences across work and personal contexts. Their feedback spans 99 sessions over four months, with five event-driven changes per user testing memory updates. We evaluate preference selection, delivered behavior, and sensitivity to context and preference change. Evaluating four memory systems across four LLM backbones, we find that current memory brings limited gains in preference selection. Retrieval-based systems (i.e. Mem0, MemOS, simple RAG) do not outperform No Memory, while File-based memory improves accuracy but applies preferences beyond relevant contexts. Our analysis shows that current memory struggles to retrieve interaction preferences and adapt to preference changes. Even with the correct preference available, overriding the agent's default interaction habits remains challenging. The environment and dataset are available at https://mempabench.com.

Jialin Li, Zhenhao Chen, Hanjun Luo +1

New York University Abu Dhabi, Abu Dhabi, UAE

Memory Agent Benchmarks Personal AI Agents User Preference Modeling Long-Horizon Agent Evaluation

Ask PDF

Abstract

Personal AI Agents

Explore similar work

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

Personalizing Large Language Model Agents with Small Policy Models

Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences