cs.LGNov 12, 2026

Experience-Handoff Reinforcement Learning for Recursive Self-Improvement

Authors: Doyoung Kim, Byeongguk Jeon, Kailin Zhang, Sherry Yang

Organizations: New York University · KAIST AI

Abstract

It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering (\ace{}) harness and train one model to act, reflect, and curate task-local playbooks. Our method, \acerl{}, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, \acerl{} achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.

Explore similar work

CardsList
  1. CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning

    Jan 21, 2026Tianshi Xu, Yuteng Chen, Meng LiAgentic RLReinforcement Learning

  2. EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

    Aug 6, 2026Zishan Xu, Zhiyuan Yao, Yuxin Chen +9LLM World ModelsAgentic RL

  3. Experiential Reflective Learning for Self-Improving LLM Agents

    Mar 25, 2026Marc-Antoine Allard, Arnaud Teinturier, Victor Xing +1Experience Learning for LLM AgentsSelf-Improving Agents