RL for LLM Agents

RL: Reinforcement Learning · LLM: Large Language Model

This field studies how to train large language model agents to act over multiple steps in environments such as web browsers, coding tools, games, and simulated workplaces. Common approaches use policy-gradient methods such as PPO and GRPO, verifiable task rewards, and process-level credit assignment to connect outcomes to particular actions or turns. Other methods teach agents to plan hierarchically, manage memory and reusable skills, or coordinate multiple agents, often using simulators and environment-generated feedback. A central challenge is making learning stable and efficient when rewards are sparse and tasks are long, while ensuring gains transfer to new environments and do not come from exploiting flawed reward signals.

Latest papers

623 papers134 papers in the last four weeks, up 139% on the four weeks before. 0.8% of all new papers.

Auto-Dreamer: Learning Offline Memory Consolidation for Language Agents
May 20, 2026Chongrui Ye, Yuxiang Liu, Yu Wang +5Continual Learning for LLM Agents Continual Learning
University of Illinois Urbana-Champaign · University of California San Diego
Training Language Agents to Learn from Experience
May 19, 2026Yuval Shalev, Zifeng Ding, Mateja JamnikContinual Learning for LLM Agents Self-Improving Agents
University of Cambridge
Rewarding Beliefs, Not Actions: Consistency-Guided Credit Assignment for Long-Horizon Agents
May 19, 2026Wenjie Tang, Minne Li, Sijie Huang +2Reinforcement Learning Partially Observable RL
College of Computer, National University of Defense Technology · 2Intelligent Game and Decision Lab (IGDL) · Institute of Artificial Intelligence, Xiamen University
Memory-Augmented Reinforcement Learning Agent for CAD Generation
May 19, 2026Yin Xiaolong, Liu Yu, Shen Jiahang +4Text-to-CAD Generation RL for Generative Models
Zhejiang University · Ocean University of China
What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents
May 19, 2026Xiaozhe Li, Tianyi Lyu, Yang Li +6Reinforcement Learning Credit Assignment in RL
1Tongji University · 2Shanghai AI Laboratory · 5Independent +2
Learning to Hand Off: Provably Convergent Workflow Learning under Interface Constraints
May 18, 2026Jiayu Li, Enpei Zhang, Dawei Zhou +2Sequential MARL Sequential Decision Making
Stern School of Business · New York University · Department of Computer Science +2
PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization
May 18, 2026Wonjoong Kim, Yeonjun In, Sangwu Park +2Reward Modeling Credit Assignment in RL
KAIST · Yonsei University
GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents
May 18, 2026Xiongbin Wu, Zhihao Luo, Shanzhe Lei +7Reinforcement Learning Group Relative Policy Optimization
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · East China Normal University +2
HydroAgent: Closing the Gap Between Frontier LLMs and Human Experts in Hydrologic Model Calibration via Simulator-Grounded RL
May 18, 2026Zhi Li, Songkun Yan, Jie Cao +4Simulation-Based RL Reinforcement Learning
Civil, Environmental, and Architectural Engineering, University of Colorado Boulder · Civil Engineering and Environmental Sciences, University of Oklahoma · Department of Computer Science, University of Oklahoma +3
Agent Bazaar: Enabling Economic Alignment in Multi-Agent Marketplaces
May 17, 2026Seth Karten, Cameron Crow, Chi JinAI Agent Benchmarks Multi-Agent Systems
Princeton University
Look Before You Leap: Autonomous Exploration for LLM Agents
May 15, 2026Ziang Ye, Wentao Shi, Yuxin Liu +6RL Exploration LLM Agents
University of Science and Technology of China · Meituan
ALSO: Adversarial Online Strategy Optimization for Social Agents
May 15, 2026Xiang Li, Liping Yi, Mingze Kong +3Non-Stationary RL Large Language Model-Based Social Simulation
School of Artificial Intelligence, Tianjin University, Tianjin, China · The Chinese University of Hong Kong, Shenzhen, China · East China Normal University, Shanghai, China.
Response-Conditioned Parallel-to-Sequential Orchestration for Multi-Agent Systems
May 15, 2026Nurbek Tastan, Alex Iacob, Lorenzo Sani +4Multi-Agent System Optimization Multi-Agent Orchestration
1MBZUAI, UAE · University of Cambridge, UK · 3Flower Labs, UK +1
Self-Distilled Agentic Reinforcement Learning
May 14, 2026Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8Agentic RL Self-Distillation
Zhejiang University · Meituan · Tsinghua University
COLLATOR: Compositional Multi-Agent Orchestration with Counterfactual Reinforcement Learning
May 14, 2026Xudong Chen, Yixin Liu, Hua Wei +1Multi-Agent System Optimization Multi-Agent Orchestration
SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestration
May 13, 2026Mingda Zhang, Tiesunlong Shen, Haoran Luo +4LLM Agent Orchestration LLM Agent Self-Improvement
The Chinese University of Hong Kong, Shenzhen · National University of Singapore · Nanyang Technological University +1
Scaling Retrieval-Augmented Reasoning with Parallel Search and Explicit Merging
May 13, 2026Jiabei Liu, Wenyu Mao, Junfei Tan +4Multi-Hop QA Agentic RAG
University of Science and Technology of China · WeChat Technical Architecture Department, Tecent Inc.
GAGPO: Generalized Advantage Grouped Policy Optimization
May 13, 2026Siyuan Zhu, Chao Yu, Rongxin Yang +4Reinforcement Learning Group Relative Policy Optimization
School of Computer Science and Engineering, Sun Yat-sen University · Meituan
ICRL: Learning to Internalize Self-Critique with Reinforcement Learning
May 13, 2026Jianbo Lin, Xiaomin Yu, Yi Xin +7RL for Language Models LLM Self-Refinement
Hong Kong University of Science and Technology (Guangzhou) · Nanjing University · Sun Yat-sen University +3
Reinforced Collaboration in Multi-Agent Flow Networks
May 13, 2026Zheng Wang, Yuang Liu, Yangkai DingMulti-Agent LLM Systems Multi-Agent Collaboration
Huawei Technologies Co., Ltd.
Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction
May 12, 2026Zhong Guan, Yongjian Guo, Haoran Sun +5Asynchronous RL Reinforcement Learning
Tianjin University · Tsinghua University · JDT AI Infra +1
SkillGraph: Skill-Augmented Reinforcement Learning for Agents via Evolving Skill Graphs
May 12, 2026Xiaoyuan Li, Moxin Li, Keqin Bao +4LLM Agent Skill Learning LLM Agent Skill Retrieval
University of Science and Technology of China · National University of Singapore · Alibaba Group
GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation
May 12, 2026Sijia Li, Yuchen Huang, Zifan Liu +7RL for Language Model Reasoning Credit Assignment in RL
Hong Kong University of Science and Technology · Microsoft Research Asia
Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty
May 12, 2026Joykirat Singh, Zaid Khan, Archiki Prasad +5Partially Observable RL Bayesian Inference
UNC Chapel Hill · Microsoft Research · The University of Texas at Austin
RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
May 11, 2026Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang +9Reinforcement Learning Rubric-Based RL
University of Illinois Urbana-Champaign · Google Cloud AI Research
Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agents
May 11, 2026Zhiyuan Fan, Wenwei Jin, Feng Zhang +4Reinforcement Learning Language Model Self-Improvement
1Xiaohongshu Inc. · School of Computer Science, Peking University
DeepRefine: Agent-Compiled Knowledge Refinement via Reinforcement Learning
May 11, 2026Haoyu Huang, Jiaxin Bai, Shujie Liu +6Knowledge Editing Knowledge Augmentation for Language Models
1HKUST · 2HKBU · 3Microsoft Research Asia
Verifiable Process Rewards for Agentic Reasoning
May 11, 2026Huining Yuan, Zelai Xu, Huaijie Wang +6Process Reward Models Credit Assignment in RL
Tsinghua University
Skill-R1: Agent Skill Evolution via Reinforcement Learning
May 10, 2026Yash Vishe, Rohan Surana, Xunyi Jiang +8Agent Skill Learning RL for LLM Agents
UC San Diego · Adobe Research
PiCA: Pivot-Based Credit Assignment for Search Agentic Reinforcement Learning
May 10, 2026Dongyi Liu, Yifan Niu, Qinwen Wang +2Reinforcement Learning Reward Shaping
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

May 20, 2026J/K move · Enter open · S save

Auto-Dreamer: Learning Offline Memory Consolidation for Language Agents

Chongrui Ye, Yuxiang Liu, Yu Wang, Haofei Yu+4

University of Illinois Urbana-Champaign · University of California San Diego

Continual Learning for LLM Agents Continual Learning LLM Agent Memory Persistent Memory for Language Models Memory Consolidation RL for LLM Agents

Open paper AskPDF arXiv

Language agents increasingly operate over streams of related tasks, yet existing memory systems struggle to convert accumulated experience into reusable knowledge. Retrieval-augmented and structured memory methods record per-session observations effectively, but often couple acquisition and consolidation into a single online process, leaving the agent without a global view across sessions to discover recurring patterns, abstract shared procedures, or prune redundant entries. Inspired by complementary learning systems theory, we propose Auto-Dreamer, a learned offline consolidator for language-agent memory. Auto-Dreamer decouples fast per-session memory acquisition from slow cross-session consolidation. Given a selected working region of a typed memory bank, the consolidator treats the region as read-only evidence, performs bounded tool-use to inspect entries and provenance-linked source trajectories, and synthesizes a fresh compact replacement set that abstracts across sessions and supersedes the original region. We train Auto-Dreamer via GRPO, using end-to-end agent performance as the reward signal to learn how to consolidate memories acquired through fast online experience. Trained on ScienceWorld trajectories alone, Auto-Dreamer outperforms fixed, RL-trained, and prompted memory baselines on ScienceWorld by 7 points while using an active memory bank 12

\times

smaller than the strongest baseline, and continues to lead on held-out ALFWorld and WebArena without retraining -- using 6

\times

less memory than the strongest baseline on ALFWorld.