RL for LLM Agents

RL: Reinforcement Learning · LLM: Large Language Model

This field studies how to train large language model agents to act over multiple steps in environments such as web browsers, coding tools, games, and simulated workplaces. Common approaches use policy-gradient methods such as PPO and GRPO, verifiable task rewards, and process-level credit assignment to connect outcomes to particular actions or turns. Other methods teach agents to plan hierarchically, manage memory and reusable skills, or coordinate multiple agents, often using simulators and environment-generated feedback. A central challenge is making learning stable and efficient when rewards are sparse and tasks are long, while ensuring gains transfer to new environments and do not come from exploiting flawed reward signals.

Latest papers

616 papers134 papers in the last four weeks, up 139% on the four weeks before. 0.8% of all new papers.

SKILLC: Learning Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignment
May 27, 2026Hongxiang Lin, Zhirui Kuai, Erpeng Xue +1LLM Agent Skill Learning Credit Assignment
Meituan
Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?
May 27, 2026Yibo Zhao, Zichen Ding, Jiayi Wu +2LLM Agents Information Retrieval
School of Data Science and Engineering, East China Normal University · Shanghai AI Laboratory
Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use
May 27, 2026Abhijit Kumar, Zoey Wu, Mohit SuleyRL for LLM Agents Reward Design for RL
Microsoft AI Redmond, WA
StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning
May 26, 2026Yanfei Zhang, Xu Lin, Chenglin WuCredit Assignment in RL Step-Level Credit Assignment in RL
Independent Researcher · Tencent · DeepWisdom
EmoDistill: Offline Emotion Skill Distillation for Language Model Agents in Adversarial Negotiation
May 26, 2026Yunbo Long, Haolang Zhao, Lukas Beckenbauer +2Multi-Agent Negotiation Automated Negotiation
University of Cambridge · Technical University of Munich · Exiger LLC +1
Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning
May 26, 2026Xin Cheng, Shuo He, Lang Feng +4Group Relative Policy Optimization Credit Assignment in RL
Nanyang Technological University, Singapore · Tongyi Lab, Alibaba Group · Southeast University, China
UnityMAS-O: A General RL Optimization Framework for LLM-Based Multi-Agent Systems
May 26, 2026Yiqun Chen, Wei Yang, Erhan Zhang +14Multi-Agent LLM Systems Multi-Agent System Optimization
Renmin University of China · Xiaohongshu Inc.
What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA
May 25, 2026Yuelyu Ji, Min Gu Kwak, Hang Zhang +3Reward Hacking Retrieval-Augmented Generation
University of Pittsburgh, Pittsburgh, PA, USA
Can LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning
May 25, 2026Wei Fan, Yining Zhou, Mufan Zhang +8Agentic Search Legal QA
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China · Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada · School of Law, Tsinghua University, Beijing, China
ATOM: Instantiating Budget-Controllable Multi-Agent Collaboration via Nucleus-Electron Hierarchy
May 25, 2026Xinkui Zhao, Sai Liu, Yifan Zhang +6Multi-Agent LLM Systems Multi-Agent System Optimization
Zhejiang University · Ningbo Global Innovation Center, Zhejiang University · Zhejiang Key Laboratory of Digital-Intelligence Service Technology +1
CODESKILL: Learning Self-Evolving Skills for Coding Agents
May 25, 2026Yanzhou Li, Yiran Zhang, Xiaoyu Zhang +2Coding Agents LLM Agent Skill Learning
Nanyang Technological University · 2Zhejiang University
Evo-Attacker: Memory-Augmented Reinforcement Learning for Long-Horizon Tool Attacks on LLM-MAS
May 25, 2026Bingyu Yan, Xiaoming Zhang, Jinyu Hou +4Adversarial Attacks on LLMs LLM Agent Security
1Beihang University · 2Beijing University of Posts and Telecommunications · 3China Academy of Information and Communications Technology
ProActor: Timing-Aware Reinforcement Learning for Proactive Task Scheduling Agents
May 24, 2026Lei Ding, Bin He, Chenguang Wang +1Proactive Assistance Conversational Agents
University of California, Santa Cruz · 2Zillow Group
Test-Time Deep Thinking to Explore Implicit Rules
May 24, 2026Wentong Chen, Xin Cong, Zhong Zhang +8RL for Language Model Reasoning LLM Agents
Renmin University of China Beijing, China · Department of Statistics and Data Science, Tsinghua University Beijing, China · School of Computer Science and Engineering, UESTC Chengdu, China +4
CoRe-Code: Collaborative Reinforcement Learning for Code Generation
May 24, 2026Zhihao Dou, Qinjian Zhao, Zhongwei Wan +2RL for Code Generation Multi-Agent Coordination
Department of Computer Science Kean University · Department of Computer Science Case Western Reserve University
Hera: Learning Long-Horizon Coordination for Device-Cloud Collaborative LLM Agents
May 23, 2026Yuxin Zhang, Mengxue Hu, Zheng Lin +8LLM Agent Training LLM Routing
1Fudan University · 2Alibaba Group · 3The University of Hong Kong +4
ECHO: Terminal Agents Learn World Models for Free
May 23, 2026Vaishnavi Shrivastava, Piero Kauffmann, Ahmed Awadallah +1World Model Learning Computer-Use Agents
Microsoft Research
When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs
May 22, 2026Yifan Zeng, Yiran Wu, Yaolun Zhang +4Multi-Agent LLM Systems LLM Agent Workflow Optimization
1Oregon State University · 2Pennsylvania State University · 3Adobe Inc. +1
Co-ReAct: Rubrics as Step-Level Collaborators for ReAct Agents
May 22, 2026Jiazheng Kang, Bowen Zhang, Zixin Song +4Deep Research Agents Agentic Reasoning
Qwen Applications Business Group of Alibaba · Tsinghua University
From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning
May 22, 2026Ranxu zhang, zeyang li, Jiacheng Huang +5Agentic RL Reinforcement Learning
University of Science and Technology of China · Alibaba Group
What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA
May 21, 2026Xinjie He, Zhiyuan Lin, Su Liu +4Memory-Augmented Neural Networks Curriculum RL
Columbia University · Independent Researcher · Johns Hopkins University +1
Spreadsheet-RL: Advancing Large Language Model Agents on Realistic Spreadsheet Tasks via Reinforcement Learning
May 21, 2026Banghao Chi, Yining Xie, Mingyuan Wu +9Tool-Using Agents LLM Agent Training
University of Illinois Urbana-Champaign · Meta
DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Agent Memory
May 21, 2026Jianing Yin, Tan Tang, Yingcai WuLLM Agent Memory Persistent Memory for Language Models
State Key Lab of CAD&CG, Zhejiang University
Maestro: Reinforcement Learning to Orchestrate Hierarchical Model-Skill Ensembles
May 21, 2026Jinyang Wu, Guocheng Zhai, Ruihan Jin +7LLM Agent Orchestration Language Model Ensembles
1Tsinghua University · 2Zhejiang University · 3The Chinese University of Hong Kong +2
Mem- $π$ : Adaptive Memory through Learning When and What to Generate
May 20, 2026Xiaoqiang Wang, Chao Wang, Hadi Nekoei +5LLM Agent Memory RL for LLM Agents
1ServiceNow AI Research · 2Mila – Quebec AI Institute · 3Université de Montréal +3
Auto-Dreamer: Learning Offline Memory Consolidation for Language Agents
May 20, 2026Chongrui Ye, Yuxiang Liu, Yu Wang +5Continual Learning for LLM Agents Continual Learning
University of Illinois Urbana-Champaign · University of California San Diego
Training Language Agents to Learn from Experience
May 19, 2026Yuval Shalev, Zifeng Ding, Mateja JamnikContinual Learning for LLM Agents Self-Improving Agents
University of Cambridge
Rewarding Beliefs, Not Actions: Consistency-Guided Credit Assignment for Long-Horizon Agents
May 19, 2026Wenjie Tang, Minne Li, Sijie Huang +2Reinforcement Learning Partially Observable RL
College of Computer, National University of Defense Technology · 2Intelligent Game and Decision Lab (IGDL) · Institute of Artificial Intelligence, Xiamen University
Memory-Augmented Reinforcement Learning Agent for CAD Generation
May 19, 2026Yin Xiaolong, Liu Yu, Shen Jiahang +4Text-to-CAD Generation RL for Generative Models
Zhejiang University · Ocean University of China
What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents
May 19, 2026Xiaozhe Li, Tianyi Lyu, Yang Li +6Reinforcement Learning Credit Assignment in RL
1Tongji University · 2Shanghai AI Laboratory · 5Independent +2

May 27, 2026J/K move · Enter open · S save

SKILLC: Learning Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignment

Hongxiang Lin, Zhirui Kuai, Erpeng Xue, Lei Wang

Meituan

LLM Agent Skill Learning Credit Assignment RL for LLM Agents

Open paper AskPDF arXiv

Structured skill prompts improve exploration in long-horizon agentic reinforcement learning (RL). Skill-augmented RL methods retain external skills at inference, while skill-internalization RL methods withdraw them during training to enable autonomous performance. However, existing internalization approaches only use skill-helpfulness contrast for curriculum control, leaving the policy update unchanged and unable to distinguish skill-dependent from autonomous success. We propose SkillC, a framework based on Contrastive Skill Credit Assignment (CSCA) that converts this contrast into a direct learning signal for internalization. SkillC samples paired skill-injected and skill-free rollouts for tasks from active skill types within the same policy update, and injects their task-level contrast into optimization via a dual-stream advantage estimator that preserves global ranking while applying a one-sided correction toward skill-free success. A smoothed validation-level signal further drives an adaptive curriculum over attribution strength, rollout allocation, and monotonic active-set pruning. Experiments on ALFWorld and WebShop show that, without runtime skill access, SkillC surpasses the strongest prior skill-internalization RL baseline by 5.5% and 4.4%, respectively, while remaining competitive with skill-augmented RL methods.