RL for LLM Agents

RL: Reinforcement Learning · LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

70 papers in the last four weeks, up 141% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 402

All topics

Safe Equilibrium Policy Optimization for Strategic Agent Policies
May 29, 2026Karthika Arumugam, Kiran Kumar Manku, Amit DhandaMulti-Agent System Optimization Game-Playing Agents
†Amazon, USA
Planner-Centric Reinforcement Learning for Deep Research with Structure-Aware Reward
May 29, 2026Mustafa Anis Hussain, Xinle Wu, Yao LuDeep Research Agents LLM Planning
National University of Singapore
Learning Agent-Compatible Context Management for Long-Horizon Tasks
May 29, 2026Lu Yi, Runlin Lei, Liuyi Yao +6Long-Horizon Agent Tasks LLM Context Management
1Renmin University of China · Work done during internship at Tongyi Lab, Alibaba Group · 2Tongyi Lab, Alibaba Group +2
SAAS: Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Search
May 28, 2026Yunbo Tang, Chengyi Yang, Shiyu Liu +4Agentic Search RL for LLM Agents
School of Informatics, Xiamen University · School of Artificial Intelligence, Jilin University
PEARL: Training Socratic Tutors with Pedagogically Aligned Reinforcement Learning
May 28, 2026Qikai Chang, Zhenrong Zhang, Linbo Chen +4Multi-Objective Reinforcement Learning Intelligent Tutoring Systems
University of Science and Technology of China · iFLYTEK Research
Learning Design Skills as Memory Policies for Agentic Photonic Inverse Design
May 28, 2026Shengchao Chen, Ting Shu, Sufen RenLLM Agent Skill Learning Physics-Informed ML
AAII, University of Technology Sydney · School of Artificial Intelligence, Shenzhen University · School of Information and Communication Engineering, Hainan University
STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments
May 28, 2026Junyang Wang, Haiyang Xu, Xi Zhang +4GUI Agents AI Agent Benchmarks
Beijing Jiaotong University · Tongyi AI Lab, Alibaba Group
GrepSeek: Training Search Agents for Direct Corpus Interaction
May 28, 2026Alireza Salemi, Chang Zeng, Atharva Nijasure +4Tool-Augmented Language Model Agents Agentic Search
University of Massachusetts Amherst · Princeton University · Carnegie Mellon University
DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agents
May 28, 2026Rongsheng Zhang, Jiji Tang, Junnan Ren +6Large Language Model-Based Role-Play Simulation Long-Horizon Agent Evaluation
1Zhejiang University · 2Fuxi AI Lab, NetEase Inc. · 3Xiamen University
PRO-CUA: Process-Reward Optimization for Computer Use Agents
May 27, 2026Yifei He, Rui Yang, Hao Bai +2Reinforcement Learning Process Reward Models
University of Illinois Urbana-Champaign
DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agents
May 27, 2026Jiamian Wang, Ruiyi Zhang, Tong Yu +5Multimodal IR Multimodal QA
Rochester Institute of Technology · ⋆Work done during an internship at Adobe Research. · Adobe Research
TRACER: Turn-level Regret Matching with Inner Reinforcement Credit for Cooperative Multi-LLM Reasoning
May 27, 2026Chusen Li, Zhou Liu, Shuigeng Zhou +1Regret Minimization Multi-Agent Reasoning
1Fudan University · 2Zhongguancun Academy · 3Academy for Advanced Interdisciplinary Studies, Peking University
ZipRL: Adaptive Multi-Turn Context Compression with Hindsight Response Replay
May 27, 2026Zhexin Hu, Li Wang, Xiaohan Wang +4LLM Compression Reinforcement Learning with Verifiable Rewards
1Meituan · Institute of Software, Chinese Academy of Sciences
Cyclical Entropy Eruption: Entropy Dynamics in Agent Reinforcement Learning
May 27, 2026Wendi Li, Shawn Im, Sharon LiAgentic RL Reinforcement Learning
Department of Computer Sciences, University of Wisconsin–Madison
SKILLC: Learning Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignment
May 27, 2026Hongxiang Lin, Zhirui Kuai, Erpeng Xue +1LLM Agent Skill Learning Credit Assignment
Meituan
Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?
May 27, 2026Yibo Zhao, Zichen Ding, Jiayi Wu +2LLM Agents Information Retrieval
School of Data Science and Engineering, East China Normal University · Shanghai AI Laboratory
Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use
May 27, 2026Abhijit Kumar, Zoey Wu, Mohit SuleyTool-Augmented Language Model Agents Learning to Defer
Microsoft AI Redmond, WA
StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning
May 26, 2026Yanfei Zhang, Xu Lin, Chenglin WuCredit Assignment in RL Step-Level Credit Assignment in RL
Independent Researcher · Tencent · DeepWisdom
EmoDistill: Offline Emotion Skill Distillation for Language Model Agents in Adversarial Negotiation
May 26, 2026Yunbo Long, Haolang Zhao, Lukas Beckenbauer +2Multi-Agent Negotiation Automated Negotiation
University of Cambridge · Technical University of Munich · Exiger LLC +1
Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning
May 26, 2026Xin Cheng, Shuo He, Lang Feng +4Group Relative Policy Optimization Credit Assignment in RL
Nanyang Technological University, Singapore · Tongyi Lab, Alibaba Group · Southeast University, China
UnityMAS-O: A General RL Optimization Framework for LLM-Based Multi-Agent Systems
May 26, 2026Yiqun Chen, Wei Yang, Erhan Zhang +14Multi-Agent LLM Systems Multi-Agent System Optimization
Renmin University of China · Xiaohongshu Inc.
What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA
May 25, 2026Yuelyu Ji, Min Gu Kwak, Hang Zhang +3Reward Hacking Retrieval-Augmented Generation
University of Pittsburgh, Pittsburgh, PA, USA
Can LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning
May 25, 2026Wei Fan, Yining Zhou, Mufan Zhang +8Agentic Search Legal QA
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China · Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada · School of Law, Tsinghua University, Beijing, China
ATOM: Instantiating Budget-Controllable Multi-Agent Collaboration via Nucleus-Electron Hierarchy
May 25, 2026Xinkui Zhao, Sai Liu, Yifan Zhang +6Multi-Agent LLM Systems Multi-Agent System Optimization
Zhejiang University · Ningbo Global Innovation Center, Zhejiang University · Zhejiang Key Laboratory of Digital-Intelligence Service Technology +1
CODESKILL: Learning Self-Evolving Skills for Coding Agents
May 25, 2026Yanzhou Li, Yiran Zhang, Xiaoyu Zhang +2Coding Agents LLM Agent Skill Learning
Nanyang Technological University · 2Zhejiang University
Evo-Attacker: Memory-Augmented Reinforcement Learning for Long-Horizon Tool Attacks on LLM-MAS
May 25, 2026Bingyu Yan, Xiaoming Zhang, Jinyu Hou +4Adversarial Attacks on LLMs LLM Agent Security
1Beihang University · 2Beijing University of Posts and Telecommunications · 3China Academy of Information and Communications Technology
ProActor: Timing-Aware Reinforcement Learning for Proactive Task Scheduling Agents
May 24, 2026Lei Ding, Bin He, Chenguang Wang +1Proactive Assistance Conversational Agents
University of California, Santa Cruz · 2Zillow Group
Test-Time Deep Thinking to Explore Implicit Rules
May 24, 2026Wentong Chen, Xin Cong, Zhong Zhang +8RL for Language Model Reasoning LLM Agents
Renmin University of China Beijing, China · Department of Statistics and Data Science, Tsinghua University Beijing, China · School of Computer Science and Engineering, UESTC Chengdu, China +4
CoRe-Code: Collaborative Reinforcement Learning for Code Generation
May 24, 2026Zhihao Dou, Qinjian Zhao, Zhongwei Wan +2RL for Code Generation Multi-Agent Coordination
Department of Computer Science Kean University · Department of Computer Science Case Western Reserve University
Hera: Learning Long-Horizon Coordination for Device-Cloud Collaborative LLM Agents
May 23, 2026Yuxin Zhang, Mengxue Hu, Zheng Lin +8LLM Agent Training LLM Routing
1Fudan University · 2Alibaba Group · 3The University of Hong Kong +4

May 29, 2026, cs.MAJ/K move · Enter open · S save

Safe Equilibrium Policy Optimization for Strategic Agent Policies

Karthika Arumugam, Kiran Kumar Manku, Amit Dhanda

†Amazon, USA

Multi-Agent System Optimization Game-Playing Agents Safe RL Policy Learning Reward Design for RL RL for LLM Agents+1

Open paper AskPDF arXiv

Language models fine-tuned with reinforcement learning typically optimize for task reward, ignoring multi-agent strategic structure. Because these agents condition on natural language game-state descriptions and emit actions through free-form generation, strategic failure modes -- exploiting weaker opponents, coordinating on harmful equilibria, and externalizing costs are inseparable from the language interface itself. We propose Safe Equilibrium Policy Optimization (\sepo{}), a training objective that augments expected payoff with explicit penalties for exploitability, collusion risk, and externality cost. We implement \sepo{} as a reward signal for Group Relative Policy Optimization (GRPO), applied to Gemma~~4 E4B-it and Qwen~~3.5-4B after supervised fine-tuning (SFT). Evaluated across five strategic domains: Iterated Prisoner's Dilemma, repeated auctions, two negotiation variants, and Kuhn Poker. \sepo{} achieves zero exploit-pool advantage in Kuhn Poker for both models, outperforms the base model on safety in four domains, and corrects the over-cooperative behavior introduced by SFT. In negotiation, \sepo{} achieves a positive-safety outcome and only the positive normalized relative advantage of any negotiation configuration. Ablation experiments confirm that per-rollout exploit computation is necessary: a shared constant penalty cancels in GRPO advantage normalization (constant control-variate property), producing zero gradient. To support further research in strategic safety for agents, we release our code and SFT datasets.

Momentum

Related topics

Latest papers 402