Self-Play RL

RL: Reinforcement Learning

Self-play reinforcement learning studies how agents can improve by competing or cooperating with other versions of themselves, reducing reliance on fixed opponents or human-labeled examples. Methods include policy-gradient training with search, equilibrium and opponent-population methods, and challenger–solver loops that generate tasks and use verifiers or rubrics for feedback. These approaches are used in games such as Go, poker, and Hanabi, as well as language-model reasoning, tool use, and interactive driving. A central challenge is that agents can overfit to their training partners or collapse into unhelpful strategies, so diverse evaluation, useful task generation, and stable rewards remain important.

Latest papers

83 papers15 papers in the last four weeks, up 88% on the four weeks before. 0.1% of all new papers.

Breaking the Impasse: Dual-Scale Evolutionary Policy Training for Social Language Agents
May 9, 2026Minzheng Wang, Run Luo, Yanbo Wang +6Reinforcement Learning Self-Play RL
MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · National University of Singapore +1
The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play
May 8, 2026Gabriele La Malfa, Emanuele La Malfa, Saar Cohen +4Self-Play RL Language Model Safety Evaluation
Department of Informatics, King’s College London · Department of Computer Science, University of Oxford · Institute for Decentralized AI +1
SPARK: Self-Play with Asymmetric Reward from Knowledge Graphs
May 7, 2026Hyobin Park, Taeseop Kim, Dong-Geol ChoiAI for Science Self-Play RL
ANTLAB, South Korea · Hanbat National University, South Korea
A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learning
May 4, 2026Arahan KujurGame-Playing Agents Self-Play RL
Independent Researcher
When Actions Disappear: Adversarial Action Removal in Self-Play Reinforcement Learning
May 4, 2026Arahan KujurSelf-Play RL Adversarial Robustness
Independent Researcher
ANCORA: Learning to Question via Manifold-Anchored Self-Play for Verifiable Reasoning
Apr 30, 2026Chengcao YangSelf-Play RL RL for Language Model Reasoning
Wuhan University Wuhan, China
Scaling Self-Play with Self-Guidance
Apr 22, 2026Luke Bailey, Kaiyue Wen, Kefan Dong +2Self-Play RL Automated Theorem Proving
Stanford University
Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
Apr 21, 2026Chengyu Huang, Sheng-Yen Chou, Zhengxin Zhang +1Self-Play RL LLM Post-Training
Department of Computer Science, Cornell University
Stratagem: Learning Transferable Reasoning via Trajectory-Modulated Game Self-Play
Apr 20, 2026Xiachong Feng, Deyi Yin, Xiaocheng Feng +9Self-Play RL RL for Language Model Reasoning
The University of Hong Kong · Harbin Institute of Technology · Harbin Institute of Technology, Shenzhen
Improving LLM Code Reasoning via Semantic Equivalence Self-Play with Formal Verification
Apr 18, 2026Antonio Valerio Miceli Barone, Poon Tsz NokAdversarial Training Self-Play RL
School of Informatics, University of Edinburgh
$π$ -Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data
Apr 15, 2026Yaocheng Zhang, Yuanheng Zhu, Wenyue Chong +7Self-Play RL Self-Evolving Agents
Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +1
Beyond Pessimism: Offline Learning in KL-regularized Games
Apr 8, 2026Yuheng Zhang, Claire Chen, Nan JiangKL-Regularized RL Zero-Sum Games
University of Illinois Urbana-Champaign. · California Institute of Technology.
Reproducing AlphaZero on Tablut: Self-Play RL for an Asymmetric Board Game
Apr 7, 2026Tõnis Lees, Tambet MatiisenGame-Playing Agents Self-Play RL
Institute of Computer Science, University of Tartu
Territory Paint Wars: Diagnosing and Mitigating Failure Modes in Competitive Multi-Agent PPO
Apr 4, 2026Diyansha SinghMulti-Agent Reinforcement Learning Self-Play RL
Independent Researcher
Heterogeneous Self-Play for Realistic Highway Traffic Simulation
Mar 31, 2026Jinkai Qiu, Alessandro Saviolo, Chaojie Wang +2Autonomous Driving Simulation Autonomous Driving Scenario Generation
PlusAI · University of Michigan - Ann Arbor
WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement
Mar 22, 2026Fangyuan Li, Pengfei Li, Shijie Wang +4Language Model Self-Improvement RL for Language Model Reasoning
Harbin Institute of Technology · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory
Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data
Feb 24, 2026Emre Can Acikgoz, Cheng Qian, Jonas Hübotter +3LLM Agent Training Self-Evolving Agents
UIUC · ETH Zurich
Regret-Guided Search Control for Efficient Learning in AlphaZero
Feb 24, 2026Yun-Jui Tsai, Wei-Yu Chen, Yan-Ru Ju +2Reinforcement Learning Game-Playing Agents
Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · College of Computing, Georgia Institute of Technology
Learning to Drive in New Cities Without Human Demonstrations
Feb 9, 2026Zilin Wang, Saeed Rahmani, Daphne Cornelisse +4Autonomous Driving Self-Play RL
WhiRL, University of Oxford · FLAIR, University of Oxford · Delft University of Technology +1
Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
Jan 26, 2026Shobhita Sundaram, John Quan, Ariel Kwiatkowski +3Self-Play RL RL for Language Model Reasoning
1MIT · 2Meta FAIR · 3New York University
Scalable Decision Making for Games of Imperfect Information
Nov 10, 2025Samuel Sokota, Eugene Vinitsky, Hengyuan Hu +3Game-Playing Agents Inference-Time Search
Carnegie Mellon University · NYU Tandon School of Engineering · Stanford University +1
OpenSIR: Open-Ended Self-Improving Reasoner
Nov 1, 2025Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis +3Self-Play RL RL for Language Model Reasoning
University of Edinburgh · Imperial College London · Huawei Technologies Research & Development (UK) Limited +1
Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models
Jun 9, 2025Mickel Liu, Liwei Jiang, Yancheng Liang +4Self-Play RL Adversarial Attacks on LLMs
Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA · Department of Computer Science, Stanford University, Stanford, CA, USA

May 9, 2026J/K move · Enter open · S save

Breaking the Impasse: Dual-Scale Evolutionary Policy Training for Social Language Agents

Minzheng Wang, Run Luo, Yanbo Wang, Zichen Liu+5

MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · National University of Singapore+1

Reinforcement Learning Self-Play RL Reinforcement Learning with Verifiable Rewards Conversational Agents Evolutionary RL

Open paper AskPDF arXiv

While Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for closed-ended tasks, extending it to open-ended social language games via self-play reveals a critical issue: evolution impasse. Due to the vast strategy space, language agents frequently converge to homogenized behaviors, leading to deterministic match outcomes that eliminate the gradient signals necessary for policy evolution. To tackle this issue, we propose Dual-scale Evolutionary Policy Training (DEPT) for social language games. DEPT introduces a time-scaled evolutionary perception mechanism that detects impasse by quantifying dual-scale value baseline divergence alongside match entropy. Upon perceiving the collapse, it then activates asymmetric advantage reshaping to dynamically modulate the optimization landscape for intervention. Thus, our method effectively restores gradient signals and enforces sustained strategic exploration. Extensive experiments on multiple social language games demonstrate that DEPT outperforms strong baselines, avoiding policy degeneration and driving the continuous evolution of social language agents.