Self-Play RL

RL: Reinforcement Learning

Self-play reinforcement learning studies how agents can improve by competing or cooperating with other versions of themselves, reducing reliance on fixed opponents or human-labeled examples. Methods include policy-gradient training with search, equilibrium and opponent-population methods, and challenger–solver loops that generate tasks and use verifiers or rubrics for feedback. These approaches are used in games such as Go, poker, and Hanabi, as well as language-model reasoning, tool use, and interactive driving. A central challenge is that agents can overfit to their training partners or collapse into unhelpful strategies, so diverse evaluation, useful task generation, and stable rewards remain important.

Latest papers

83 papers15 papers in the last four weeks, up 88% on the four weeks before. 0.1% of all new papers.

Tempo-Zero: Recursive Self-Play in Executable Worlds for Video Reasoning
Oct 23, 2026Jingmin Zhu, Yanyan Zou, Xiuyu Li +10Video Reasoning Self-Play RL
Monash University · Institute of Automation, Chinese Academy of Sciences · Communication University of China +2
TerraTransfer++: Demonstration-Free End-to-End Planner
Oct 21, 2026Zikang Xiong, Weixin Li, Zhouchonghao Wu +6End-to-End Autonomous Driving Autonomous Driving Planning
Applied Intuition · University of California, Los Angeles · University of California, Berkeley
The Predictive--Optimization Gap in GuanDan Self-Play
Oct 21, 2026Qicheng Zou, Biao Yuan, Siyu Song +2Predict-Then-Optimize Self-Play RL
the University of California San Diego, La Jolla, CA, USA. · Chabiyue (Shanghai) Information Technology Co., Ltd., Shanghai, China.
Is Anyone Enforcing the Cartel? What Reinforcement Learning Teaches LLM Agents in Markets
Oct 20, 2026Hieu Tran, Hong YuAlgorithmic Collusion RL for LLM Agents
University of Massachusetts Amherst · University of Massachusetts Lowell
CasePlay: Self-Play Reinforcement Learning from Case Reports for Medical Reasoning
Oct 18, 2026Hao Wang, Zihan Wang, Kai Wu +6Self-Play RL RL for Language Model Reasoning
City University of Hong Kong · Northeastern University · Tongji University +2
R $^2$ Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop
Oct 15, 2026Mingda Zhang, Yun Huang, Qiang Huang +7LLM Agent Orchestration Self-Improving Agents
The Chinese University of Hong Kong, Shenzhen, China · Fudan University, China · University of Illinois Urbana-Champaign, USA +2
Stop Training Against Soft Users: Mentalized Adversarial Self-Play for Proactive Dialogue
Oct 15, 2026Zanwu Liu, Dan Zhang, Xiaodi Wang +6Conversational Agents Proactive Agents
School of Artificial Intelligence, Beihang University, Beijing, China · Dongchezu Technology Co., Ltd., Beijing, China
SHRED: Self-Play for Heterogeneous Road-Users without Expert Demonstrations
Oct 14, 2026Wei-Jer Chang, Zhouchonghao Wu, Akshay Rangesh +3Self-Play RL Autonomous Driving Simulation
Applied Intuition · University of California, Berkeley
Explorer-Zero: Improving Self-Play for Long-Horizon Tasks with Decomposable Verifiable Rewards
Oct 14, 2026Tianyu Liu, Davide Belli, Bence Major +2Reward Decomposition Long-Horizon LLM Agents
ETH Zurich · Qualcomm AI Research‡
Uncertainty-Driven Self-Play for Robust Multi-Agent Driving Policies
Oct 14, 2026Sajad Marvi, Julian Schmidt, Alessandro Canevaro +2Uncertainty-Aware Exploration Autonomous Driving Planning
Mercedes-Benz AG, Germany. · Department of Computer Science, University of Freiburg, Germany.
Learning to Play: Reinforcement Learning and Games
Oct 12, 2026Aske PlaatRL for Game AI Reinforcement Learning
From Hitter to Player: Separating Motor Execution and Strategy in Competitive Humanoid Table Tennis
Oct 7, 2026Fanrong Dong, Ziyi Sun, Ziqi Han +6Robotic Table Tennis Robotics
Zhejiang University. · RoboParty Lab. · Ocean University of China. +3
Faynt: Scaling and Optimizing Policies for Competitive Melee
Oct 1, 2026Ali Janati, Nikita Kuzmin, Rohit Swamy +1Reinforcement Learning Self-Play RL
Frisson Labs
Regularized policy gradient with learned mixtures of Gaussians for games with continuous actions
Sep 29, 2026Ondřej Kubíček, Viliam Lisý, Tuomas SandholmGame-Playing Agents Self-Play RL
Czech Technical University in Prague · Carnegie Mellon University · Artificial Intelligence Center
Learning to Optimize through Solver-Grounded Self-Play
Sep 28, 2026Xia Jiang, Yaoxin Wu, Chenyu Zhou +3Self-Play RL Natural Language to Optimization Modeling
Eindhoven University of Technology · Shanghai Jiaotong University
Engineering Efficient Self-Play Chess: Search, Replay, and Throughput Under Limited Compute
Sep 27, 2026Bertil BraunSelf-Play RL Chess
CompassPlay: Rewarding the Proposer for Where It Moves the Solver
Sep 26, 2026Sophia Xiao Pu, Ximeng Sun, Jiang Liu +4Self-Play RL Reward Design for RL
University of California, Santa Barbara · Advanced Micro Devices
CounterPlay: Counterfactual Post-Training for Self-Play Driving Policies
Sep 18, 2026Jiarong Wei, Yin Wu, Runkai He +1Autonomous Driving RL Post-Training
Department of Computer Science, University of Freiburg, Germany. · CARIAD SE, Germany. · Karlsruhe Institute of Technology, Germany.
Steering Equilibrium Selection in Regularized Self-Play via the Reference Policy
Sep 17, 2026Luis LealNash Equilibrium Self-Play RL
The Surprising Effectiveness of Approximate Value Iteration in Self-Play
Sep 8, 2026Raphael Boige, Amine Boumaza, Bruno ScherrerGame-Playing Agents Self-Play RL
Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France
SQL-Zero: Self-Evolving Text-to-SQL
Sep 4, 2026Daniel Machado Pedrozo, Julia Soares Dollis, Bryan Lincoln Marques de Oliveira +3Reinforcement Learning Self-Play RL
Instituto de Informática, Universidade Federal de Goiás Goiânia, GO, Brazil
Local Updates, Global Learning (LUGL): Playing Games with non-incremental Learners
Sep 3, 2026David Milec, Spyridon Samothrakis, Michael Fairbank +1Gradient-Boosted Decision Trees Game-Playing Agents
Faculty of Electrical Engineering, Czech Technical University in Prague, Prague, Czech Republic · School of Computer Science and Electronic Engineering, University of Essex, Colchester, United Kingdom · Maastricht University, the Netherlands
What Emerges and What Breaks in Self-Play Driving
Aug 31, 2026Laur Sisask, Ardi Tampuu, Tambet MatiisenReward Hacking Self-Play RL
University of Tartu, Tartu, Estonia
PokaiTrainer: Scaling Equilibrium Search to Competitive Pokémon VGC
Aug 29, 2026Max YuGame-Playing Agents Self-Play RL
Independent Researcher
Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chess
Aug 27, 2026Szymon Miłosz, Piotr Duch, Szymon GrabowskiChess RL Exploration
Institute of Applied Computer Science, Lodz University of Technology
ShuttleArena: Interpretable Self-Play in Physics-Based Badminton
Aug 26, 2026Peize DingGame-Playing Agents Self-Play RL
Columbia University New York, NY 10027, USA
The Chase Is the Curriculum, the Capture Anchors the Credit: Pursuit-Evasion Self-Play for Zero-Data LLM Reasoning
Aug 22, 2026Jing Yu, Shengchao Chen, Yiyun TanSelf-Play RL RL for Language Model Reasoning
College of Computer and Information Engineering, Xi’an University of Architecture and Technology · Australian Artificial Intelligence Institute, University of Technology Sydney
SPADE: Self-Play in Adaptive Synthetic Executable Environments
Aug 19, 2026Bo Liu, Simon Yu, Yiding Jiang +15Self-Play RL LLM Agent Self-Improvement
University of Washington · Stanford University · Northeastern University +6
UC-PSRO: Utility-Conditioned Policy-Space Response Oracles with a Communication-Dropout Curriculum for Game-Theoretic Course-of-Action Generation in Adversarial Swarms
Aug 15, 2026Phillip JiangMulti-Agent Coordination Decentralized MARL
Appsofa LLC
PureTD: Reinforcement Learning for Backgammon Money Games with No Evaluation-time Search
Aug 15, 2026Alexander L. StrehlRL for Game AI Game-Playing Agents
Independent Researcher

Oct 23, 2026J/K move · Enter open · S save

Tempo-Zero: Recursive Self-Play in Executable Worlds for Video Reasoning

Jingmin Zhu, Yanyan Zou, Xiuyu Li, James Bailey+9

Monash University · Institute of Automation, Chinese Academy of Sciences · Communication University of China+2

Video Reasoning Self-Play RL Temporal Video Grounding

Open paper AskPDF arXiv

Video reasoning requires understanding how events unfold over time, yet videos with accurate temporal annotations are expensive to collect. Self-play offers an alternative by letting a model generate and solve its own tasks without manual annotation. However, existing video self-play methods train on fixed sets of recorded videos, so they cannot alter the underlying event dynamics and rely on noisy pseudo-labels derived from the model's own outputs, such as majority voting, that risk reinforcing shared model errors. To overcome these limitations, we introduce Tempo-Zero, a self-improvement framework that closes a loop of executable environments, execution-verified self-play, and environment evolution, where executing a program renders a video with the ground-truth option and interval of every question. A single policy plays both Questioner and Solver, proposing questions and answering them with timestamped evidence lines, and is trained on execution-grounded rewards instead of model consensus. The Solver's performance steers a curriculum over tasks and difficulty levels, and the policy selects rule changes that enrich the event dynamics of later rounds. Despite being trained solely on simple programs, without supervised fine-tuning or external video data which prior methods typically require, Tempo-Zero outperforms the base models and recent self-play baselines on average across twelve real-world video benchmarks at different model scales, with particularly strong gains on temporal grounding tasks. Project page: https://alchemist-jo.github.io/tempo-zero/