Distributed RL

RL: Reinforcement Learning

Distributed reinforcement learning studies how agents can collect experience and learn policies across multiple machines, robots, or networked environments, making training more scalable and enabling decentralized cooperation. Common approaches include parallel actor-learner systems for gathering rollouts, asynchronous policy updates, and consensus-based methods such as distributed Q-learning for agents sharing information over a network. Work spans multi-agent games, robotics, and long-horizon language-model agents, while systems research targets rollout scheduling, experience storage, and efficient model-weight transfer. A central tension is how to increase throughput and reduce communication without letting stale experience, network constraints, or failures undermine learning and reliable deployment.

Latest papers

20 papers1 paper in the last four weeks, down 67% on the four weeks before. 0.0% of all new papers.

NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale
Oct 6, 2026Songlin Jiang, Zhiyu Li, Terry Kong +5Communication-Efficient Distributed Training Distributed RL
Aalto University · NVIDIA
WeightBridge: An Efficient Weight Transfer Library for Reinforcement Learning
Sep 21, 2026Xuanlin Jiang, Samuel Hsia, Michael Kuchnik +3RL for Language Models Distributed RL
FAIR at Meta · Harvard University · Work carried out at Meta
Conduit: An Experience Data Plane for Distributed Reinforcement Learning
Sep 21, 2026Sitong Zhang, Tuo Shi, Mario Di Francesco +2Distributed RL Reinforcement Learning
Aalto University · Shenzhen University of Advanced Technology · Zhejiang University
Online Reinforcement Learning in the Met Office Unified Model through Distributed Model-Agent Coupling
Sep 2, 2026Pritthijit Nath, Sebastian Schemm, Peter Haynes +2Weather Forecasting Distributed RL
Department of Applied Mathematics and Theoretical Physics, University of Cambridge · Department of Computer Science and Technology, University of Cambridge · Met Office Hadley Centre
Variance-Reduced Q-Learning over Static and Time-Varying Networks
Jul 24, 2026Sreejeet Maity, Feng Zhu, Aritra Mitra +1Reinforcement Learning Q-Learning
Department of Electrical and Computer Engineering, North Carolina State University · Department of Electrical and Computer Engineering, University of California San Diego, San Diego, USA
Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning
Jul 22, 2026Jian Hu, Huiying Li, Hao Zhang +8Agentic RL Reinforcement Learning
NVIDIA
Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation
Jun 23, 2026Sijie Wang, Zhengyu Qing, Zhiqiang Tan +6Reinforcement Learning Distributed RL
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Ltd.
AgentJet: A Flexible Swarm Training Framework for Agentic Reinforcement Learning
Jun 3, 2026Qingxu Fu, Boyin Liu, Shuchang Tao +2Multi-Agent LLM Systems Agentic RL
Tongyi Lab, Alibaba Group
Libra: Efficient Resource Management for Agentic RL Post-Training
Jun 2, 2026Kaiwen Chen, Xin Tan, Jingzong Li +6Agentic RL Reinforcement Learning
The Chinese University of Hong Kong · The Hang Seng University of Hong Kong · Independent Researcher
Darwin Mobile Agent: A Roadmap for Self-Evolution
May 26, 2026Daniel Beechey, Derek Yuen, Jianheng Liu +5Reinforcement Learning RL for GUI Agents
1Huawei Noah’s Ark Lab · University College London
SparseRL-Sync: Lossless Weight Synchronization with ~100x Less Communication
May 8, 2026Lucas Hu, Ranchi Zhao, Isaac Zhu +4Reinforcement Learning Communication-Efficient Distributed Training
Scitix
MADQRL: Distributed Quantum Reinforcement Learning Framework for Multi-Agent Environments
Apr 13, 2026Abhishek Sawaika, Samuel Yen-Chi Chen, Udaya Parampalli +1Quantum RL Distributed RL
Quantum Cloud Computing and Distributed Systems Lab (qCLOUDS), School of Computing and Information Systems, The University of Melbourne, Australia · Brookhaven National Laboratory, Upton, NY, USA
Fully Byzantine-Resilient Distributed Multi-Agent Q-Learning
Apr 3, 2026Haejoon Lee, Dimitra PanagouMulti-Agent Reinforcement Learning Distributed RL
Department of Robotics, University of Michigan, Ann Arbor, MI, USA
Heddle: A Distributed Orchestration System for Agentic RL Rollout
Mar 30, 2026Zili Zhang, Yinmin Zhong, Chengxu Yang +5LLM Agent Orchestration Agentic RL
School of Computer Science, Peking University · Independent Researcher
Delightful Distributed Policy Gradient
Mar 20, 2026Ian OsbandPolicy Gradient Methods Policy Learning
Google DeepMind
CALF: Communication-Aware Learning Framework for Distributed Reinforcement Learning
Mar 13, 2026Carlos Purves, Pietro Lio'Distributed RL Robust RL
University of Cambridge
Cooperative-Competitive Team Play of Real-World Craft Robots
Feb 24, 2026Rui Zhao, Xihui Li, Yizheng Zhang +6Cooperative MARL Multi-Robot Coordination
The authors are with Tencent Robotics X Laboratory, Shenzhen, Guangdong, China. · The author is with Tsinghua University, Shenzhen, Guangdong, China.
USER: A Unified and Extensible System for Real-World Online Policy Learning in Embodied AI
Feb 8, 2026Hongzhi Zang, Shu'ang Yu, Hao Lin +14Robot Policy Learning Reinforcement Learning
Tsinghua University · Shanghai AI Laboratory · Infinigence AI +3
AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
Jul 2, 2025Zhenyu Han, Ansheng You, Haibo Wang +16RL for Language Models Asynchronous RL
Huawei · Individual Researcher
A Generalist Hanabi Agent
Mar 17, 2025Arjun V Sudhakar, Hadi Nekoei, Mathieu Reymond +3RL for Game AI Cooperative MARL
Chandar Research Lab · Mila – Quebec AI Institute · Polytechnique Montréal +4

Oct 6, 2026J/K move · Enter open · S save

NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale

Songlin Jiang, Zhiyu Li, Terry Kong, Yu Yao+4

Aalto University · NVIDIA

Communication-Efficient Distributed Training Distributed RL Agentic RL

Open paper AskPDF arXiv

Agentic reinforcement learning (RL) disaggregates training from rollout, so each policy update must reach the rollout clusters before the next batch. Transferring a full 1T checkpoint for such weight synchronization (refit) takes 87.5 min between two AWS regions. Measurements of BF16 training show that about 1% of weights change their stored values per step. Recent systems exploit this sparsity but fall short on placement, exactness, or efficiency: they reimplement placement rules, assemble full tensors, rebuild values arithmetically, or use a cross-cluster collective, and none fully recovers from mid-refit failures. We present NeMo-DCR (Delta-Compressed Refit), which sends only changes yet is bit-exact: receivers obtain the same parameter and buffer bits as a dense refit. For placement, fixed affine mappings project changes from training shards into the checkpoint's canonical coordinates, residual conversion covers the other changes, and the serving runtime's native loader places all changes in receiver storage. For exactness, compressible XOR masks carry affine changes whose projection and loader preserve stored bits, and overwrites carry the others. Receivers apply both in place, retries overwrite partial writes, and a joint commit binds the policy to the baseline for the next delta. For efficiency, object storage or a relay tree streams payloads during delta construction, without a cross-cluster collective. Even at 3% and 5% change rates, NeMo-DCR refits of 30B-1T models are 12-40

\times

faster than a transport-only full-checkpoint reference. A 1T relay-tree refit at 3% takes 150 s instead of 87.5 min, making refits practical for cross-cluster agentic RL at trillion-parameter scale.