Experience Replay

Follow this topic Ask about this topic

Experience replay studies how learners can store and reuse past interactions, training examples, or trajectories instead of relying only on newly generated data. Common approaches include prioritized, diverse, or recency-weighted sampling; hindsight relabeling of failed attempts; and compact memories for continual learning. The methods are used in off-policy reinforcement learning, robotic control, continual vision and language learning, and increasingly in language-model reinforcement learning, where generating new rollouts can be costly. A central tension is that replay saves data and compute but stale or imbalanced memories can hurt adaptation, forget old skills, or narrow the range of learned solutions.

Latest papers

45 papers10 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

Experience Constrained Hierarchical Federated Reinforcement Learning for Large-scale UAV Teams in Hazardous Environments
May 4, 2026Qinwei Huang, Rui Zuo, Simon Khan +1Hierarchical Federated Learning Federated RL
EECS Syracuse University Syracuse, USA · AFRL Air Force Research Laboratory Rome, USA
Data Deletion Can Help in Adaptive RL
Apr 30, 2026Param Budhraja, Aditya Gangrade, Alex Olshevsky +1Non-Stationary RL Reinforcement Learning
Department of Electrical and Computer Engineering, Boston University, Boston, MA 02215, USA · Faculty of Electrical and Computer Engineering, Systems Engineering, Boston University, Boston, MA 02215, USA
E $^2$ DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulation
Apr 30, 2026Kaiyan Zhao, Borong Zhang, Yiming Wang +4Long-Horizon Robotic Manipulation Transformer
School of Computer Science, Wuhan University, Wuhan 430072, China · State Key Lab of Internet of Things for Smart City and Department of Computer Information Science, University of Macau, Macao 999078, China
Replay-buffer engineering for noise-aware quantum circuit optimization
Apr 23, 2026Akash Kundu, Sebastian FeldQuantum Architecture Search Reinforcement Learning
Delft University of Technology, Delft, The Netherlands · Quantum Computing Division, QuTech, The Netherlands
ReConText3D: Replay-based Continual Text-to-3D Generation
Apr 15, 2026Muhammad Ahmed Ullah Khan, Muhammad Haris Bin Amir, Didier Stricker +1Text-to-3D Generation 3D Asset Generation
DFKI · RPTU Kaiserslautern-Landau
Efficient RL Training for LLMs with Experience Replay
Apr 9, 2026Charles Arnal, Vivien Cabannes, Taco Cohen +2Experience Replay RL Post-Training
FAIR at Meta · NYU Courant Institute and CDS
Leveraging Complementary Embeddings for Replay Selection in Continual Learning with Small Buffers
Apr 9, 2026Danit Yanowsky, Daphna WeinshallExperience Replay Replay-Based Continual Learning
School of Computer Science and Engineering The Hebrew University of Jerusalem
OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models
Apr 5, 2026Liyu Zhang, Kehan Li, Tingrui Han +4RL for Generative Models Off-Policy RL
College of Control Science and Engineering, Zhejiang University · Central Research Institute, Huawei
Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalization
Mar 25, 2026Fei Bai, Zhipeng Chen, Chuan Hao +6RL for Language Model Reasoning Reinforcement Learning
Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research · Beihang University
AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabeling
Mar 22, 2026Liang DingHindsight Experience Replay LLM Agent Training
The University of Sydney
DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay
Mar 17, 2026Long Li, Zhijian Zhou, Tianyi Wang +7Behavioral Diversity in RL Reinforcement Learning
Griffith University, Australia · Fudan University, China · Shanghai Innovation Institute, China +3
Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
Feb 24, 2026Xu Wan, Yansheng Wang, Wenqi Huang +1RL for Language Model Reasoning RL Post-Training
Zhejiang University · Bytedance Seed Robotics · Peking University +1
CONTHER: Context-Aware Reinforcement Learning for Robotic Manipulation with Sparse Rewards
Mar 20, 2025Maria Makarova, Qian Liu, Dzmitry TsetserukouReinforcement Learning Experience Replay
Intelligent Space Robotics Laboratory, Center for Digital Engineering, Skolkovo Institute of Science and Technology (Skoltech), 121205 Moscow, Russia · Department of Computer Science and Technology, Dalian University of Technology, China
Efficient Diversity-based Experience Replay for Deep Reinforcement Learning
Oct 27, 2024Kaiyan Zhao, Yiming Wang, Yuyang Chen +3Experience Replay Determinantal Point Process
School of Computer Science, Wuhan University, Wuhan, China · State Key Laboratory of Internet of Things for Smart City, University of Macau, Macao, China · School of Artificial Intelligence Shenzhen Polytechnic University, China
ETHER: Aligning Emergent Communication for Hindsight Experience Replay
Jul 28, 2023Kevin Yandoka Denamganaï, Daniel Hernandez, Ozan Vardal +2Reinforcement Learning Experience Replay
Department of Computer Science University of York York, UK

May 4, 2026J/K move · Enter open · S save

Experience Constrained Hierarchical Federated Reinforcement Learning for Large-scale UAV Teams in Hazardous Environments

Qinwei Huang, Rui Zuo, Simon Khan, Qinru Qiu

EECS Syracuse University Syracuse, USA · AFRL Air Force Research Laboratory Rome, USA

Hierarchical Federated Learning Federated RL Experience Replay Constrained RL Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Conventional federated learning assumes that greater learner participation improves training performance, by leveraging abundant, independently generated local data. However, in federated reinforcement learning (FRL) for unmanned aerial vehicle (UAV) teams in hazardous environments where experience generation is severely constrained by safety considerations, energy limitations, and mission duration, this assumption may break. This work introduces Experience-Constrained Hierarchical Federated Reinforcement Learning (EC-HFRL), a framework in which clusters act as federated learning agents, while multiple intra-cluster learners represent parallel learning resources that reuse a shared experience pool. We show that increasing participation does not necessarily improve learning performance. Instead, learning performance is strongly associated with experience reuse strategy and the dominance of key analytically identified gradient transition experiences within a cluster. In particular, minibatch size primarily determines effective replay exposure, while higher intra-cluster participation increases reuse level. Empirical results demonstrate that the performance regimes are strongly associated with the structure of the learning signal, rather than federated aggregation effects, clarifying the limited and secondary role of learner participation in experience-constrained FRL.