Experience Replay

Ask about this topic

Overview

Experience replay studies how learners can store and reuse past interactions, training examples, or trajectories instead of relying only on newly generated data. Common approaches include prioritized, diverse, or recency-weighted sampling; hindsight relabeling of failed attempts; and compact memories for continual learning. The methods are used in off-policy reinforcement learning, robotic control, continual vision and language learning, and increasingly in language-model reinforcement learning, where generating new rollouts can be costly. A central tension is that replay saves data and compute but stale or imbalanced memories can hurt adaptation, forget old skills, or narrow the range of learned solutions.

Latest papers

CardsList

45 papers10 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

  1. CARES: Coverage-Aware Replay Exemplar Selection for Continual Learning

    Oct 23, 2026Arundhati S. Shanbhag, Brian B. Moser, Federico Raue +2Replay-Based Continual LearningExperience Replay

    University of Kaiserslautern-Landau, Germany · German Research Center for Artificial Intelligence, Germany

  2. Hippo: Training-Free Agent Self-Evolution via Hippocampal Replay

    Oct 17, 2026Yutong Song, Jiang Wu, Yuxuan Ren +2Memory Consolidation for LLM AgentsLLM Agent Memory

    University of California, Irvine · Independent Researcher · Rice University +1

  3. ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training

    Oct 15, 2026Yukun Tian, Junwei Su, Haofeng Xu +6RL Post-TrainingRL for Language Model Reasoning

    The University of Hong Kong · Alibaba Group · University of Science and Technology of China +1

  4. Evo-RL: Iterative Robot Reinforcement Learning from Selective Experience

    Oct 9, 2026Kai Ye, Yifan Jin, Qianhang Luo +7Robotic RLRobot Learning

    School of AI, Shanghai Jiao Tong University · Central South University · South China Normal University +1

  5. Measuring and Mitigating Solution Mode Collapse in RLVR

    Oct 8, 2026Liv G. d'Aliberti, Marwa Abdulhai, Sofiia Druchyna +2Mode CollapseRL Post-Training

    Princeton University

  6. EVISKILL: Grounding Skill Evolution in Replayable Evidence

    Oct 4, 2026Yan Zhou, Yili Wang, Yiwei Dai +2Continual Learning for LLM AgentsExperience Replay

    School of Artificial Intelligence, Jilin University, Changchun, China

  7. Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression

    Oct 3, 2026Nianli Peng, Geoffrey J. Gordon, Kianté BrantleyActor-Critic MethodsExperience Replay

    Harvard University · Carnegie Mellon University

  8. Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?

    Oct 1, 2026Alessandro Montenegro, Riccardo Venturelli, Marco Mussi +2Experience ReplayProximal Policy Optimization

    Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy

  9. ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models

    Sep 29, 2026Jingnan Pu, Zi-En Fan, Feng LianMemory-Augmented Language ModelsJoint-Embedding Predictive Architecture

    School of Automation Science and Technology, Xi’an Jiaotong University No. 28, West Xianning Road, Xi’an, Shaanxi 710049, China

  10. ReplayLens: Auditing Agents' Use of Outcomes

    Sep 28, 2026Dong Xu, Zhangfan Yang, Jiantao Wu +5AI Agent AuditingLLM Agent Memory

    School of Artificial Intelligence, Shenzhen University · EasternDawn · School of Computer Science, University of Nottingham Ningbo

  11. Reliable Replay through Spatial Coherence in Online Continual Learning

    Sep 27, 2026Haixiang Sun, Jiefu Zhang, Yinghao He +4Continual Learning for LLMsExperience Replay

    Purdue University West Lafayette, IN 47907, USA

  12. Engineering Efficient Self-Play Chess: Search, Replay, and Throughput Under Limited Compute

    Sep 27, 2026Bertil BraunSelf-Play RLChess

  13. CORTEX: A Verified Experience Layer for Generalist Agents

    Sep 27, 2026Garapati Keerthana, Manik GuptaAI Agent ReliabilityLifelong Learning Agents

    Birla Institute of Technology and Science, Pilani, Hyderabad, India

  14. On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning

    Sep 22, 2026Baptiste Bonin, Caro Strickland, Audrey DurandMulti-Objective Reinforcement LearningExperience Replay

    Mila (Quebec AI Institute) · Université Laval · CIFAR AI Chair

  15. Conduit: An Experience Data Plane for Distributed Reinforcement Learning

    Sep 21, 2026Sitong Zhang, Tuo Shi, Mario Di Francesco +2Distributed RLReinforcement Learning

    Aalto University · Shenzhen University of Advanced Technology · Zhejiang University

  16. Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learning

    Sep 16, 2026Everest Yang, Skye Thompson, George D. KonidarisNon-Stationary RLContinual Robot Learning

    Brown University, Providence, RI, USA

  17. Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning

    Sep 16, 2026Xinxin Song, Siyuan Li, Tingxiong Xiao +1Credit Assignment in RLExperience Replay

    Department of Automation, Tsinghua University · Institute for Brain and Cognitive Science, Tsinghua University

  18. Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO

    Sep 3, 2026Hyun Bin Park, Du-Seong ChangRL for Language Model ReasoningGroup Relative Policy Optimization

    Department of Artificial Intelligence, Sogang University, Seoul, Republic of Korea

  19. Memory Anchors for Continual Robot Learning

    Aug 27, 2026Maximilian Du, Zhanyi Sun, Chen Xu +3Continual Robot LearningCatastrophic Forgetting

    Stanford University · Toyota Research Institute

  20. WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation

    Aug 25, 2026Zihao Wu, Hongyao Tang, Yi Ma +9Reinforcement LearningOff-Policy RL

    Tianjin University · Shanxi University · Imperial College London

  21. Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control

    Aug 7, 2026Qi Zhao, Guozheng Ma, Yilun Kong +9Reinforcement LearningExperience Replay

    Tsinghua University · Nanyang Technological University · Mila - Quebec Artificial Intelligence Institute +1

  22. Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning

    Jul 27, 2026Parham Mohammad Panahi, Armin Ashrafi, Haoyu Du +3Deep Q-LearningExperience Replay

    Department of Computing Science, University of Alberta, Canada · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair

  23. Rethinking Transfer in Continual Learning: A Replay-Based Realisation

    Jul 17, 2026Yang Meng, Zhenya Liu, Zhuokai Zhao +1Continual Learning for LLMsExperience Replay

    Department of Computer Science, University of Chicago

  24. Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

    Jun 13, 2026Zirui Pang, Chenlong Zhang, Haosheng Tan +3Experience ReplayMachine Unlearning

    The Hong Kong University of Science and Technology (Guangzhou) · Institute of Automation, Chinese Academy of Sciences · University of Glasgow

  25. Focus, Align, and Sustain: Counteracting Gradient Dilution in Incremental Object Detection

    Jun 13, 2026Aoting Zhang, Dongbao Yang, Chang Liu +2Class-Incremental LearningDetection Transformer

    Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Nankai University +2

  26. Rollout-Level Advantage-Prioritized Experience Replay for GRPO

    Jun 3, 2026Gyeongtae Yoo, Sanghyeok Park, Soohyuk Jang +2Sample-Efficient RLGroup Relative Policy Optimization

    Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in AI, Seoul National University · AIIS, ASRI, INMC, and ISRC, Seoul National University

  27. Quantifying the Energy Floor: Direct Measurement and Replay Buffer Bias in SAC-Based HVAC Control on sbsim

    Jun 1, 2026Bo Li, Chen ZhangExperience ReplayBuilding Energy Management

    Shanghai Jiao Tong University College of Smart Energy Shanghai, China

  28. ParkingWorld: End-to-End Autonomous Parking Reinforcement Learning from Corrective Experience in 3DGS Simulation

    May 24, 2026Zhengcheng Yu, Changze Li, Haoran Liu +1End-to-End Autonomous DrivingAutonomous Parking

    Tsinghua University · Shanghai Jiaotong University

  29. When Does Non-Uniform Replay Matter in Reinforcement Learning?

    May 11, 2026Michal Korniak, Mikołaj Czarnecki, Yarden As +3Experience ReplaySample-Efficient RL

    1ETH Zurich · University of Warsaw · 3UC Berkeley +1

  30. From Passive Reuse to Active Reasoning: Grounding Large Language Models for Neuro-Symbolic Experience Replay

    May 10, 2026Yanan Xiao, Yixiang Tang, Zechen Feng +3Reinforcement LearningExperience Replay

    1Affiliation 1 · 2Affiliation 2 · 3Affiliation 3