Continual RL

RL: Reinforcement Learning

Continual reinforcement learning asks how an agent can keep learning as tasks or environmental dynamics change without losing earlier capabilities. Common approaches include replaying past experience, regularizing updates to protect prior behavior, reusing or expanding network components, and consolidating skills through world models or learned representations such as successor features. Researchers evaluate these methods on sequential control, navigation, Atari, robotics, and multi-agent benchmarks, with growing attention to partial observability and real-world adaptation. The central tension is balancing stability with plasticity: methods that preserve old skills can slow learning, while aggressive adaptation can cause forgetting, policy collapse, or unsafe behavior.

Latest papers

50 papers5 papers in the last four weeks, up 25% on the four weeks before. 0.0% of all new papers.

Remember Less, Adapt Longer: Bounded-State In-Context Reinforcement Learning with Recurrent Summary Memory
Oct 19, 2026Jiawen Wang, Cheng Xue, Peng Zhang +1Reinforcement Learning In-Context RL
The Australian National University
Continual Graph Multi-Agent Reinforcement Learning
Oct 7, 2026Tommaso Marzi, Ahmed Hendawy, Jan Peters +3Multi-Agent Reinforcement Learning Continual RL
IDSIA USI-SUPSI, Università della Svizzera italiana, Lugano, Switzerland · Technical University of Darmstadt, Darmstadt, Germany · Robotics Institute Germany (RIG) +4
Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning
Oct 5, 2026Chen Henry Wu, Thomas Zhang, Aditi RaghunathanContinual Learning for LLMs On-Policy Distillation
Carnegie Mellon University
How to Find and Reuse Policies for Continuous Adaptation in Lifelong Reinforcement Learning
Oct 2, 2026Saptarshi Nath, Inish M. D'Souza, Antonio Carta +2Continual RL Lifelong Learning Agents
Department of Computer Science Loughborough University · Department of Computer Science University of Pisa · Department of Computer Science
Continual Reinforcement Learning with Neuroevolution
Oct 1, 2026Eleni Nisioti, Andrea Cossu, Kathrin Korte +1Neuroevolution Loss of Plasticity
IT University of Copenhagen · University of Pisa · Sakana AI
Self-discovering RL in the Era of Experience: Is Learning History an Asset or a Burden?
Sep 27, 2026Haomin LuoReinforcement Learning Interpretability in RL
University of Cambridge · Models2 AI
Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptation
Sep 10, 2026Anqi Peter Li, Kaden KimCounterfactual Evaluation World Model Learning
Substrate Labs · UC Berkeley
When Validation Stops Learning: Auditing Update Admission for Continual Embodied Agents
Sep 9, 2026Qinzhen Ma, Ruihai WuContinual Robot Learning Safe RL
Rice University · University of California, Berkeley
A Bellman Optimality Equation for Plasticity
Sep 9, 2026Jeremy Lucas, Doina PrecupStability-Plasticity Dilemma Reinforcement Learning
Mila – Quebec Artificial Intelligence Institute Montreal, Quebec, Canada
CIDER: Continual Interactive Distillation for Embodied Reinforcement Learning
Aug 22, 2026Houlin Li, Minghui Xu, Guo Xu +8Continual Robot Learning Continual RL
AgiBot · Shanghai Jiao Tong University
Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR
Aug 19, 2026Lirui Luo, Guoxi Zhang, Hongming Xu +3RL for Language Model Reasoning Replay-Based Continual Learning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, BIGAI · Beijing Institute of Technology +1
SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learning
Aug 18, 2026Tyler Kastner, Nimrod De La Vega, Amir-massoud FarahmandLoss of Plasticity Continual Learning
University of Toronto, Vector Institute, Mila · Polytechnique Montréal, Mila
Catastrophic Forgetting in Continual Reinforcement Learning
Aug 9, 2026Emma GrahamContinual RL Catastrophic Forgetting
UCL · Machine Learning MSc · [cs.LG]
ATLAS: Adaptive Topological Learning with Abstract Successors for Continual Learning
Aug 5, 2026R. Blake Lawlor, Daniel S. BrownNon-Stationary RL Reinforcement Learning
1Kahlert School of Computing, University of Utah
Verifier-Induced Support Reshaping in On-Policy Optimization
Jul 31, 2026Shaohang Wei, Zikun Su, Feifan Song +4RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Peking University · BUPT
Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning
Jul 27, 2026Luc McCutcheon, Evangelos Chatzaroulas, Saber FallahNon-Stationary RL Neural Network Initialization
Department of Mechanical Engineering, University of Surrey, United Kingdom · Department of Computing Science, University of Surrey, United Kingdom
Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platform
Jul 27, 2026Joel Siegert, Edoardo Ghignone, Michele MagnoBackpropagation Continual Robot Learning
Center for Project-Based Learning, D-ITET, ETH Zurich
NeuroSynth: A Biologically Inspired Continual Reinforcement Learning Architecture for Mitigating Catastrophic Forgetting
Jul 24, 2026Yash KiniContinual Learning Continual RL
James Madison High School Vienna, Virginia, USA
Adaptive Multi-Horizon Reinforcement Learning
Jul 22, 2026Manoosh Samiei, Doina Precup, Paul MassetContinual RL
Department of Computer Science, McGill University, Canada · 2Mila - Quebec Artificial Intelligence Institute · Department of Psychology, McGill University, Canada
Extreme-RGMT: Continual Learning of Highly Dynamic Skills for Robust Generalist Humanoid Control
Jul 22, 2026Yubiao Ma, Han Yu, Kai Guo +5Continual Robot Learning Robot Skill Learning
School of Automation, Beijing Institute of Technology, Beijing 100081, China. · Humanoid Robotics (Shanghai) Co., Ltd., Shanghai 201203, China. · School of Mechanical Engineering, Shandong University, Jinan 250061, China.
The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL
Jul 22, 2026Gurp NijjerGenerative Replay Reinforcement Learning
Quantegra Research, Canada
RL Forgets! Towards Continual Policy Optimization
Jul 5, 2026Mao-Lin Luo, Zhe-Xu Wang, Zi-Hao Zhou +4RL Post-Training Continual Learning for VLMs
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Zhongguancun Academy +1
FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvement
Jul 1, 2026Haoran Hao, Shahram Najam Syed, Jeffrey Ichnowski +1Robot Policy Learning Robot Policy Adaptation
Carnegie Mellon University
Learning from Mistakes: Rollout-Retrieval Lifelong Policy Learning for Autonomous Driving
Jun 29, 2026Cheng Gong, Haoyang Wang, Chao Lu +2Autonomous Driving Planning Continual RL
School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081, China · School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798
Continual Robot Policy Learning via Variational Neural Dynamics
Jun 25, 2026Jiaxu Xing, Zhiyuan Zhu, Yunfan Ren +4Robot Policy Learning Robot Policy Adaptation
Robotics and Perception Group, University of Zurich
Stagnant Neuron: Towards Understanding the Plasticity Loss in Multi-Agent Reinforcement Learning Value Factorization Methods
Jun 24, 2026Zhengzhu Liu, Zeming Gao, Haoyuan Qin +7Loss of Plasticity Multi-Agent Reinforcement Learning
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, School of Informatics, Xiamen University (XMU), China · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU, China
Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training
Jun 17, 2026Jianzhe LinContinual RL RL for Code Generation
MetaAI · *Work done at Meta
When Robots Sleep: Offline Skill Consolidation for Shared-Policy Robot Learning
Jun 16, 2026Nethmi Jayasinghe, Diana Gontero, Amit Ranjan TrivediRobot Policy Learning Continual Robot Learning
Department of Electrical and Computer Engineering, University of Illinois at Chicago, Chicago, IL, USA.
Position: Deployed Reinforcement Learning should be Continual
Jun 1, 2026Parnian Behdin, Kevin Roice, Golnaz MesbahiNon-Stationary RL Reinforcement Learning
Alberta Machine Intelligence Institute (Amii), Edmonton, Canada.
Task diversity produces systematic transfer but inhibits continual reinforcement learning
May 30, 2026Purab Seth, Neil Shah, Ishaan Sinha +4Transfer Learning Reinforcement Learning
Harvard College · University of Washington · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University +1

Oct 19, 2026J/K move · Enter open · S save

Remember Less, Adapt Longer: Bounded-State In-Context Reinforcement Learning with Recurrent Summary Memory

Jiawen Wang, Cheng Xue, Peng Zhang, Jochen Renz

The Australian National University

Reinforcement Learning In-Context RL Memory-Augmented RL Transformer-Based RL Continual RL

Open paper AskPDF arXiv

In-context reinforcement learning (ICRL) enables agents to adapt to new tasks without parameter updates, relying on memory to retain what they learn from experience. For agents operating continually, existing approaches using memory trade off bounded cost against access to past experience. Approaches using growing memories, such as full-history attention and accumulating summaries, incur storage and per-decision costs that increase with task length. Fixed-size memories bound storage but restrict access: sliding windows discard older steps, while recurrent networks compress even recent interactions into a hidden state. We introduce Recurrent Summary Memory (RSM), a Transformer policy that attends exactly to a short working buffer of recent interactions and carries a fixed-capacity summary of earlier experience. When the buffer fills, RSM rewrites the summary from its previous contents and the buffer, then clears the buffer, so deployment memory and per-decision cost stay

O(1)

in task length. On long-running partially observable tasks, frozen RSM policies sustain their adapted performance for eight times the training horizon with about 1% of full-history memory, ending ahead of every evaluated baseline while growing memories deteriorate. When the task configuration changes mid-run, RSM re-adapts better than growing and recurrent memories. Clearing the summary or blocking the policy's access to it removes its benefit, while swapping in another task's summary steers behaviour towards that task. Together, these results show that our fixed-capacity summary can sustain in-context adaptation far beyond the training horizon.