cs.ROSep 23, 2026

An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics

Authors: Teeratham Vitchutripop, Alyssa Quarles, Wenhe Zhang, Richard Xue, Daniel Rakita

Organizations: Department of Computer Science, Yale University, New Haven, CT 06520, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Over the course of a lifetime, robots may encounter novel scenarios unaccounted for in its original training that result in performance degradation. One common approach to mitigating this issue is to further grow the offline training dataset in hopes of producing a policy robust to these changes. In contrast, biological learning occurs moment-to-moment via a stream of experience, unlike the predominantly batch-based and offline nature of deep learning. Although recent works show the feasibility of stream-based deep reinforcement learning, where updates use only the latest experience, none have shown it to be a viable continual learning framework for adapting robotic policies to unseen changes. In this paper, we present the first analysis of streaming deep reinforcement learning for adaptive continual learning in robotics. In particular, we show that, following an initial pretraining phase, streaming deep RL can enable a robot to successfully adapt to unforeseen changes to itself, its environment, or goals. Our primary experiments within quadruped locomotion demonstrate that a deep neural network robotic policy with certain optimizers and plasticity loss mitigation techniques can successfully leverage domain task knowledge from its pretraining to quickly adapt online to diverse changes via stream learning, outperforming batch-based on-policy methods and improving task success rates by up to 90% over the pretrained policy. Furthermore, we perform additional evaluations on robotic manipulation tasks to determine if our previous observations extend to different robotic morphologies and scenarios. Our results show that the successes observed in quadruped locomotion can be partially realized in manipulation with stability and performance limitations. We conclude with a discussion on the limitations of our work and its implications for the future of continual robot learning.

Figures & tables

Fig. 1: An overview of our analysis on streaming deep reinforcement learning for continual adaptation in robotic systems. We examine if pretrained RL policies are able to adapt online in a stream-fashion with no replay buffers to a set of scenarios that introduce a sudden change to the robot, its environment, or goals.
MethodBroken LegGoal ShiftSlippery Floor
MaxMeanMaxMeanMaxMean
Adam0.188 ±\pm 0.0270.080 ±\pm 0.0170.108 ±\pm 0.0180.026 ±\pm 0.0020.032 ±\pm 0.0110.003 ±\pm 0.002
ObGD0.936 ±\pm 0.0300.644 ±\pm 0.0320.808 ±\pm 0.0690.306 ±\pm 0.0850.420 ±\pm 0.1910.113 ±\pm 0.056
AdaptiveObGD0.968 ±\pm 0.0110.747 ±\pm 0.0190.656 ±\pm 0.0920.181 ±\pm 0.0740.368 ±\pm 0.0440.097 ±\pm 0.028
AdaptiveObGD + CBP0.920 ±\pm 0.0250.697 ±\pm 0.0340.604 ±\pm 0.0590.246 ±\pm 0.0660.344 ±\pm 0.1730.147 ±\pm 0.088
AdaptiveObGD + LN0.864 ±\pm 0.0260.538 ±\pm 0.0110.848 ±\pm 0.0340.436 ±\pm 0.0460.676 ±\pm 0.1070.278 ±\pm 0.026
TABLE I: Table 1. Average Success Rates ±\pm STD Across 5 Seeds on Quadruped Locomotion Adaptation Scenarios.
MethodBroken LegGoal ShiftSlippery Floor
Adam0.039 ±\pm 0.000.024 ±\pm 0.000.004 ±\pm 0.00
ObGD0.124 ±\pm 0.010.099 ±\pm 0.020.059 ±\pm 0.02
AdaptiveObGD0.067 ±\pm 0.000.046 ±\pm 0.010.032 ±\pm 0.01
+ CBP0.067 ±\pm 0.010.066 ±\pm 0.010.045 ±\pm 0.02
+ LN0.062 ±\pm 0.000.063 ±\pm 0.010.061 ±\pm 0.01
TABLE II: Table 2. Average MASD ±\pm STD Across 5 Seeds During Quadruped Locomotion Adaptation Phase (lower is more stable)
Fig. 2: The performance of our different Stream-AC variants on the quadruped goal shift adaptation is shown. AdaptiveObGD + LayerNorm outperforms other variants. ObGD is able to achieve a higher performance than AdaptiveObGD, but at the cost of stability.
MethodPush Cube (Goal Shift)Transport Box (Impaired Joints)
MaxMeanMaxMean
ObGD0.396 ±\pm 0.2240.147 ±\pm 0.0840.132 ±\pm 0.0110.048 ±\pm 0.006
AdaptiveObGD0.496 ±\pm 0.0170.234 ±\pm 0.0080.148 ±\pm 0.0300.059 ±\pm 0.012
AdaptiveObGD + CBP0.396 ±\pm 0.0590.110 ±\pm 0.0170.152 ±\pm 0.0230.067 ±\pm 0.009
AdaptiveObGD + LN0.784 ±\pm 0.0460.233 ±\pm 0.0180.248 ±\pm 0.0300.122 ±\pm 0.014
Pretrained Policy0.028-0.080-
TABLE III: Table 3. Average Success Rates ±\pm STD Across 5 Seeds on Manipulation Adaptation Scenarios
Fig. 3: The performance of 4 Stream-AC variants on the Push Cube goal shift adaptation is shown. All methods show early signs of strong recovery, but performance declines over time after peaking.

Explore similar work

CardsList
Oct 18, 2024cs.LG

Streaming Deep Reinforcement Learning Finally Works

Learning from a stream of experience as it arrives, also known as streaming learning, is a core part of natural learning. However, reliable streaming learning has remained a persistent challenge in modern deep reinforcement learning (RL). Instead, most deep RL algorithms learn from old experience by storing past interactions in a buffer. We show that both classical streaming RL, such as Q-learning and actor-critic, when used with deep neural networks, and batch deep RL, such as PPO, SAC, and DQN, when adapted to the streaming setting, often fail to learn. Across 58 Atari games and 50 continuous-control tasks, we find that these methods, in aggregate, perform close to random policies despite extensive task-specific hyperparameter searches. We call this pattern stream barrier. Here, we introduce Stream-X, a shared recipe for streaming deep RL algorithms that combines signal normalization, representation stabilization, and controlled parameter updates. By applying Stream-X to several base streaming RL algorithms, we provide the first family of deep RL algorithms to overcome the stream barrier. Using one prescribed hyperparameter configuration per algorithm across tasks, Stream-X substantially improves aggregate performance, often on par with batch RL algorithms. Beyond these benchmarks, we demonstrate learning with Stream-X algorithms under nonstationarity and resource constraints. Stream-AC, one of the Stream-X algorithms, repeatedly recovers performance across alternating floor-friction regimes in simulation, outperforming the evaluated PPO and SAC baselines. It also learns a heading tracking task on a robot using proprioceptive and visual features from the on-board camera in a naturally changing laboratory environment. Stream-Q learns a Pong game from pixels directly on an ESP32-S3 microcontroller, a device with limited compute and memory.

Mohamed Elsayed, Elena Sorina Lupu, Gautham Vasan +1

Department of Computing Science, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair

Offline Reinforcement LearningQ-Learning
AskPDF
May 19, 2026cs.RO

Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning

Pretrained imitation policies have become a strong foundation for robot manipulation, but they often require online improvement to overcome execution errors, limited dataset coverage, and deployment mismatch. A central question is therefore how reinforcement learning (RL) should adapt policies after offline pretraining. Existing lightweight methods commonly apply residual corrections directly in action space, but this often leads to noisy and poorly structured exploration. In this work, we propose Z-Perturbation Reinforcement Learning (ZPRL), an approach that steers pretrained policies through a compact bottleneck latent rather than through policy weights or output actions. During offline training, we augment the policy with a plug-and-play variational information bottleneck (VIB) module to extract a task-relevant latent interface from observation embeddings. During online finetuning, the base policy is frozen and RL learns only a residual perturbation on this latent, whose decoded representation conditions the frozen action generator. We instantiate ZPRL on flow-matching policies and evaluate it on eight simulation tasks and four real-world tasks. Across diverse manipulation settings, ZPRL improves both sample efficiency and final performance over strong post-training baselines. In the real world, ZPRL improves the average success rate on four tasks by 33.7% over imitation base policies while producing smoother exploration behaviors than an action residual counterpart. These results suggest that a compact, task-aligned bottleneck latent provides an effective interface for online RL adaptation. More videos can be found at https://manutdmoon.github.io/ZPRL/.

Dongjie Yu, Kun Lei, Zhennan Jiang +2

School of Computing and Data Science, The University of Hong Kong, Hong Kong SAR. · Shanghai Qizhi Institute, Shanghai, China. · Shanghai Jiao Tong University, Shanghai, China. +2

Robot PoliciesTemporally Coherent Imitation LearningRobotic ManipulationRobot SystemsAction Generation
AskPDF
Jul 27, 2026cs.RO

Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platform

A key challenge in modern robotics is to adapt to changing environments, a challenge that is exacerbated when simulations cannot encompass every possible real-world configuration, and therefore Reinforcement Learning (RL) in the physical world becomes necessary. Continual Reinforcement Learning provides the tools to address this challenge; however, both the frameworks and the methods remain underexplored. Autonomous Racing and in particular the RoboRacer competition provide a testing ground for such methods, as learning to drive on a new track-floor combination with the least amount of new experience naturally frames a continual learning problem. This work tries to address this gap by proposing a continual RL framework based on Continual Backpropagation that is able, with only real-world data, to train a generalistic policy on a set of tracks and then fine- tune it within 15 minutes to outperform classical controllers. Furthermore, a comparison method based on offline RL is proposed, and a simulation analysis of the plasticity properties of the methods is conducted.

Joel Siegert, Edoardo Ghignone, Michele Magno

Center for Project-Based Learning, D-ITET, ETH Zurich

Autonomous RacingReinforcement Learning ControlOffline Reinforcement LearningIcr-RlRobocasaPlasticity
AskPDF