Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control

Riccardo De Monte; Matteo Cederle; Gian Antonio Susto

cs.LGMar 9, 2026

Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control

Authors: Riccardo De Monte, Matteo Cederle, Gian Antonio Susto

Organizations: Department of Information Engineering University of Padova

Abstract

State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable performance in continuous control tasks, yet their computational complexity is often incompatible with the constraints of resource-limited hardware, due to their reliance on replay buffers, batch updates, and target networks. The emerging paradigm of streaming deep RL addresses this limitation through purely online updates, achieving strong empirical performance on standard benchmarks. In this work, we propose two novel streaming deep RL algorithms, Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC), explicitly designed to be compatible with state-of-the-art batch RL methods, making them particularly suitable for on-device finetuning applications such as Sim2Real transfer. Both algorithms achieve performance comparable to state-of-the-art streaming baselines on standard benchmarks without requiring tedious per-environment hyperparameter tuning. We further investigate the batch-to-streaming transition, showing that a naive transition does not guarantee preservation of pre-trained policy performance, and propose a principled approach to address this challenge.

Figures & tables

Figure 1: Results for streaming DRL algorithms SDAC, S2AC, and Stream AC $(\lambda)$ on MuJoCo Gym and DM Control Suite tasks.

Figure 2: Ablation study for SDAC and S2AC.

Figure 3: Result for the batch methods on MuJoCo Gym and DM Control Suite tasks. TD3-norm and SAC-norm denote the versions of TD3 and SAC with data normalization and the same network architectures used for the streaming approaches.

Figure 4: Top: finetuning performance of SDAC after pre-training with TD3-norm using Adam as critic optimizer. Bottom:

L^{2}

-norm of the critic’s weights across training.

Figure 5: Finetuning performance of SDAC after pre-training with TD3-norm using SGDC as the critic optimizer. The colored horizontal dashed lines represent the agent performance before finetuning, while the gray dashed line represents SDAC from scratch, trained over 20M steps.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

	Hyperparameter	Value
	entropy coefficient $\tau$	$0.01$
Actor Optimizer	Optimizer	ObGD [ 6 ]
	Learning rate	$1.0$
	$\lambda$	$0.8$
	$\kappa$	$3.0$
Critic Optimizer	Optimizer	ObGD [ 6 ]

Appendix

Table 1: Stream AC(

\lambda

) Hyperparameters.

	Hyperparameter	Value
	Target Noise	$\mathcal{N}(0,0.2^{2})$
	Exploration noise	$\mathcal{N}(0,0.2^{2})$
Actor Optimizer	Optimizer	Adam [ 17 ]
	Learning rate	$3\text{e}-4$
	Betas	$\beta_{1}=0.9,\beta_{2}=0.999$
Critic Optimizer	Optimizer	ObGD [ 6 ]

Appendix

Table 2: SDAC Hyperparameters.

	Hyperparameter	Value
	log std max	$2$
	log std min	$-20$
Actor Optimizer	Optimizer	Adam [ 17 ]
	Learning rate	$3\text{e}-4$
	Betas	$\beta_{1}=0.9,\beta_{2}=0.999$
Critic Optimizer	Optimizer	ObGD [ 6 ]

Appendix

Table 3: S2AC Hyperparameters.

	Environment	Temperature $\alpha$
MuJoCo Gym [ 32 ]	Humanoid-v4	$0.2$
	HumanoidStandup-v4
	HalfCheetah-v4
	Hopper-v4
	Walker2d-v4
	Ant-v4

Appendix

Table 4: S2AC alpha per environment.

Figure 6: Results for streaming DRL algorithms SDAC, S2AC, and Stream AC $(\lambda)$ on MuJoCo Gym and DM Control Suite tasks. TD3 and SAC in the legend denote results with state normalization and reward scaling, trained over 3M steps.

Figure 7: Additional results for streaming DRL algorithms SDAC, S2AC, and Stream AC $(\lambda)$ on MuJoCo Gym and DM Control Suite tasks. TD3 and SAC in the legend denote results with state normalization and reward scaling, trained over 3M steps.

Figure 8: Results for streaming DRL algorithms SDAC, S2AC, Stream AC $(\lambda)$ , and AVG on MuJoCo Gym and DM Control Suite tasks.

Figure 9: Additional ablation studies for SDAC.

Figure 10: Additional ablation studies for S2AC.

	Hyperparameter	Value
No streaming params	Buffer size	$10^{6}$
	Batch size	$256$
	Exploration time steps	$5\times 10^{3}$
	Target smoothing coeff.	$\tau=0.005$
	Frequency policy update	$2$
	$\alpha$	Autotune

Appendix

Table 5: SAC and SAC-norm Hyperparameters.

	Hyperparameter	Value
No streaming params	Buffer size	$10^{6}$
	Batch size	$256$
	Exploration time steps	$25\times 10^{3}$
	Target smoothing coeff.	$\tau=0.005$
	Frequency policy update	$2$
	Target Noise	$\mathcal{N}(0,0.2^{2})$

Appendix

Table 6: TD3 and TD3-norm Hyperparameters.

Figure 11: Additional results for TD3 and SAC.

Figure 12: Results for TD3 with SGDC.

	Hyperparameter	Value
No streaming params	Buffer size	$10^{6}$
	Batch size	$256$
	Exploration time steps	$25\times 10^{3}$
	Target smoothing coeff.	$\tau=0.005$
	Frequency policy update	$2$
	Target Noise	$\mathcal{N}(0,0.2^{2})$

Appendix

Table 7: TD3-norm with SGDC Hyperparameters.

	Hyperparameter	Value
	Target Noise	$\mathcal{N}(0,0.1^{2})$
	Exploration noise	$\mathcal{N}(0,0.1^{2})$
	$Q-$ warm-up	$5,000$ steps
Actor Optimizer	Optimizer	Adam [ 17 ]
	Learning rate	$3\text{e}-4/256$
	Betas	$\beta_{1}=0.9,\beta_{2}=0.999$

Appendix

Table 8: SDAC Hyperparameters for finetuning.

Figure 13: Left: finetuning performance of SDAC after pre-training with TD3-norm using Adam as critic optimizer. Right: $L^{2}$ -norm of the critic’s weights across training.

Figure 14: Finetuning performance of SDAC after pre-training with TD3-norm using SGDC as the critic optimizer. The colored horizontal dashed lines represent the agent performance before finetuning, while the gray dashed line represents SDAC from scratch, trained over 20M steps.

Figure 15: Finetuning performance of SDAC without

Q

-warm-up after pre-training with TD3-norm using SGDC as the critic optimizer.

Figure 16: Finetuning performance of SDAC with no exploration noise after pre-training with TD3-norm using SGDC as the critic optimizer.

Figure 17: Results for TD3-norm trained from scratch in the perturbed environments.

Figure 18: Results for SDAC and Stream AC(

\lambda

) trained from scratch in the perturbed environments.

Figure 19: Finetuning performance of S2AC after pre-training with SAC-norm using SGDC as the critic optimizer.

Figure 20: Results for SAC with SGDC.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Streaming Deep Reinforcement Learning Finally Works
Oct 18, 2024Mohamed Elsayed, Elena Sorina Lupu, Gautham Vasan +1Reinforcement Learning Actor-Critic Methods
Department of Computing Science, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair
Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning
May 23, 2026Noah Farr, Aryaman Reddi, Carlo D'Eramo +1Reinforcement Learning Partially Observable RL
Technical University of Darmstadt · Zuse School ELIZA · Center for Artificial Intelligence and Data Science, University of Würzburg +1
Intentional Updates for Streaming Reinforcement Learning
Apr 21, 2026Arsalan Sharifnassab, Mohamed Elsayed, Kris De Asis +2Reinforcement Learning Temporal-Difference Learning
Openmind Research Institute · Department of Computing Science, University of Alberta.

Oct 18, 2024J/K move · Enter open · S save

Streaming Deep Reinforcement Learning Finally Works

Mohamed Elsayed, Elena Sorina Lupu, Gautham Vasan, A. Rupam Mahmood

Department of Computing Science, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair

Reinforcement Learning Actor-Critic Methods Policy Learning Streaming Algorithms

Open paper AskPDF arXiv

Learning from a stream of experience as it arrives, also known as streaming learning, is a core part of natural learning. However, reliable streaming learning has remained a persistent challenge in modern deep reinforcement learning (RL). Instead, most deep RL algorithms learn from old experience by storing past interactions in a buffer. We show that both classical streaming RL, such as Q-learning and actor-critic, when used with deep neural networks, and batch deep RL, such as PPO, SAC, and DQN, when adapted to the streaming setting, often fail to learn. Across 58 Atari games and 50 continuous-control tasks, we find that these methods, in aggregate, perform close to random policies despite extensive task-specific hyperparameter searches. We call this pattern stream barrier. Here, we introduce Stream-X, a shared recipe for streaming deep RL algorithms that combines signal normalization, representation stabilization, and controlled parameter updates. By applying Stream-X to several base streaming RL algorithms, we provide the first family of deep RL algorithms to overcome the stream barrier. Using one prescribed hyperparameter configuration per algorithm across tasks, Stream-X substantially improves aggregate performance, often on par with batch RL algorithms. Beyond these benchmarks, we demonstrate learning with Stream-X algorithms under nonstationarity and resource constraints. Stream-AC, one of the Stream-X algorithms, repeatedly recovers performance across alternating floor-friction regimes in simulation, outperforming the evaluated PPO and SAC baselines. It also learns a heading tracking task on a robot using proprioceptive and visual features from the on-board camera in a naturally changing laboratory environment. Stream-Q learns a Pong game from pixels directly on an ESP32-S3 microcontroller, a device with limited compute and memory.

Abstract

Reinforcement Learning

Explore similar work

Streaming Deep Reinforcement Learning Finally Works

Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning

Intentional Updates for Streaming Reinforcement Learning