D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learning in Robotic Manipulation

Yu Zhang; Karl Mason

cs.ROMar 28, 2026

D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learning in Robotic Manipulation

Authors: Yu Zhang, Karl Mason

Organizations: School of Computer Science University of Galway Galway, Ireland

Abstract

Robotic manipulation remains challenging for reinforcement learning due to contact-rich dynamics, long horizons, and training instability. Although off-policy actor-critic algorithms such as SAC and TD3 perform well in simulation, they often suffer from policy oscillations and performance collapse in realistic settings, partly due to experience replay strategies that ignore the differing data requirements of the actor and the critic. We propose D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay, a replay framework that decouples actor and critic sampling while maintaining a shared replay buffer. The critic leverages prioritized replay for efficient value learning, whereas the actor is updated using low-error transitions to stabilize policy optimization. An adaptive anchor mechanism balances uniform and prioritized sampling based on the coefficient of variation of TD errors, and a Huber-based critic objective further improves robustness under heterogeneous reward scales. We evaluate D-SPEAR on challenging robotic manipulation tasks from the robosuite benchmark, including Block-Lifting and Door-Opening. Results demonstrate that D-SPEAR consistently outperforms strong off-policy baselines, including SAC, TD3, and DDPG, in both final performance and training stability, with ablation studies confirming the complementary roles of the actorside and critic-side replay streams.

Figures & tables

Fig. 1: Overview of the proposed D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay (D-SPEAR). The replay buffer is decomposed into an anchor set sampled uniformly and two prioritized streams: a high-TD stream for critic updates and a low-TD inverse-priority stream for actor updates. An adaptive controller adjusts the anchor ratio $\lambda$ based on the coefficient of variation (CV) of TD errors.

Parameter	Value
Environment	Lift, Door
Robot	Franka Panda
Controller	OSC_POSITION
Control frequency	20 Hz
Episode horizon	500 steps
Observation type	Low-dimensional state

TABLE I: Robotic manipulation environment and training setup parameters.

Parameter	Value
Base algorithm	SAC
Actor/Critic network	2-layer MLP (256 units)
Optimizer	Adam
Replay buffer	Shared
Batch size $N$	256
Discount factor $\gamma$	0.99

TABLE II: Algorithm and training hyperparameters.

Fig. 2: Robotic manipulation environments used in our experiments. (a) Lift : object grasping and lifting. (b) Door : contact-rich articulated door opening.

Fig. 3: Performance comparison on robosuite manipulation tasks. Episode return as a function of environment steps on Lift and Door . Solid lines denote the mean performance over $5$ random seeds, and shaded regions indicate one standard deviation.

Environment	D-SPEAR	SAC	TD3	DDPG
Lift	305.44	156.91	164.87	24.64
Door	210.37	149.27	164.40	11.35

TABLE III: Final performance on robosuite manipulation tasks. Results are reported as the mean episode return over the final 10 evaluation episodes, averaged across 5 random seeds.

Fig. 4: Ablation study on robosuite manipulation tasks. We compare the full D-SPEAR framework with variants that remove key components: w/o dual-stream , w/o low-actor , and w/o high-critic . Results are averaged over $5$ random seeds, with shaded regions indicating one standard deviation.

Prioritized Experience Replay

See all 10 papers

Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition
Aug 10, 2026Changhao Li, Yifang Zhang, Heng Zhang +6Reinforcement Learning Robotic RL
HHCM, Istituto Italiano di Tecnologia, Genoa, Italy · DIBRIS, University of Genova, Genova, Italy · Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genova, Italy +1
E $^2$ DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulation
Apr 30, 2026Kaiyan Zhao, Borong Zhang, Yiming Wang +4Long-Horizon Robotic Manipulation Transformer
School of Computer Science, Wuhan University, Wuhan 430072, China · State Key Lab of Internet of Things for Smart City and Department of Computer Information Science, University of Macau, Macao 999078, China
Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation
Sep 14, 2026Emek Barış Küçüktabak, Karankumar Patel, Zhaodong Yang +3Dexterous Manipulation Contact-Rich Robotic Manipulation
Honda Research Institute USA, San Jose, CA, USA · Skild AI, San Mateo, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA

Aug 10, 2026J/K move · Enter open · S save

Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition

Changhao Li, Yifang Zhang, Heng Zhang, Davide Torielli+5

HHCM, Istituto Italiano di Tecnologia, Genoa, Italy · DIBRIS, University of Genova, Genova, Italy · Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genova, Italy+1

Reinforcement Learning Robotic RL Robotic Manipulation Data-Efficient Robot Learning Centralized Training with Decentralized Execution

Open paper AskPDF arXiv

Real-world online reinforcement learning (RL) provides a promising approach for training robotic manipulation policies directly in the physical world, avoiding the sim-to-real gap and enabling continuous policy refinement through human-in-the-loop interaction. Recent methods have demonstrated sample-efficient learning through human intervention but remain limited to small randomization ranges and encounter challenges with the non-stationarity induced by concurrently training multiple agents. To address these limitations, we introduce a unified framework that combines centralized training with decentralized execution (CTDE) and a Hybrid Reward Architecture (HRA). This enables multiple actors to share a centralized multi-head critic. The critic is decomposed into task and grasp heads, corresponding to the sparse task reward and a potential-based grasping reward, respectively. We accordingly reformulate the critic and actor objectives to exploit the decomposed Q-values while explicitly accounting for the categorical action distribution of the discrete gripper policy. Experimental results demonstrate that the proposed framework substantially improves both sample efficiency and policy performance. We validate our approach on two robotic arms and a simulated humanoid robot across tennis ball and banana pick-and-place, pot reset, and simulated block relocation tasks under dimension-wise domain randomization, approximately 5-25x larger than those considered in prior work. Compared with a state-of-the-art baseline, our method improves the success rate from 60% to 80% on tennis ball pick-and-place, from 60% to 90% on banana pick-and-place, and from 25% to 95% on simulated block relocation, while also successfully accomplishing a task where the baseline consistently fails. Videos and more details are available at our project website: https://hil-harc.github.io/.

Abstract

Prioritized Experience Replay

Explore similar work

Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition

E2^22DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulation

Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

E $^2$ DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulation