Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradient

Haoxiang You; Yilang Liu; Davis Zong; Qian Wang; Teeratham Vitchutripop; Qi Wang; Daniel Rakita; Ian Abraham

cs.ROMay 26, 2026

Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradient

Authors: Haoxiang You, Yilang Liu, Davis Zong, Qian Wang, Teeratham Vitchutripop, Qi Wang, Daniel Rakita, Ian Abraham

Organizations: Yale University, Department of Mechanical Engineering. · Yale University, Department of Computer Science. · Shanghai Jiao Tong University, School of Computer Science. · University of Sydney, School of Electrical and Computer Engineering.

Abstract

We present the stochastic decoupled policy gradient (SDPG), a lightweight visual reinforcement learning (RL) method that trains diverse visuomotor control policies end-to-end within a few hours on a single NVIDIA RTX 4080 GPU. SDPG estimates policy gradients via random perturbations of trajectory rollouts, requiring orders of magnitude fewer batch-rendered environments and substantially reducing compute and memory overhead. On visual MuJoCo benchmarks, SDPG consistently outperforms baseline methods in training time, memory usage, and rewards. Finally, to support future research, we introduce a suite of realistic visual robotics benchmarks spanning dexterous manipulation, challenging locomotion, and demonstrate effective sim-to-real transfer on physical hardware.

Explore similar work

Accelerating Visual Policy Learning with Sampling-Based Model Predictive Control
Sep 17, 2026Yilang Liu, Haoxiang You, Qian Wang +2Visuomotor Policy Model Predictive Control
V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control
Aug 8, 2026Donghu Kim, Youngdo Lee, Hojoon Lee +6Simulation-Based Reinforcement Learning Soft Actor-Critic
Task-Relevant Representation Decoupling for Visual Reinforcement Learning Generalization
Jul 1, 2026Jinwen Wang, Youfang Lin, Xiaobo Hu +4Deepmind Control Suite Self-Supervised Representations

Sep 17, 2026 · cs.ROJ/K move · Enter open · S save

Accelerating Visual Policy Learning with Sampling-Based Model Predictive Control

Yilang Liu, Haoxiang You, Qian Wang, Daniel Rakita+1

Learning visual policies for locomotion and manipulation requires coordinating contact with the environment and can incur substantial computation and GPU memory costs. First-order policy gradients (FoPG) reduce training cost through differentiable simulation, but local optimization can converge to unintended contact patterns. To address this shortfall, we propose Sampling-Guided Policy Search (SGPS), which couples recurring action-target refinement by sampling-based model-predictive control with first-order policy optimization. Behavior cloning initializes the policy from sampled actions; training then alternates sampling-based refinement with short-horizon FoPG updates under perturbed initial states and randomized dynamics. For visual policy training, we use a decoupled FoPG formulation that excludes rendering from the computation graph, enabling direct learning from depth observations without a state-policy teacher. On a single GPU, SGPS learns policies for locomotion, obstacle traversal, crate pushing, and bimanual carrying on simulated Unitree Go2 and G1 robots. Our experiments further show that refinement improves policy learning beyond initialization and tracking alone. For hardware deployment, the distilled policy transfers zero-shot to a real Go2 and uses onboard depth to autonomously trot, crawl, clear hurdles, and switch between these behaviors.

Visuomotor Policy Model Predictive Control Visual In-Context Learning Refinement

Open paper Ask about it PDF

Abstract

Explore similar work

Accelerating Visual Policy Learning with Sampling-Based Model Predictive Control

V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control

Task-Relevant Representation Decoupling for Visual Reinforcement Learning Generalization