FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control
Authors: Donghu Kim, Youngdo Lee, Minho Park, Kinam Kim, I Made Aswin Nahendra, Takuma Seno, Sehee Min, Daniel Palenicek, Florian Vogt, Danica Kragic, Jan Peters, Jaegul Choo, Hojoon Lee
Organizations: Holiday Robotics · KAIST · KRAFTON · Turing Inc · TU Darmstadt · KTH Royal Institute of Technology · German Research Center for AI (DFKI) · Robotics Institute Germany (RIG)
Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable. On-policy methods such as Proximal Policy Optimization (PPO) are widely used for their stability, but their reliance on narrowly distributed on-policy data limits accurate policy evaluation in high-dimensional state and action spaces. Off-policy methods can overcome this limitation by learning from a broader state-action distribution, yet suffer from slow convergence and instability, as fitting a value function over diverse data requires many gradient updates, causing critic errors to accumulate through bootstrapping. We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic. Motivated by scaling laws observed in supervised learning, FlashSAC sharply reduces gradient updates while compensating with larger models and higher data throughput. To maintain stability at increased scale, FlashSAC explicitly bounds weight, feature, and gradient norms, curbing critic error accumulation. Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency, with the largest gains on high-dimensional tasks such as dexterous manipulation. In sim-to-real humanoid locomotion, FlashSAC reduces training time from hours to minutes, demonstrating the promise of off-policy RL for sim-to-real transfer.
Figures & tables
Figure 2: \textbfs FlashSAC Architecture. The architecture consists of stacked inverted residual blocks with pre-activation batch normalization and post-RMS normalization.
Figure 3: \textbfs Results on State-Based RL, GPU-based Simulators. Learning curves on select tasks from IsaacLab [ 61 ] , ManiSkill [ 95 ] , Genesis [ 4 ] , and MuJoCo Playground [ 112 ] . We evaluate performance on \textbfs (a) low-dimensional tasks with gripper manipulation and quadruped locomotion, and \textbfs (b) high-dimensional tasks involving dexterous manipulation and humanoid locomotion. While FlashSAC is comparable to PPO [ 86 ] in low-dimensional tasks, FlashSAC significantly outperforms PPO in high-dimensional tasks.
Figure 4: \textbfs Results on State-Based RL, CPU-based Simulators. Learning curves on select tasks from MuJoCo [ 99 , 98 ] , DMC [ 96 ] , HumanoidBench [ 89 ] and MyoSuite [ 9 ] . We primarily evaluate high-dimensional tasks, involving dexterous manipulation and humanoid locomotion. FlashSAC significantly outperforms PPO, as well as strong off-policy RL and model-based RL baselines in both compute efficiency and asymptotic performance.
Figure 5: \textbfs Results on Vision-Based RL. Learning curves on selected tasks from vision-based DMControl Suite [ 96 ] . We assess learning performance in low-dimensional environments, including pendulum manipulation and bipedal locomotion. FlashSAC achieves better compute efficiency and higher asymptotic performance.
Figure 6: \textbfs Sim-to-real Stair Climbing on Unitree G1. FlashSAC achieves stable real-world stair climbing after only 4 hours of training on simulation, whereas PPO requires nearly 20 hours to reach the same capability.
Figure 7: \textbfs Off-Policy vs. On-Policy Data Coverage. We train FlashSAC for 1M steps on the Shadow Hand cube reorientation task. 2D density plots show the joint distribution of object y-position and fingertip joint actions for \textbfs (a) off-policy samples from the replay buffer and \textbfs (b) on-policy samples collected by rolling out the final policy for an additional 1M steps. Off-policy data covers a substantially broader region of the state–action space.
Figure 8: \textbfs Scaling Results. \textbfs (a) Replay buffer size trades off training stability and efficiency. \textbfs (b–e) Scaling batch size, and model capacity while reducing the UTD ratio accelerates convergence.
Figure 9: \textbfs Architectural Ablations. Starting from a standard MLP, each component is incrementally added to build up to FlashSAC . Each addition stabilizes training by constraining weight, feature, and gradient norms while reducing the condition number.
Figure 10: \textbfs Exploration Ablation Results. \textbfs (a) Unified entropy target: The optimal entropy target σtgt lies in the range 0.15 to 0.2 across tasks, enabling a unified setting without task-specific tuning. \textbfs (b) Noise repetition: Repeating action noise accelerates convergence and improves asymptotic performance.
Figure 11: Terrain configuration. Blind locomotion policies are trained on flat and diverse rough terrains, including pyramid stairs, random grids, random uniform terrain, wave terrain, and pit terrain. A terrain curriculum progressively increases difficulty across 10 levels as the policy successfully traverses the environment, enabling stable and adaptive locomotion without exteroceptive perception.
Figure 30: \textbfs Ablation: Buffer Size. Each configuration indicates the maximum size of the replay buffer. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 31: \textbfs Ablation: Batch Size. Each configuration indicates the size of the mini-batch used for training. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 32: \textbfs Ablation: Network Width. Each configuration indicates the hidden dimension of the critic network, while the actor network is proportionally scaled as well. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 33: \textbfs Ablation: Network Depth. Each configuration indicates the number of residual blocks in both actor and critic networks. Average episode returns in IsaacLab environments, plotted against the number of environment steps. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 34: \textbfs Ablation: UTD Ratio. Each configuration indicates the update-to-data ratio for training. Average episode returns in IsaacLab environments, plotted against environment steps. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 35: \textbfs Architectural Ablation Learning Curves. Average episode returns in IsaacLab environments, plotted against environment steps. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 36: \textbfs Ablation: Entropy target σtgt . Each configuration indicates the target standard deviation used for automatic temperature tuning. Average episode returns in IsaacLab environments, plotted against environment steps. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 37: \textbfs Ablation: Noise Repeat. Each configuration indicates whether exploration via noise repeat is enabled or not. Average episode returns in IsaacLab environments, plotted against environment steps. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Off-policy reinforcement learning studies how agents learn from data generated by other or earlier policies, improving sample efficiency while managing distribution shift.
Proximal Policy Optimization (PPO) has become the de facto standard for training legged robots, thanks to its robustness and scalability in massively parallel simulation environments like IsaacLab. However, its on-policy nature makes it inherently sample-inefficient, preventing its use for continuous adaptation and fine-tuning on real hardware. Soft Actor-Critic (SAC), by contrast, is an off-policy algorithm that can reuse past experience, making it a natural candidate for sim-to-real transfer workflows where the same algorithm can be used both in simulation and for online learning on the real robot. Despite these advantages, SAC has consistently failed to match PPO's empirical performance in massively parallel training settings. This work identifies the root causes of this gap and introduces targeted modifications, covering policy initialization, timeout-aware critic targets, and multi-step return estimation, that enable SAC to train stably at scale. Evaluated across multiple legged robot platforms and diverse locomotion tasks, our approach closes the performance gap with PPO entirely.
Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for data-limited replay. Through controlled experiments across eight benchmark families, we show that these stabilizers are data-regime-dependent: parameter normalization helps with narrow replay coverage but restricts value fitting when data are abundant, while clipped double-Q can be relaxed in high-throughput manipulation. Age-biased replay weighting improves learning efficiency across regimes, especially with limited network capacity. Based on these findings, we propose WarpSAC, a regime-aware family of off-policy RL algorithms. WarpSAC uses Sample Weight Decay for efficient exploitation and provides two variants: WarpSAC-L (Norm ON, clipped double-Q) for data-limited CPU-scale training, and WarpSAC-A (Norm OFF, single-Q) for data-abundant GPU-parallel training. WarpSAC improves normalized score--step AUC over FlashSAC by 4.5% across nine CPU-scale environments and 23.1% across fourteen GPU-parallel environments. It increases UnitreeG1TransportBox-v1 success rate from 19.8% to 96.4%, improves mean normalized wall-time AUC on MuJoCo Playground by 19.1%, and achieves 36.4% faster sim-to-real deployment on Unitree G1 than FlashSAC. These results show that scalable off-policy RL should adapt its stabilizers to the available data regime.
Zihao Wu, Hongyao Tang, Yi Ma +9
Tianjin University · Shanxi University · Imperial College London
Generative models such as diffusion and flow matching have become dominant paradigms for visuomotor policy learning, yet their reliance on iterative denoising incurs high inference latency incompatible with real-time robotic control. We present Fast Legendre-polynomial Action policy via Sparse History-anchored flow (FLASH Policy), which replaces discrete action-chunk generation with continuous Legendre polynomial trajectory representation. Specifically, by fitting expert demonstrations under sparse temporal sampling, FLASH enables a single inference to cover a significantly extended action horizon. To further accelerate generation, FLASH initiates the flow matching process from history polynomial coefficients rather than uninformative Gaussian noise, shortening the transport distance and enabling accurate single-step inference. Moreover, analytic polynomial differentiation directly provides desired velocity feed-forward signals to the torque controller without numerical approximation. Extensive experiments on five simulated and two real-world manipulation tasks demonstrate that FLASH achieves state-of-the-art success rates (≥92% across all tasks), a per-episode inference time of 31.40ms (up to 175× faster than diffusion policies and 18× faster than prior flow matching policies), up to 4× faster training convergence than ACT, and 5× to 7× reduction in controller tracking error compared to discrete-action baselines.
Jiaqi Bai, Jindou Jia, Yuxuan Hu +5
MARS Lab, Nanyang Technological University, Singapore