FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control
Authors: Donghu Kim, Youngdo Lee, Minho Park, Kinam Kim, I Made Aswin Nahendra, Takuma Seno, Sehee Min, Daniel Palenicek, Florian Vogt, Danica Kragic, Jan Peters, Jaegul Choo, Hojoon Lee
Organizations: Holiday Robotics · KAIST · KRAFTON · Turing Inc · TU Darmstadt · KTH Royal Institute of Technology · German Research Center for AI (DFKI) · Robotics Institute Germany (RIG)
Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable. On-policy methods such as Proximal Policy Optimization (PPO) are widely used for their stability, but their reliance on narrowly distributed on-policy data limits accurate policy evaluation in high-dimensional state and action spaces. Off-policy methods can overcome this limitation by learning from a broader state-action distribution, yet suffer from slow convergence and instability, as fitting a value function over diverse data requires many gradient updates, causing critic errors to accumulate through bootstrapping. We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic. Motivated by scaling laws observed in supervised learning, FlashSAC sharply reduces gradient updates while compensating with larger models and higher data throughput. To maintain stability at increased scale, FlashSAC explicitly bounds weight, feature, and gradient norms, curbing critic error accumulation. Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency, with the largest gains on high-dimensional tasks such as dexterous manipulation. In sim-to-real humanoid locomotion, FlashSAC reduces training time from hours to minutes, demonstrating the promise of off-policy RL for sim-to-real transfer.
Figures & tables
Figure 2: \textbfs FlashSAC Architecture. The architecture consists of stacked inverted residual blocks with pre-activation batch normalization and post-RMS normalization.
Figure 3: \textbfs Results on State-Based RL, GPU-based Simulators. Learning curves on select tasks from IsaacLab [ 61 ] , ManiSkill [ 95 ] , Genesis [ 4 ] , and MuJoCo Playground [ 112 ] . We evaluate performance on \textbfs (a) low-dimensional tasks with gripper manipulation and quadruped locomotion, and \textbfs (b) high-dimensional tasks involving dexterous manipulation and humanoid locomotion. While FlashSAC is comparable to PPO [ 86 ] in low-dimensional tasks, FlashSAC significantly outperforms PPO in high-dimensional tasks.
Figure 4: \textbfs Results on State-Based RL, CPU-based Simulators. Learning curves on select tasks from MuJoCo [ 99 , 98 ] , DMC [ 96 ] , HumanoidBench [ 89 ] and MyoSuite [ 9 ] . We primarily evaluate high-dimensional tasks, involving dexterous manipulation and humanoid locomotion. FlashSAC significantly outperforms PPO, as well as strong off-policy RL and model-based RL baselines in both compute efficiency and asymptotic performance.
Figure 5: \textbfs Results on Vision-Based RL. Learning curves on selected tasks from vision-based DMControl Suite [ 96 ] . We assess learning performance in low-dimensional environments, including pendulum manipulation and bipedal locomotion. FlashSAC achieves better compute efficiency and higher asymptotic performance.
Figure 6: \textbfs Sim-to-real Stair Climbing on Unitree G1. FlashSAC achieves stable real-world stair climbing after only 4 hours of training on simulation, whereas PPO requires nearly 20 hours to reach the same capability.
Figure 7: \textbfs Off-Policy vs. On-Policy Data Coverage. We train FlashSAC for 1M steps on the Shadow Hand cube reorientation task. 2D density plots show the joint distribution of object y-position and fingertip joint actions for \textbfs (a) off-policy samples from the replay buffer and \textbfs (b) on-policy samples collected by rolling out the final policy for an additional 1M steps. Off-policy data covers a substantially broader region of the state–action space.
Figure 8: \textbfs Scaling Results. \textbfs (a) Replay buffer size trades off training stability and efficiency. \textbfs (b–e) Scaling batch size, and model capacity while reducing the UTD ratio accelerates convergence.
Figure 9: \textbfs Architectural Ablations. Starting from a standard MLP, each component is incrementally added to build up to FlashSAC . Each addition stabilizes training by constraining weight, feature, and gradient norms while reducing the condition number.
Figure 10: \textbfs Exploration Ablation Results. \textbfs (a) Unified entropy target: The optimal entropy target σtgt lies in the range 0.15 to 0.2 across tasks, enabling a unified setting without task-specific tuning. \textbfs (b) Noise repetition: Repeating action noise accelerates convergence and improves asymptotic performance.
Figure 11: Terrain configuration. Blind locomotion policies are trained on flat and diverse rough terrains, including pyramid stairs, random grids, random uniform terrain, wave terrain, and pit terrain. A terrain curriculum progressively increases difficulty across 10 levels as the policy successfully traverses the environment, enabling stable and adaptive locomotion without exteroceptive perception.
Figure 30: \textbfs Ablation: Buffer Size. Each configuration indicates the maximum size of the replay buffer. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 31: \textbfs Ablation: Batch Size. Each configuration indicates the size of the mini-batch used for training. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 32: \textbfs Ablation: Network Width. Each configuration indicates the hidden dimension of the critic network, while the actor network is proportionally scaled as well. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 33: \textbfs Ablation: Network Depth. Each configuration indicates the number of residual blocks in both actor and critic networks. Average episode returns in IsaacLab environments, plotted against the number of environment steps. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 34: \textbfs Ablation: UTD Ratio. Each configuration indicates the update-to-data ratio for training. Average episode returns in IsaacLab environments, plotted against environment steps. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 35: \textbfs Architectural Ablation Learning Curves. Average episode returns in IsaacLab environments, plotted against environment steps. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 36: \textbfs Ablation: Entropy target σtgt . Each configuration indicates the target standard deviation used for automatic temperature tuning. Average episode returns in IsaacLab environments, plotted against environment steps. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Figure 37: \textbfs Ablation: Noise Repeat. Each configuration indicates whether exploration via noise repeat is enabled or not. Average episode returns in IsaacLab environments, plotted against environment steps. Results are averaged over random seeds of each configuration, with shaded regions indicating 95% bootstrap confidence intervals and dotted lines denoting normalized score.
Off-policy reinforcement learning studies how agents learn from data generated by other or earlier policies, improving sample efficiency while managing distribution shift.