Organizations: College of Information Science and Technology, Eastern Institute of Technology, Ningbo, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · College of Computer Science and Technology, China University of Petroleum(East China), Qingdao, China
Scaling Maximum Entropy Reinforcement Learning (RL) to high-dimensional humanoid control remains a fundamental challenge, as the ''curse of dimensionality'' induces severe exploration inefficiency and training instability. Consequently, highly optimized deterministic policy gradients currently dominate high-throughput regimes. We address this limitation with FastDSAC, a framework that effectively unlocks the potential of maximum entropy stochastic policies for complex continuous control. We introduce Dimension-wise Entropy Modulation (DEM) to dynamically redistribute the exploration budget, alongside a continuous distributional critic tailored to ensure accurate value estimation by mitigating both high-dimensional overestimation and discrete quantization artifacts. Extensive evaluations on HumanoidBench and a diverse set of continuous control tasks demonstrate that FastDSAC establishes state-of-the-art performance for high-dimensional stochastic policies on the evaluated benchmarks. Our method is competitive with and often outperforms strong deterministic baselines, with gains of 180% and 350% on the challenging Basketball and Balance Hard tasks, respectively.
Figures & tables
Figure 1: Performance of FastDSAC on very high-dimensional humanoid control tasks. (a) Visualizations of the challenging Basketball and Balance Hard environments. (b, c) Evaluation curves comparing FastDSAC with the FastTD3 baseline. Curves show the mean over 5 seeds, with shaded regions spanning the min-max range across seeds. FastDSAC outperforms FastTD3, achieving final returns 1.8 × and 3.5 × higher on the respective tasks with stronger sample efficiency.
Figure 2: Comparative evaluation on high-dimensional continuous control benchmarks. Learning curves across selected tasks from HumanoidBench (top three rows), IsaacLab (fourth row), and MuJoCo Playground (bottom row). FastDSAC matches or often outperforms FastTD3 and FastSAC (Standard) . It performs strongly on precision-demanding (e.g., Basketball , Insert ) and stability-critical (e.g., Balance Hard ) tasks, while retaining robust dexterous manipulation and rough-terrain locomotion performance.
Figure 3: DEM improves performance and stability in high-dimensional control. Learning curves across 12 tasks from three benchmarks comparing FastDSAC, its ablations (w/o DEM and auto- τ ), and FastTD3/FastSAC.
Figure 4: Qualitative comparison on Basketball . FastDSAC throws well and stays stable, while FastTD3 loses balance and fails.
Figure 5: Continuous critics outperform discrete C51 critics on 9 high-dimensional tasks. Curves show the mean episode return of FastDSAC compared to FastSAC (C51 + DEM) and FastSAC (standard).
Figure 6: Sim-to-real on Unitree G1. Zero-shot joystick-conditioned forward walking on the G1 robot, showing effectiveness of FastDSAC in real-world deployments.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
HumanoidBench
Isaac-Velocity
Isaac-General
MuJoCo PG
Parallel Environments ( Nenv )
128
4,096
4,096
1,024
Multi-step Return ( Nsteps )
1
8
1
1
Updates per Step ( Nupdates )
2
4
2
2
Discount Factor ( γ )
0.99
0.99
0.99
0.97
Replay Buffer Size
5×104
5×103
104
104
AMP / AMP Dtype
True / bf16
True / bf16
True / bf16
True / bf16
Appendix
Table 1: Domain-specific training protocols inherited from FastTD3 [ 24 ] . These parameters are defined by the underlying high-throughput framework rather than being tuned for FastDSAC.
Hyperparameter
HumanoidBench
IsaacLab / MuJoCo PG
DEM Temperature ( τ )
1.0 (Default)
1.0 (Default)
Log-Std Max / Min
1.0/−10
1.0/−10
Target Entropy ( H )
0
0
Initial Alpha ( αinit )
0.001
0.01
Layer Normalization
Enabled
Disabled
Polyak Update Rate ( τpolyak )
0.005
0.005
Appendix
Table 2: Unified FastDSAC algorithmic hyperparameters. These values define our core method and are fixed across all tasks within their respective domains.
Figure 7: Overview of the FastDSAC architecture. (Left) Actor with DEM: The policy dynamically redistributes the exploration budget by modulating the base standard deviation σ^ϕ(s) with weights wi (via element-wise multiplication ⊙ ). These weights are derived from logits l(s) and an environment-conditioned heterogeneity factor βe using a normalized Softmax. (Middle) Environment: Massively parallel environments collect experiences into a shared Replay Buffer for high-throughput training. (Right) Continuous Critic: The critic models the value function as a continuous Gaussian distribution to ensure accurate value estimation. It minimizes the KL Divergence between the current and target distributions, avoiding the quantization artifacts of discrete baselines (marked by × ).
Figure 8: Additional Sim-to-Real Results. Time-lapse sequences of zero-shot FastDSAC policies deployed on the Unitree G1. The robot executes backward walking under real-time joystick commands and a “squat-carry-walk-squat” motion-tracking sequence.
Figure 9: Performance comparison against concurrent baselines on 61-DoF HumanoidBench tasks. (a) FastSAC-ERA plateaus early, as global entropy thresholds restrict localized variance pruning. (b) FlashSAC struggles to scale to the 61-DoF regime despite extended training budgets, highlighting the necessity of algorithmic exploration mechanisms.
Figure 10: Wall-clock time comparison between FastDSAC and FastTD3. Evaluated on representative tasks from HumanoidBench, IsaacLab, and MuJoCo Playground. Left: absolute per-task total training time (hours). Right: relative wall-clock time normalized against FastTD3, detailing per-task ratios, the geometric-mean ratio, and the overall total-time ratio across all nine tasks.
Benchmark
Task
Steps
FastTD3 (h)
FastDSAC (h)
Rel. Time
HB
Balance Hard
250K
3.01
3.33
1.10 ×
HB
Basketball
250K
2.73
3.39
1.24 ×
HB
Hurdle
300K
4.08
4.21
1.03 ×
HB
Sit Hard
150K
1.78
2.05
1.15 ×
HB
Window
250K
2.96
3.44
1.16 ×
IL
Allegro Repose
200K
2.17
2.35
1.08 ×
Appendix
Table 3: Wall-clock time comparison on representative HumanoidBench, IsaacLab, and MuJoCo Playground tasks. All timings were measured locally on a single RTX 4090 GPU.
Figure 11: Exploration weights ( wi ) during a Basketball episode. High weights (bright) concentrate on the left thumb and wrist, while lower weights (dark) on the legs, torso, and right arm.
Figure 12: Visualization of exploration weights at different temperatures. Lower temperatures ( τ=0.5 ) result in highly concentrated weights on specific active joints, whereas higher temperatures ( τ=10.0 ) lead to a nearly uniform distribution across all joints.
Figure 13: Support sensitivity of C51 + DEM on Basketball . We vary the C51 support range while keeping the remaining training pipeline unchanged. The resulting performance differences show that discrete critics can be sensitive to support specification on this task, whereas FastDSAC avoids this additional tuning axis.
Figure 14: Ablation study of Layer Normalization. Top row (HumanoidBench, ∣A∣=61 ): Removing LayerNorm induces seed sensitivity, severe oscillations, and poor sample efficiency. Bottom row (IsaacLab & MuJoCo Playground, ∣A∣<30 ): FastDSAC remains highly stable and sample-efficient even without LayerNorm, indicating that the core algorithm is robust in lower-dimensional settings.
Figure 15: Ablation on target entropy in Basketball . The standard heuristic ( H=−61 ) forces rapid variance decay, causing high inter-seed variance and premature convergence (orange curve). FastDSAC maintains H=0 to provide sufficient exploration capacity, which is structurally stabilized by DEM. Both curves report results over 3 seeds.
Figure 16: Sensitivity to the DEM temperature τ . Learning curves on Hurdle , Room , and Stair under different fixed temperatures. The default setting τ=1 remains a robust choice overall. Room favors τ=1 , Stair shows similar performance across τ∈{0.5,1,5} , and Hurdle benefits from a larger temperature, with τ=10 achieving the best final return.
Figure 17: Full results across all benchmarks. Curves show mean return over training steps with shaded regions indicating the min-max range across 3 seeds. (a) HumanoidBench (25 tasks): FastDSAC (blue) is compared against FastTD3 (black), FastSAC Standard (red), and leading baselines including DreamerV3 (orange), TD-MPC2 (pink), SAC (purple), and PPO (green). FastDSAC often outperforms or matches these baselines across locomotion (e.g., Run , Hurdle ), fine manipulation (e.g., Insert ), and complex whole-body coordination (e.g., Basketball , Balance ), combining strong exploration with stable training. (b) IsaacLab and MuJoCo Playground (10 tasks): FastDSAC (blue) is compared against FastTD3 (black), FastSAC Standard (red), and PPO (green). On IsaacLab, FastDSAC performs strongly on locomotion (Velocity Flat/Rough H1/G1) and Repose Cube Allegro, while remaining competitive on Repose Cube Shadow. On MuJoCo Playground, FastDSAC consistently outperforms baselines on G1/T1 Joystick with lower variance and higher asymptotic returns.
Scalable reinforcement learning has popularized high-throughput sampling architectures, which significantly compresses the training time for off-policy methods in robotic locomotion. However, the rapid increase of data volume and update frequency undermines the stability of value-based methods and diminishes the plasticity of policy networks. To address these challenges, this work presents FastDSAC, a fast and high-performance variant of the Distributional Actor-Critic algorithm designed for parallel sampling scenarios. Specifically, we introduce a truncated Gaussian distribution to approximate the learned policy, which effectively excludes out-of-distribution actions that strain target value estimation while keeping necessary stochasticity for exploration. The proposed action constraint functions as an implicit regularization, which counteracts the plasticity loss typically caused by aggressive gradient updates. This preservation of network adaptability enhances sample efficiency, particularly in scenarios with a high update-to-data ratio, and accelerates the early training process. In contrast to prior fast reinforcement learning approaches that rely on discrete value distributions, our method utilizes a continuous Gaussian representation equipped with adaptive variance regulation, which improves value estimation accuracy by sampling confident and informative transitions. Extensive experiments on MuJoCo Playground and HumanoidBench demonstrate that FastDSAC not only stabilizes the overall training process but also achieves superior asymptotic performance and faster convergence compared to state-of-the-art baselines.
Guanchen Lu, Yajuan Dun, Yi Zhou +4
Chongqing University, Chongqing 400044, China · Ministry of Industry and Information Technology, Beijing 100804, China · Tsinghua University, Beijing 100084, China +1
Maximum entropy reinforcement learning (MaxEnt-RL) enables robust exploration, yet practical implementations often restrict policies to simple Gaussians. While recent approaches incorporate expressive generative policies via importance-weighted supervised learning, they are prone to importance weight collapse, which limits their scalability in high-dimensional action spaces. Our key insight is to mitigate this limitation by localizing the sampling region, avoiding the weight degeneracy induced by importance sampling over the entire action space. To instantiate this insight, we introduce FLAG (Flow policy with Latent-Augmented Guidance). FLAG augments the state space with a flow latent variable and optimizes a provably consistent proxy MaxEnt-RL objective. We empirically demonstrate that FLAG enables expressive policy optimization with limited importance samples and scales to high-dimensional control tasks. Furthermore, FLAG achieves state-of-the-art performance across challenging benchmarks. Our project webpage: https://flag-rl.github.io/
Sungha Kim, Gawon Lee, Jusuk Lee +3
Seoul National University · Georgia Institute of Technology
Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable. On-policy methods such as Proximal Policy Optimization (PPO) are widely used for their stability, but their reliance on narrowly distributed on-policy data limits accurate policy evaluation in high-dimensional state and action spaces. Off-policy methods can overcome this limitation by learning from a broader state-action distribution, yet suffer from slow convergence and instability, as fitting a value function over diverse data requires many gradient updates, causing critic errors to accumulate through bootstrapping. We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic. Motivated by scaling laws observed in supervised learning, FlashSAC sharply reduces gradient updates while compensating with larger models and higher data throughput. To maintain stability at increased scale, FlashSAC explicitly bounds weight, feature, and gradient norms, curbing critic error accumulation. Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency, with the largest gains on high-dimensional tasks such as dexterous manipulation. In sim-to-real humanoid locomotion, FlashSAC reduces training time from hours to minutes, demonstrating the promise of off-policy RL for sim-to-real transfer.