FastDSAC: Unlocking the Potential of Maximum Entropy RL in High-Dimensional Humanoid Control

Jun Xue; Junze Wang; Shanze Wang; Xinming Zhang; Yanjun Chen; Wei Zhang

cs.LGMar 13, 2026

FastDSAC: Unlocking the Potential of Maximum Entropy RL in High-Dimensional Humanoid Control

Authors: Jun Xue, Junze Wang, Shanze Wang, Xinming Zhang, Yanjun Chen, Wei Zhang

Organizations: College of Information Science and Technology, Eastern Institute of Technology, Ningbo, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · College of Computer Science and Technology, China University of Petroleum(East China), Qingdao, China

Abstract

Scaling Maximum Entropy Reinforcement Learning (RL) to high-dimensional humanoid control remains a fundamental challenge, as the ''curse of dimensionality'' induces severe exploration inefficiency and training instability. Consequently, highly optimized deterministic policy gradients currently dominate high-throughput regimes. We address this limitation with FastDSAC, a framework that effectively unlocks the potential of maximum entropy stochastic policies for complex continuous control. We introduce Dimension-wise Entropy Modulation (DEM) to dynamically redistribute the exploration budget, alongside a continuous distributional critic tailored to ensure accurate value estimation by mitigating both high-dimensional overestimation and discrete quantization artifacts. Extensive evaluations on HumanoidBench and a diverse set of continuous control tasks demonstrate that FastDSAC establishes state-of-the-art performance for high-dimensional stochastic policies on the evaluated benchmarks. Our method is competitive with and often outperforms strong deterministic baselines, with gains of 180% and 350% on the challenging Basketball and Balance Hard tasks, respectively.

Figures & tables

Panel of Figure 1 — Figure 1: Performance of FastDSAC on very high-dimensional humanoid control tasks. (a) Visualizations of the challenging Basketball and Balance Hard environments. (b, c) Evaluation curves comparing FastDSAC with the FastTD3 baseline. Curves show the mean over 5 seeds, with shaded regions spanning the min-max range across seeds. FastDSAC outperforms FastTD3, achieving final returns 1.8 $\times$ and 3.5 $\times$ higher on the respective tasks with stronger sample efficiency.

Figure 2: Comparative evaluation on high-dimensional continuous control benchmarks. Learning curves across selected tasks from HumanoidBench (top three rows), IsaacLab (fourth row), and MuJoCo Playground (bottom row). FastDSAC matches or often outperforms FastTD3 and FastSAC (Standard) . It performs strongly on precision-demanding (e.g., Basketball , Insert ) and stability-critical (e.g., Balance Hard ) tasks, while retaining robust dexterous manipulation and rough-terrain locomotion performance.

Figure 3: DEM improves performance and stability in high-dimensional control. Learning curves across 12 tasks from three benchmarks comparing FastDSAC, its ablations (w/o DEM and auto-

\tau

), and FastTD3/FastSAC.

Figure 4: Qualitative comparison on Basketball . FastDSAC throws well and stays stable, while FastTD3 loses balance and fails.

Figure 5: Continuous critics outperform discrete C51 critics on 9 high-dimensional tasks. Curves show the mean episode return of FastDSAC compared to FastSAC (C51

+

DEM) and FastSAC (standard).

Figure 6: Sim-to-real on Unitree G1. Zero-shot joystick-conditioned forward walking on the G1 robot, showing effectiveness of FastDSAC in real-world deployments.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	HumanoidBench	Isaac-Velocity	Isaac-General	MuJoCo PG
Parallel Environments ( $N_{env}$ )	128	4,096	4,096	1,024
Multi-step Return ( $N_{steps}$ )	1	8	1	1
Updates per Step ( $N_{updates}$ )	2	4	2	2
Discount Factor ( $\gamma$ )	0.99	0.99	0.99	0.97
Replay Buffer Size	$5\times 10^{4}$	$5\times 10^{3}$	$10^{4}$	$10^{4}$
AMP / AMP Dtype	True / bf16	True / bf16	True / bf16	True / bf16

Appendix

Table 1: Domain-specific training protocols inherited from FastTD3 [ 24 ] . These parameters are defined by the underlying high-throughput framework rather than being tuned for FastDSAC.

Hyperparameter	HumanoidBench	IsaacLab / MuJoCo PG
DEM Temperature ( $\tau$ )	1.0 (Default)	1.0 (Default)
Log-Std Max / Min	$1.0\,/\,-10$	$1.0\,/\,-10$
Target Entropy ( $\mathcal{H}$ )	0	0
Initial Alpha ( $\alpha_{init}$ )	0.001	0.01
Layer Normalization	Enabled	Disabled
Polyak Update Rate ( $\tau_{polyak}$ )	0.005	0.005

Appendix

Table 2: Unified FastDSAC algorithmic hyperparameters. These values define our core method and are fixed across all tasks within their respective domains.

Figure 7: Overview of the FastDSAC architecture. (Left) Actor with DEM: The policy dynamically redistributes the exploration budget by modulating the base standard deviation $\hat{\sigma}_{\phi}(s)$ with weights $w_{i}$ (via element-wise multiplication $\odot$ ). These weights are derived from logits $l(s)$ and an environment-conditioned heterogeneity factor $\beta_{e}$ using a normalized Softmax. (Middle) Environment: Massively parallel environments collect experiences into a shared Replay Buffer for high-throughput training. (Right) Continuous Critic: The critic models the value function as a continuous Gaussian distribution to ensure accurate value estimation. It minimizes the KL Divergence between the current and target distributions, avoiding the quantization artifacts of discrete baselines (marked by $\times$ ).

Panel of Figure 8 — Figure 8: Additional Sim-to-Real Results. Time-lapse sequences of zero-shot FastDSAC policies deployed on the Unitree G1. The robot executes backward walking under real-time joystick commands and a “squat-carry-walk-squat” motion-tracking sequence.

Panel of Figure 9 — Figure 9: Performance comparison against concurrent baselines on 61-DoF HumanoidBench tasks. (a) FastSAC-ERA plateaus early, as global entropy thresholds restrict localized variance pruning. (b) FlashSAC struggles to scale to the 61-DoF regime despite extended training budgets, highlighting the necessity of algorithmic exploration mechanisms.

Figure 10: Wall-clock time comparison between FastDSAC and FastTD3. Evaluated on representative tasks from HumanoidBench, IsaacLab, and MuJoCo Playground. Left: absolute per-task total training time (hours). Right: relative wall-clock time normalized against FastTD3, detailing per-task ratios, the geometric-mean ratio, and the overall total-time ratio across all nine tasks.

Benchmark	Task	Steps	FastTD3 (h)	FastDSAC (h)	Rel. Time
HB	Balance Hard	250K	3.01	3.33	1.10 $\times$
HB	Basketball	250K	2.73	3.39	1.24 $\times$
HB	Hurdle	300K	4.08	4.21	1.03 $\times$
HB	Sit Hard	150K	1.78	2.05	1.15 $\times$
HB	Window	250K	2.96	3.44	1.16 $\times$
IL	Allegro Repose	200K	2.17	2.35	1.08 $\times$

Appendix

Table 3: Wall-clock time comparison on representative HumanoidBench, IsaacLab, and MuJoCo Playground tasks. All timings were measured locally on a single RTX 4090 GPU.

Figure 11: Exploration weights ( $w_{i}$ ) during a Basketball episode. High weights (bright) concentrate on the left thumb and wrist, while lower weights (dark) on the legs, torso, and right arm.

Figure 12: Visualization of exploration weights at different temperatures. Lower temperatures ( $\tau=0.5$ ) result in highly concentrated weights on specific active joints, whereas higher temperatures ( $\tau=10.0$ ) lead to a nearly uniform distribution across all joints.

Figure 13: Support sensitivity of C51

+

DEM on Basketball . We vary the C51 support range while keeping the remaining training pipeline unchanged. The resulting performance differences show that discrete critics can be sensitive to support specification on this task, whereas FastDSAC avoids this additional tuning axis.

Figure 14: Ablation study of Layer Normalization. Top row (HumanoidBench,

|\mathcal{A}|=61

): Removing LayerNorm induces seed sensitivity, severe oscillations, and poor sample efficiency. Bottom row (IsaacLab & MuJoCo Playground,

|\mathcal{A}|<30

): FastDSAC remains highly stable and sample-efficient even without LayerNorm, indicating that the core algorithm is robust in lower-dimensional settings.

Figure 15: Ablation on target entropy in Basketball . The standard heuristic ( $\mathcal{H}=-61$ ) forces rapid variance decay, causing high inter-seed variance and premature convergence (orange curve). FastDSAC maintains $\mathcal{H}=0$ to provide sufficient exploration capacity, which is structurally stabilized by DEM. Both curves report results over 3 seeds.

Figure 16: Sensitivity to the DEM temperature

\tau

. Learning curves on Hurdle , Room , and Stair under different fixed temperatures. The default setting

\tau=1

remains a robust choice overall. Room favors

\tau=1

, Stair shows similar performance across

\tau\in\{0.5,1,5\}

, and Hurdle benefits from a larger temperature, with

\tau=10

achieving the best final return.

Panel of Figure 17 — Figure 17: Full results across all benchmarks. Curves show mean return over training steps with shaded regions indicating the min-max range across 3 seeds. (a) HumanoidBench (25 tasks): FastDSAC (blue) is compared against FastTD3 (black), FastSAC Standard (red), and leading baselines including DreamerV3 (orange), TD-MPC2 (pink), SAC (purple), and PPO (green). FastDSAC often outperforms or matches these baselines across locomotion (e.g., Run , Hurdle ), fine manipulation (e.g., Insert ), and complex whole-body coordination (e.g., Basketball , Balance ), combining strong exploration with stable training. (b) IsaacLab and MuJoCo Playground (10 tasks): FastDSAC (blue) is compared against FastTD3 (black), FastSAC Standard (red), and PPO (green). On IsaacLab, FastDSAC performs strongly on locomotion (Velocity Flat/Rough H1/G1) and Repose Cube Allegro, while remaining competitive on Repose Cube Shadow. On MuJoCo Playground, FastDSAC consistently outperforms baselines on G1/T1 Joystick with lower variance and higher asymptotic returns.

Maximum Entropy RL

Maximum entropy reinforcement learning balances reward seeking with stochastic policies that preserve exploration and support diverse behavior.

See all 69 papers

FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion
Jun 30, 2026Guanchen Lu, Yajuan Dun, Yi Zhou +4Distributional RL Reinforcement Learning
Chongqing University, Chongqing 400044, China · Ministry of Industry and Information Technology, Beijing 100804, China · Tsinghua University, Beijing 100084, China +1
FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance
May 29, 2026Sungha Kim, Gawon Lee, Jusuk Lee +3RL Control Maximum Entropy RL
Seoul National University · Georgia Institute of Technology
FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control
Apr 6, 2026Donghu Kim, Youngdo Lee, Minho Park +10Off-Policy RL Robotic RL
Holiday Robotics · KAIST · KRAFTON +5

Jun 30, 2026J/K move · Enter open · S save

FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion

Guanchen Lu, Yajuan Dun, Yi Zhou, Letian Tao+3

Chongqing University, Chongqing 400044, China · Ministry of Industry and Information Technology, Beijing 100804, China · Tsinghua University, Beijing 100084, China+1

Distributional RL Reinforcement Learning RL for Legged Locomotion Sample-Efficient RL

Open paper AskPDF arXiv

Scalable reinforcement learning has popularized high-throughput sampling architectures, which significantly compresses the training time for off-policy methods in robotic locomotion. However, the rapid increase of data volume and update frequency undermines the stability of value-based methods and diminishes the plasticity of policy networks. To address these challenges, this work presents FastDSAC, a fast and high-performance variant of the Distributional Actor-Critic algorithm designed for parallel sampling scenarios. Specifically, we introduce a truncated Gaussian distribution to approximate the learned policy, which effectively excludes out-of-distribution actions that strain target value estimation while keeping necessary stochasticity for exploration. The proposed action constraint functions as an implicit regularization, which counteracts the plasticity loss typically caused by aggressive gradient updates. This preservation of network adaptability enhances sample efficiency, particularly in scenarios with a high update-to-data ratio, and accelerates the early training process. In contrast to prior fast reinforcement learning approaches that rely on discrete value distributions, our method utilizes a continuous Gaussian representation equipped with adaptive variance regulation, which improves value estimation accuracy by sampling confident and informative transitions. Extensive experiments on MuJoCo Playground and HumanoidBench demonstrate that FastDSAC not only stabilizes the overall training process but also achieves superior asymptotic performance and faster convergence compared to state-of-the-art baselines.

Abstract

Maximum Entropy RL

Explore similar work

FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion

FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance

FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control