Robot policy learning is dominated by zeroth-order methods such as PPO, which treat the simulator as a black box and pay for it with enormous sample counts. First-order policy learning differentiates through the dynamics, but on contact-rich whole-body control it has rarely worked: its gradients explode. A recent world-model method showed that derivative information can make humanoid motion tracking far more efficient, using surrogate gradients of a learned model. We show that exact gradients of the simulator itself do better, once two obstacles are removed. Exact gradients contain rare unbounded samples, because contact dynamics is not differentiable everywhere and its derivative diverges near some events; and they are expensive, because the simulator's adjoint has to replay the contact solve. ∇Mimic bounds these samples with median-relative per-sample gradient clipping at every step of the backward pass, and reuses each costly simulated window by multistep regression onto gradient-derived action targets, which bounds every action target independently of the gradient scale. NablaEngine, a differentiable GPU simulation engine for the Unitree G1, removes the cost: it computes MuJoCo's full contact dynamics with fused per-world kernels and the adjoint of a whole control step as a single megakernel, with analytic gradients that match float64 finite differences, 11.5× faster than the available MuJoCo Warp adjoint. On whole-body tracking of all 40 LAFAN1 clips with one policy on one GPU, ∇Mimic reaches the score reported by the state-of-the-art world-model method, 0.82, in 20 minutes instead of 1.2 h, and 0.874 in 1.8 h. SHAC, the standard first-order baseline, stops on exploding gradients within minutes. We release NablaEngine and ∇Mimic as open-source infrastructure for first-order humanoid policy learning.
Figures & tables
Figure 1: ∇ Mimic trains a high-quality tracking policy in minutes. (a) LAFAN1 (all 40 clips): normalized return against training wall clock (linear) on one RTX PRO 6000, mean ± sd over 3 seeds. ∇ Mimic reaches 0.82, the score of SuperMimic, in 20 min; SuperMimic reports reaching it within 1.2 h on one H100 [ 39 , §4.1] . PPO trained on the same engine stays below it after 6 h. (b) Simulation time per 20 ms control step for 4096 worlds on the same GPU (log scale): forward step, with Isaac Lab on PhysX (its own G1 asset and task defaults) and MJX (the same model), and gradient of one step; multiples relative to NablaEngine.
Exact gradient (simulator)
Surrogate gradient (world model)
Bias
no model error: derivative of the simulated system
derivative of a different, learned system
Unbounded derivatives
near non-differentiable events
none (a smooth network)
Cost per update
high (simulator adjoint)
low (one network pass)
Table 1: Exact and surrogate gradients: bias, unbounded derivatives and cost.
Figure 2: Exploding gradients sit in single samples. Per-sample action-gradient norms of SHAC on exact gradients (one seed, raw, before any clipping). (a) Distribution within one update batch of 131,072 samples, relative to the batch median: an ordinary update and the update in which SHAC’s guard stops training, where one sample lies at 2.2×109 times the median. (b) Median, 99.9th percentile and maximum per update: the bulk stays flat, only the maximum explodes.
Figure 3: The tail of the per-sample action gradient grows with the number of steps it is backpropagated through, while its median stays flat: (a) 99.9th percentile and (b) maximum, each relative to the median. SHAC: raw exact gradients, 3 seeds, all recorded updates. Same policy: a SuperMimic reproduction’s policy, the same objective and 32-step windows, exact (NablaEngine) and world-model gradients, no clipping.
Figure 4: One panel per component of ∇ Mimic: the control (blue) against the recipe with that component changed (gray), LAFAN1, mean ± sd over 3 seeds. (a) Per-sample clipping: without it one seed diverges and the others stay far below. (b) Multistep regression: one Adam step per window learns more slowly. (c) Window length: H=32 plateaus lower. (d) Residual actions learn faster and plateau higher than absolute actions.
Change
Absolute actions
Residual actions ( ∇ Mimic)
No per-sample clipping
−0.293 (1 of 3 seeds diverged)
all 3 seeds diverged
No multistep regression (one Adam step per window)
−0.049
−0.044
H=32 instead of 64
−0.036
−0.042 a
Residual instead of absolute actions
+0.009
—
No replay starts (continued windows)
+0.019
+0.018
Table 2: Component ablations on LAFAN1: one change to the recipe at a time, 3 seeds each, against a 3-seed control at matched wall clock (about 1.6 h on one RTX PRO 6000). Change in normalized return for the absolute-action recipe (control 0.855), on which the ablations were first run ( Fig. 4 ), and for the residual-action recipe of ∇ Mimic (control 0.867). Full tables in Appendix H .
RTX PRO 6000
H100
RTX 3090
V100
Forward step
NablaEngine
1.21
1.70
3.28
5.79
MuJoCo Warp
5.84
8.65
12.5
18.4
Step and its gradient
NablaEngine
1.78
2.73
5.26
10.6
MuJoCo Warp with adjoint pull request
20.5
25.0
43.0
63.2
Table 3: Time per 20 ms control step for 4096 G1 worlds, ms (median over three processes).
Method
0.5 h
1 h
2 h
3 h
To 0.82
Falls (/40)
∇ Mimic
0.824 ± 0.008
0.854 ± 0.005
0.872 ± 0.001
0.875 ± 0.001
0.34 h
0.0
∇ Mimic, absolute actions
0.821 ± 0.007
0.844 ± 0.004
0.861 ± 0.001
0.864 ± 0.001
0.50 h
0.0
PPO, BeyondMimic configuration a
0.465
0.529
0.641
0.638
not reached
8
PPO, SuperMimic actor and resets a
0.428
0.489
0.586
0.658
not reached
3
SHAC (changed guards)
0.724 ± 0.019
0.790 ± 0.019
0.807 ± 0.008
0.816 ± 0.005
not reached
2.0
SHAC (published guard)
stopped at 0.06 h by its gradient guard (return 0.373 ± 0.093)
not reached
23.0
Table 4: LAFAN1, all 40 clips: normalized return at equal training wall clock, time to SuperMimic’s score of 0.82 (first crossing of the mean curve), and clips with a fall at the end of training. Ours: one RTX PRO 6000 per run, mean ± sd over 3 seeds unless noted. All runs except SuperMimic train on NablaEngine.
Figure 5: The engine changes speed, not learning. ∇ Mimic (absolute actions) on NablaEngine (3 seeds, mean ± sd) and on the MuJoCo Warp adjoint with the engine’s gradient semantics (1 seed), on LAFAN1. (a) Per policy update the curves overlap. (b) Per hour of training, NablaEngine is faster.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Engine
RTX PRO 6000
H100
RTX 3090
V100
NablaEngine
1.21
1.70
3.28
5.79
MuJoCo Warp 3.17.0
5.84
8.65
12.5
18.4
Genesis 1.4.5
13.45
16.36
36.73
139.11
MJX 3.17 (same model)
255
216
510
710
MJX 3.17 (Menagerie G1, fewer contacts)
50.7
58.4
129.2
150.7
Isaac Lab 3.0, Newton (3 box colliders)
12.9
n/s
35.15
n/s
Appendix
Table 5: Forward time per control step, 4096 worlds, ms (process medians, held-out states). n/s: not supported, because Isaac Sim requires an RT-core GPU. Speed-ups hold within one GPU.
RTX PRO 6000
H100
RTX 3090
V100
NablaEngine (forward with tape + adjoint)
1.78
2.73
5.26
10.6
MuJoCo Warp + PR #1535 (adjoint graph)
20.5
25.0
43.0
63.2
Speed-up
11.5×
9.2×
8.2×
6.0×
Appendix
Table 6: One control step and its gradient from the input state, 4096 worlds, ms.
Figure 6: Gradient time per control step for 4096 worlds, split into the forward pass with tape (dark) and the backward pass (light), for NablaEngine and MuJoCo Warp with the open pull request [ 13 ] . The two parts are timed separately, so their sum can differ from the single-call times in Table 6 .
Gradient (median relative VJP error)
Engine
Position
Velocity
Action
NablaEngine
3.6×10−5
1.0×10−5
1.9×10−6
MJX, forward-mode Jacobian
9.6×10−5
2.1×10−5
2.9×10−6
MuJoCo C finite differences (step 10−6 )
6.7×10−4
1.5×10−4
7.8×10−6
Genesis (velocity only)
—
2.3×10−1
—
MuJoCo Warp, open pull request
0.29
0.34
0.015
Appendix
Table 7: Accuracy on held-out states against MuJoCo C 3.17 in float64. Forward: median relative qvel error. Gradient: median relative VJP error against float64 central finite differences, for position / velocity / action. The truth itself is known to about 10−7 .
G0
G4
G5
G6
NablaEngine
8.6e-7 / 4.7e-6 (5.4e-7)
4.6e-7 / 6.6e-6 (1.8e-6)
8.4e-7 / 5.5e-6 (6.5e-7)
9.4e-7 / 6.9e-6 (1.0e-6)
MuJoCo Warp
1.3e-6 / 8.8e-6 (1.1e-6)
6.3e-7 / 7.2e-6 (1.5e-6)
1.2e-6 / 6.8e-6 (9.2e-7)
1.3e-6 / 8.8e-6 (9.9e-7)
MuJoCo Warp, open PR
1.3e-6 / 9.3e-6 (1.2e-6)
6.4e-7 / 1.0e-5 (3.4e-6)
1.2e-6 / 8.6e-6 (1.2e-6)
1.4e-6 / 1.1e-5 (1.4e-6)
MJX (float32)
2.7e-6 / 6.3e-4 (7.2e-5)
1.3e-6 / 4.2e-4 (1.1e-4)
2.6e-6 / 5.1e-4 (6.1e-5)
5.5e-6 / 6.6e-4 (7.3e-5)
Genesis (float32)
4.9e-2 / 4.8e-1 (6.0e-2)
5.7e-2 / 5.4e-1 (1.4e-1)
4.9e-2 / 4.8e-1 (6.1e-2)
1.0e-1 / 5.1e-1 (7.1e-2)
Reference: MuJoCo Warp against itself, run to run
Appendix
Table 8: Forward error per group, 10 solver iterations: median / 99th percentile of the relative qvel error, and in parentheses the 99th percentile of the qpos error (rad). States per group: G0 13,480, G4 476, G5 5,645, G6 5,668. MuJoCo C in float64 with 10 iterations has median 0 and 99th percentile at most 1.9×10−15 in every group.
G0
G4
G5
G6
NablaEngine, exact
3.6e-5 / 1.0e-5 / 1.9e-6
4.3e-5 / 1.3e-5 / 2.0e-6
3.8e-5 / 1.1e-5 / 1.8e-6
3.1e-5 / 1.0e-5 / 2.3e-6
NablaEngine, default
same as exact
n/c
same as exact
same as exact
MJX, forward-mode Jacobian
9.6e-5 / 2.1e-5 / 2.9e-6
1.2e-4 / 2.3e-5 / 3.0e-6
9.4e-5 / 2.0e-5 / 3.1e-6
1.0e-4 / 2.5e-5 / 4.8e-6
MuJoCo C, finite differences
6.7e-4 / 1.5e-4 / 7.8e-6
7.1e-4 / 1.8e-4 / 7.2e-6
6.6e-4 / 1.5e-4 / 7.9e-6
5.5e-4 / 1.3e-4 / 7.8e-6
Genesis (velocity only)
— / 2.3e-1 / —
— / 3.4e-1 / —
— / 2.7e-1 / —
— / 3.4e-1 / —
MuJoCo Warp, open PR
2.9e-1 / 3.4e-1 / 1.5e-2
5.0e-1 / 4.8e-1 / 8.9e-2
3.1e-1 / 3.5e-1 / 1.8e-2
3.5e-1 / 4.0e-1 / 1.9e-2
Appendix
Table 9: Gradient error per group and input block, 10 solver iterations, 256 states per group: median floored relative VJP error for position / velocity / action. Exact and default denote NablaEngine’s gradient modes. “n/c”: not comparable (the truth is the exact derivative and has no counterpart of the default mode on clamped actuators); “—”: not supported.
Item
Setting
Environments
4096 BPTT training windows per update
Window length
H=64 simulated steps (1.28 s)
Discount, TD
γ=0.99 , λ=0.95
Actor
encoder–decoder MLP 2048–1024–512–256; anchor orientation into the decoder
Critic
MLP 256–128, fit on the pre-transition state
Actions
residual: offset to the reference joint pose
Appendix
Table 10: ∇ Mimic hyperparameters.
Figure 7: All LAFAN1 runs: full-clip normalized return against training wall clock. Lines are seed means with a ±1 sd band when at least two seeds are evaluated. Every run trains on NablaEngine on one RTX PRO 6000. SuperMimic and PPO-Sim are the numbers stated in Liao et al. [39, §4.1] .
Method
Operator
Training hardware
Seeds n
Budget (h)
Return @0.5 h
Return @1 h
Return @2 h
Return @3 h
Return @6 h
Time to 0.82 (h)
Per-seed time to 0.82 (h)
Final return (h)
Failed clips at end (of 40)
∇ Mimic (NablaEngine)
NablaEngine
RTX PRO 6000 (Blackwell)
3
3
0.824 ± 0.008
0.854 ± 0.005
0.872 ± 0.001
0.875 ± 0.001
—
0.34
0.36 ± 0.05
0.875 ± 0.001 (3.10 h)
0.0 ± 0.0
∇ Mimic, absolute actions (NablaEngine)
NablaEngine
RTX PRO 6000 (Blackwell)
3
3
0.821 ± 0.007
0.844 ± 0.004
0.861 ± 0.001
0.864 ± 0.001
—
0.50
0.46 ± 0.09
0.865 ± 0.003 (3.14 h)
0.0 ± 0.0
∇ Mimic, absolute actions (NablaEngine, H100) ‡
NablaEngine
H100
1
3
0.816
0.848
0.861
0.862
—
0.41
0.41
0.858 (3.10 h)
0
∇ Mimic, absolute actions (MuJoCo Warp) ‡
MuJoCo Warp
RTX PRO 6000 (Blackwell)
1
3
0.625
0.764
0.783
0.825
—
2.49
2.49
0.826 (3.05 h)
1
SHAC (Xu et al., 2022; published guard), our scale (NablaEngine)
NablaEngine
RTX PRO 6000 (Blackwell)
3
3
—
—
—
—
—
not reached (best 0.373)
—
0.373 ± 0.093 (0.06 h)
23.0 ± 3.6
SHAC (published guard), our scale (MuJoCo Warp) ‡
MuJoCo Warp
RTX PRO 6000 (Blackwell)
1
3
—
—
—
—
—
not reached (best 0.484)
—
0.484 (0.36 h)
17
Appendix
Table 11: LAFAN1: full-clip normalized return at equal training wall clock
Method
Hardware
n
Time to 0.82 (h)
Policy updates there
Differentiable window steps
Forward-only simulator steps
Total simulator transitions
Per-seed time to 0.82 (h)
∇ Mimic (NablaEngine)
RTX PRO 6000 (Blackwell)
3
0.34
3,235 updates
8.48e8
3.31e7
8.81e8
0.36 ± 0.05
∇ Mimic, absolute actions (NablaEngine)
RTX PRO 6000 (Blackwell)
3
0.50
4,650 updates
1.22e9
4.76e7
1.27e9
0.46 ± 0.09
∇ Mimic, absolute actions (NablaEngine, H100) ‡
H100
1
0.41
3,719 updates
9.75e8
3.81e7
1.01e9
0.41
∇ Mimic, absolute actions (MuJoCo Warp) ‡
RTX PRO 6000 (Blackwell)
1
2.49
4,918 updates
1.29e9
5.04e7
1.34e9
2.49
SHAC (Xu et al., 2022; published guard), our scale (NablaEngine)
RTX PRO 6000 (Blackwell)
3
not reached (best 0.373 at 0.06 h)
—
—
—
—
—
SHAC (published guard), our scale (MuJoCo Warp) ‡
RTX PRO 6000 (Blackwell)
1
not reached (best 0.484 at 0.36 h)
—
—
—
—
—
Appendix
Table 12: Cost to reach SuperMimic’s LAFAN1 score of 0.82
Change to the final recipe
Component origin
Seeds
Matched at
Return: control
Return: variant
Δ return
Failed clips (of 40): control
Failed clips (of 40): variant
No PODS action targets (one Adam step per window)
ours
3
1.64 h (control checkpoints 1.14-1.63 h)
0.855 ± 0.001
0.806 ± 0.011
-0.049
0.1 ± 0.1
1.9 ± 0.8
No per-sample gradient clipping a
ours
2 of 3 (s0 diverged at 0.54 h, update 5,060)
1.65 h (control checkpoints 1.18-1.66 h)
0.855 ± 0.001
0.562 ± 0.079 (n=2 of 3)
-0.293
0.1 ± 0.1
10.9 ± 4.7 (n=2 of 3)
Residual actions instead of absolute setpoints b
action parameterization
3
1.64 h (control checkpoints 1.14-1.63 h)
0.855 ± 0.001
0.864 ± 0.001
+0.009
0.1 ± 0.1
0.0 ± 0.0
Uniform resets instead of return-based reset sampling
Tracking large human-motion datasets has become the dominant route to general, expressive humanoid whole-body control, but scaling across many motions is expensive. Current pipelines use model-free reinforcement learning, either merging per-motion specialists through lossy distillation, or brute-forcing a single policy with cluster-scale compute. SuperMimic amortizes learning across all motions with one differentiable world model of their shared dynamics, fit from logged transitions of a standard rigid-body simulator. The world model and the tracking policy train together in one loop, and backpropagation-through-time over short imagined rollouts gives the policy analytic gradients. Fast imagined rollouts and analytic gradients make policy learning efficient. In 3 hours on a single GPU, SuperMimic surpasses the return of a state-of-the-art motion tracker baseline trained for 7 days on 128 GPUs. It reduces GPU-hours by more than 1000× and fails on only 25 of 129,514 benchmark motions, versus 179 failures for the baseline. After 9.8 hours on two GPUs, SuperMimic achieves zero failures. From 1 to 264 hours of motion, the world model's prediction error stays stable, and GPU-hours to a target return grow slowly. Our policy transfers to a physical Unitree G1 across diverse and acrobatic motions. SuperMimic removes compute as the bottleneck to scaling humanoid whole-body control, making it practical to measure motion-tracking scaling laws.
Differentiable simulation provides analytic gradients of robot dynamics, enabling fast and sample-efficient first-order policy optimization. However, obtaining smooth and informative gradients through rigid-body contact typically requires softened contact models, often at the expense of physical fidelity and thereby limiting learned policies largely to simulation. This trade-off becomes particularly consequential for dynamic humanoid motions, where accurate contact dynamics are critical for transferring policies to the real world. Increasing contact stiffness in rigid-body simulation improves the fidelity of interactions, but also makes the dynamics increasingly sensitive to small state perturbations, producing high-variance gradients that can destabilize first-order policy learning. To address this, we propose Bundled Contact Gradients (BCG), a contact-local randomized smoothing framework for differentiable policy learning. When stiff contact is detected, our method evaluates a local bundle of randomized perturbation rollouts around the stiff contact configuration and aggregates their gradient signal thereby reducing gradient variance. We demonstrate the effectiveness of our method by successfully training and transferring dynamic motions zero-shot onto a real-world Unitree G1 humanoid platform. Videos and supplementary information can be found at https://bundledcontactgradients.github.io/
Dyuman Aditya, Jin Cheng, Clemens Schwarke +4
University of Southern California, LA, USA. · ETH Zurich, Switzerland. · NVIDIA. +1
Whole-body motion tracking lets a humanoid robot follow reference motions, but training a tracking policy for a library of motions has usually taken hours of GPU time. SuperMimic showed that the policy can instead be trained by backpropagating through a world model learned from parallel simulation, and InstantTrack added Muon updates, a residual policy around the reference and an averaged deployed policy. We build on both and present BlinkMimic, a recipe that makes this training faster. On a simulated Unitree G1, BlinkMimic trains one policy for all 40 LAFAN1 clips in 3.4 minutes on a single RTX 4090, measured until three consecutive checkpoints reach a normalized return of at least 0.82 with no failed clip (mean of three seeds). We split training time into the cost of an iteration, the iterations needed to fit the motions, and the iterations needed to remove the last failures, and assign each component, inherited or new, to one of these terms. The new components are a step size that decreases once training has progressed, resets directed by failures of the deployed policy under deployment conditions, and GPU graph capture of the whole iteration. A leave-one-out ablation with three seeds per component shows which term each component affects, and further measurements of the gradient, the update rule and the failure cases explain these effects. Code and per-seed results are available at https://github.com/fuyichen0725/blinkmimic.