Mimic: Ultra-Efficient Humanoid Control with Differentiable Simulation
Organizations: UC Davis
Abstract
Robot policy learning is dominated by zeroth-order methods such as PPO, which treat the simulator as a black box and pay for it with enormous sample counts. First-order policy learning differentiates through the dynamics, but on contact-rich whole-body control it has rarely worked: its gradients explode. A recent world-model method showed that derivative information can make humanoid motion tracking far more efficient, using surrogate gradients of a learned model. We show that exact gradients of the simulator itself do better, once two obstacles are removed. Exact gradients contain rare unbounded samples, because contact dynamics is not differentiable everywhere and its derivative diverges near some events; and they are expensive, because the simulator's adjoint has to replay the contact solve. Mimic bounds these samples with median-relative per-sample gradient clipping at every step of the backward pass, and reuses each costly simulated window by multistep regression onto gradient-derived action targets, which bounds every action target independently of the gradient scale. NablaEngine, a differentiable GPU simulation engine for the Unitree G1, removes the cost: it computes MuJoCo's full contact dynamics with fused per-world kernels and the adjoint of a whole control step as a single megakernel, with analytic gradients that match float64 finite differences, faster than the available MuJoCo Warp adjoint. On whole-body tracking of all 40 LAFAN1 clips with one policy on one GPU, Mimic reaches the score reported by the state-of-the-art world-model method, 0.82, in 20 minutes instead of 1.2 h, and 0.874 in 1.8 h. SHAC, the standard first-order baseline, stops on exploding gradients within minutes. We release NablaEngine and Mimic as open-source infrastructure for first-order humanoid policy learning.
Figures & tables
| Exact gradient (simulator) | Surrogate gradient (world model) | |
|---|---|---|
| Bias | no model error: derivative of the simulated system | derivative of a different, learned system |
| Unbounded derivatives | near non-differentiable events | none (a smooth network) |
| Cost per update | high (simulator adjoint) | low (one network pass) |
| Change | Absolute actions | Residual actions ( Mimic) |
|---|---|---|
| No per-sample clipping | (1 of 3 seeds diverged) | all 3 seeds diverged |
| No multistep regression (one Adam step per window) | ||
| instead of 64 | a | |
| Residual instead of absolute actions | — | |
| No replay starts (continued windows) |
| RTX PRO 6000 | H100 | RTX 3090 | V100 | |
| Forward step | ||||
| NablaEngine | 1.21 | 1.70 | 3.28 | 5.79 |
| MuJoCo Warp | 5.84 | 8.65 | 12.5 | 18.4 |
| Step and its gradient | ||||
| NablaEngine | 1.78 | 2.73 | 5.26 | 10.6 |
| MuJoCo Warp with adjoint pull request | 20.5 | 25.0 | 43.0 | 63.2 |
| Method | 0.5 h | 1 h | 2 h | 3 h | To 0.82 | Falls (/40) |
|---|---|---|---|---|---|---|
| Mimic | 0.824 0.008 | 0.854 0.005 | 0.872 0.001 | 0.875 0.001 | 0.34 h | 0.0 |
| Mimic, absolute actions | 0.821 0.007 | 0.844 0.004 | 0.861 0.001 | 0.864 0.001 | 0.50 h | 0.0 |
| PPO, BeyondMimic configuration a | 0.465 | 0.529 | 0.641 | 0.638 | not reached | 8 |
| PPO, SuperMimic actor and resets a | 0.428 | 0.489 | 0.586 | 0.658 | not reached | 3 |
| SHAC (changed guards) | 0.724 0.019 | 0.790 0.019 | 0.807 0.008 | 0.816 0.005 | not reached | 2.0 |
| SHAC (published guard) | stopped at 0.06 h by its gradient guard (return 0.373 0.093) | not reached | 23.0 | |||
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Engine | RTX PRO 6000 | H100 | RTX 3090 | V100 |
| NablaEngine | 1.21 | 1.70 | 3.28 | 5.79 |
| MuJoCo Warp 3.17.0 | 5.84 | 8.65 | 12.5 | 18.4 |
| Genesis 1.4.5 | 13.45 | 16.36 | 36.73 | 139.11 |
| MJX 3.17 (same model) | 255 | 216 | 510 | 710 |
| MJX 3.17 (Menagerie G1, fewer contacts) | 50.7 | 58.4 | 129.2 | 150.7 |
| Isaac Lab 3.0, Newton (3 box colliders) | 12.9 | n/s | 35.15 | n/s |
| RTX PRO 6000 | H100 | RTX 3090 | V100 | |
| NablaEngine (forward with tape + adjoint) | 1.78 | 2.73 | 5.26 | 10.6 |
| MuJoCo Warp + PR #1535 (adjoint graph) | 20.5 | 25.0 | 43.0 | 63.2 |
| Speed-up |
| Gradient (median relative VJP error) | |||
| Engine | Position | Velocity | Action |
| NablaEngine | |||
| MJX, forward-mode Jacobian | |||
| MuJoCo C finite differences (step ) | |||
| Genesis (velocity only) | — | — | |
| MuJoCo Warp, open pull request | 0.29 | 0.34 | 0.015 |
| G0 | G4 | G5 | G6 | |
| NablaEngine | 8.6e-7 / 4.7e-6 (5.4e-7) | 4.6e-7 / 6.6e-6 (1.8e-6) | 8.4e-7 / 5.5e-6 (6.5e-7) | 9.4e-7 / 6.9e-6 (1.0e-6) |
| MuJoCo Warp | 1.3e-6 / 8.8e-6 (1.1e-6) | 6.3e-7 / 7.2e-6 (1.5e-6) | 1.2e-6 / 6.8e-6 (9.2e-7) | 1.3e-6 / 8.8e-6 (9.9e-7) |
| MuJoCo Warp, open PR | 1.3e-6 / 9.3e-6 (1.2e-6) | 6.4e-7 / 1.0e-5 (3.4e-6) | 1.2e-6 / 8.6e-6 (1.2e-6) | 1.4e-6 / 1.1e-5 (1.4e-6) |
| MJX (float32) | 2.7e-6 / 6.3e-4 (7.2e-5) | 1.3e-6 / 4.2e-4 (1.1e-4) | 2.6e-6 / 5.1e-4 (6.1e-5) | 5.5e-6 / 6.6e-4 (7.3e-5) |
| Genesis (float32) | 4.9e-2 / 4.8e-1 (6.0e-2) | 5.7e-2 / 5.4e-1 (1.4e-1) | 4.9e-2 / 4.8e-1 (6.1e-2) | 1.0e-1 / 5.1e-1 (7.1e-2) |
| Reference: MuJoCo Warp against itself, run to run | ||||
| G0 | G4 | G5 | G6 | |
|---|---|---|---|---|
| NablaEngine, exact | 3.6e-5 / 1.0e-5 / 1.9e-6 | 4.3e-5 / 1.3e-5 / 2.0e-6 | 3.8e-5 / 1.1e-5 / 1.8e-6 | 3.1e-5 / 1.0e-5 / 2.3e-6 |
| NablaEngine, default | same as exact | n/c | same as exact | same as exact |
| MJX, forward-mode Jacobian | 9.6e-5 / 2.1e-5 / 2.9e-6 | 1.2e-4 / 2.3e-5 / 3.0e-6 | 9.4e-5 / 2.0e-5 / 3.1e-6 | 1.0e-4 / 2.5e-5 / 4.8e-6 |
| MuJoCo C, finite differences | 6.7e-4 / 1.5e-4 / 7.8e-6 | 7.1e-4 / 1.8e-4 / 7.2e-6 | 6.6e-4 / 1.5e-4 / 7.9e-6 | 5.5e-4 / 1.3e-4 / 7.8e-6 |
| Genesis (velocity only) | — / 2.3e-1 / — | — / 3.4e-1 / — | — / 2.7e-1 / — | — / 3.4e-1 / — |
| MuJoCo Warp, open PR | 2.9e-1 / 3.4e-1 / 1.5e-2 | 5.0e-1 / 4.8e-1 / 8.9e-2 | 3.1e-1 / 3.5e-1 / 1.8e-2 | 3.5e-1 / 4.0e-1 / 1.9e-2 |
| Item | Setting |
|---|---|
| Environments | 4096 BPTT training windows per update |
| Window length | simulated steps (1.28 s) |
| Discount, TD | , |
| Actor | encoder–decoder MLP 2048–1024–512–256; anchor orientation into the decoder |
| Critic | MLP 256–128, fit on the pre-transition state |
| Actions | residual: offset to the reference joint pose |
| Method | Operator | Training hardware | Seeds n | Budget (h) | Return @0.5 h | Return @1 h | Return @2 h | Return @3 h | Return @6 h | Time to 0.82 (h) | Per-seed time to 0.82 (h) | Final return (h) | Failed clips at end (of 40) |
| Mimic (NablaEngine) | NablaEngine | RTX PRO 6000 (Blackwell) | 3 | 3 | 0.824 0.008 | 0.854 0.005 | 0.872 0.001 | 0.875 0.001 | — | 0.34 | 0.36 0.05 | 0.875 0.001 (3.10 h) | 0.0 0.0 |
| Mimic, absolute actions (NablaEngine) | NablaEngine | RTX PRO 6000 (Blackwell) | 3 | 3 | 0.821 0.007 | 0.844 0.004 | 0.861 0.001 | 0.864 0.001 | — | 0.50 | 0.46 0.09 | 0.865 0.003 (3.14 h) | 0.0 0.0 |
| Mimic, absolute actions (NablaEngine, H100) ‡ | NablaEngine | H100 | 1 | 3 | 0.816 | 0.848 | 0.861 | 0.862 | — | 0.41 | 0.41 | 0.858 (3.10 h) | 0 |
| Mimic, absolute actions (MuJoCo Warp) ‡ | MuJoCo Warp | RTX PRO 6000 (Blackwell) | 1 | 3 | 0.625 | 0.764 | 0.783 | 0.825 | — | 2.49 | 2.49 | 0.826 (3.05 h) | 1 |
| SHAC (Xu et al., 2022; published guard), our scale (NablaEngine) | NablaEngine | RTX PRO 6000 (Blackwell) | 3 | 3 | — | — | — | — | — | not reached (best 0.373) | — | 0.373 0.093 (0.06 h) | 23.0 3.6 |
| SHAC (published guard), our scale (MuJoCo Warp) ‡ | MuJoCo Warp | RTX PRO 6000 (Blackwell) | 1 | 3 | — | — | — | — | — | not reached (best 0.484) | — | 0.484 (0.36 h) | 17 |
| Method | Hardware | n | Time to 0.82 (h) | Policy updates there | Differentiable window steps | Forward-only simulator steps | Total simulator transitions | Per-seed time to 0.82 (h) |
|---|---|---|---|---|---|---|---|---|
| Mimic (NablaEngine) | RTX PRO 6000 (Blackwell) | 3 | 0.34 | 3,235 updates | 8.48e8 | 3.31e7 | 8.81e8 | 0.36 0.05 |
| Mimic, absolute actions (NablaEngine) | RTX PRO 6000 (Blackwell) | 3 | 0.50 | 4,650 updates | 1.22e9 | 4.76e7 | 1.27e9 | 0.46 0.09 |
| Mimic, absolute actions (NablaEngine, H100) ‡ | H100 | 1 | 0.41 | 3,719 updates | 9.75e8 | 3.81e7 | 1.01e9 | 0.41 |
| Mimic, absolute actions (MuJoCo Warp) ‡ | RTX PRO 6000 (Blackwell) | 1 | 2.49 | 4,918 updates | 1.29e9 | 5.04e7 | 1.34e9 | 2.49 |
| SHAC (Xu et al., 2022; published guard), our scale (NablaEngine) | RTX PRO 6000 (Blackwell) | 3 | not reached (best 0.373 at 0.06 h) | — | — | — | — | — |
| SHAC (published guard), our scale (MuJoCo Warp) ‡ | RTX PRO 6000 (Blackwell) | 1 | not reached (best 0.484 at 0.36 h) | — | — | — | — | — |
| Change to the final recipe | Component origin | Seeds | Matched at | Return: control | Return: variant | return | Failed clips (of 40): control | Failed clips (of 40): variant |
|---|---|---|---|---|---|---|---|---|
| No PODS action targets (one Adam step per window) | ours | 3 | 1.64 h (control checkpoints 1.14-1.63 h) | 0.855 0.001 | 0.806 0.011 | -0.049 | 0.1 0.1 | 1.9 0.8 |
| No per-sample gradient clipping a | ours | 2 of 3 (s0 diverged at 0.54 h, update 5,060) | 1.65 h (control checkpoints 1.18-1.66 h) | 0.855 0.001 | 0.562 0.079 (n=2 of 3) | -0.293 | 0.1 0.1 | 10.9 4.7 (n=2 of 3) |
| Residual actions instead of absolute setpoints b | action parameterization | 3 | 1.64 h (control checkpoints 1.14-1.63 h) | 0.855 0.001 | 0.864 0.001 | +0.009 | 0.1 0.1 | 0.0 0.0 |
| Uniform resets instead of return-based reset sampling | inherited: SuperMimic (reset sampling follows BeyondMimic) | 3 | 1.65 h (control checkpoints 1.18-1.66 h) | 0.855 0.001 | 0.861 0.003 | +0.005 | 0.1 0.1 | 0.1 0.1 |
| No replay starts (windows continue from the previous window’s end state) | inherited: SuperMimic (replay-buffer starts) | 3 | 1.62 h (control checkpoints 1.14-1.63 h) | 0.855 0.001 | 0.874 0.002 | +0.019 | 0.1 0.1 | 0.0 0.0 |
| H=32 windows instead of H=64 | ours | 3 | 1.64 h (control checkpoints 1.14-1.63 h) | 0.855 0.001 | 0.818 0.019 | -0.036 | 0.1 0.1 | 0.2 0.2 |
| Change to the final recipe | Component origin | Seeds | Matched at | Return: control | Return: variant | return | Failed clips (of 40): control | Failed clips (of 40): variant |
|---|---|---|---|---|---|---|---|---|
| No PODS action targets (one Adam step per window) | ours | 3 | 1.63 h (control checkpoints 1.14-1.58 h) | 0.866 0.002 | 0.822 0.012 | -0.044 | 0.1 0.2 | 1.3 0.8 |
| No per-sample gradient clipping | ours | 0 of 3 (s0 diverged at 0.59 h, update 5,510; s1 diverged at 0.31 h, update 2,930; s2 diverged at 0.37 h, update 3,500) | — | — | — | — | — | — |
| Uniform resets instead of return-based reset sampling a | inherited: SuperMimic (reset sampling follows BeyondMimic) | 3 | 1.64 h (control checkpoints 1.14-1.58 h) | 0.866 0.002 | 0.811 0.065 | -0.055 | 0.1 0.2 | 2.8 3.0 |
| No replay starts (windows continue from the previous window’s end state) | inherited: SuperMimic (replay-buffer starts) | 3 | 1.64 h (control checkpoints 1.14-1.58 h) | 0.866 0.002 | 0.884 0.006 | +0.018 | 0.1 0.2 | 0.0 0.0 |
| H=32 windows instead of H=64 b | ours | 3 of 4 (s2 collapsed (critic value runaway) at 1.10 h, update 17,080) | 1.65 h (control checkpoints 1.15-1.66 h) | 0.867 0.002 | 0.824 0.010 (n=3 of 4) | -0.042 | 0.1 0.2 | 0.4 0.0 (n=3 of 4) |
| H=32 windows instead of H=64, at matched updates c | ours | 3 of 4 (s2 collapsed (critic value runaway) at 1.10 h, update 17,080) | update 25,583 (control updates 21,000-25,000, 2.18-2.63 h) | 0.873 0.002 | 0.824 0.010 (n=3 of 4) | -0.048 | 0.0 0.0 | 0.4 0.0 (n=3 of 4) |
| Method | Operator | Hardware | Seconds per policy update (median) | Policy updates per hour (wall-clock slope) | Differentiable steps per update | Forward-only steps per update | Simulated steps per second | Engine speed-up vs MuJoCo Warp |
|---|---|---|---|---|---|---|---|---|
| Mimic a | NablaEngine | RTX PRO 6000 (Blackwell) | 0.374 | 9,503 | 262,144 | 10,240 | 729k | 4.4x |
| Mimic, absolute actions b | NablaEngine | RTX PRO 6000 (Blackwell) | 0.373 | 9,383 | 262,144 | 10,240 | 729k | 4.4x |
| Mimic, absolute actions, hoisted encoder (timing run) c | NablaEngine | RTX PRO 6000 (Blackwell) | 0.369 | — | 262,144 | 10,240 | 738k | 4.4x |
| Mimic, absolute actions | MuJoCo Warp | RTX PRO 6000 (Blackwell) | 1.633 | 1,966 | 262,144 | 10,240 | 167k | — |
| Mimic, absolute actions (H100) | NablaEngine | H100 | 0.382 | 9,275 | 262,144 | 10,240 | 712k | 5.9x § |
| Mimic, absolute actions (H100) d | MuJoCo Warp | H100 | 2.258 | — | 262,144 | 10,240 | 121k | — |
Robot Policy Learning
Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.
See all 403 papers