From Festival Video to the Stage: A Sim-to-Real Ladder for Long-Horizon Expressive Whole-Body Dance on a Commercial Humanoid
Organizations: Research & Development Division, Arithmer Inc., Tokyo, Japan
Abstract
We present a pipeline for transferring Awa Odori, a traditional Japanese festival dance, from monocular video to a commercial Unitree G1 EDU humanoid. The pipeline combines human mesh recovery, kinematic retargeting, reinforcement learning for motion tracking in simulation, and onboard deployment via ONNX. Training ran on a single RTX 5070 GPU. An eleven-version deployment ladder evaluated on hardware examines three interventions. We detect and repair silent corruption of deployment artifacts, model state-estimator drift during training, and edit the reference motion to support stable execution and a controlled ending. The final policies achieved three consecutive rope-free completions of a 43-second routine, a 46-second performance with automatic handoff, and a continuous 9.7-minute rope-free performance composed from the captured motion. In a live handover, a single remote request let the robot start from and return to the manufacturer's walking controller on stage, with a measured torque-free window of 45-140 ms. We report the training dynamics and hardware outcomes and discuss failure boundaries involving compliant flooring and occlusion in monocular capture.
Figures & tables
| Stage | Purpose | Key tools | Output |
|---|---|---|---|
| A | Monocular 3D human-motion capture | PromptHMR [ 34 ] , SMPL-X [ 35 ] (video2robot pipeline) | Per-frame SMPL-X parameters |
| B | Retargeting to robot skeleton | GMR [ 19 , 36 ] | Robot reference motion, 29 DoF + root, 24 fps |
| C | Editable representation and resampling | 36-column per-frame CSV; csv_to_npz (Isaac Lab) | 50 Hz .npz (2,148/2,298 frames) |
| D | Whole-body tracking policy by RL | BeyondMimic whole_body_tracking [ 11 ] ; Isaac Sim 5.1 / Isaac Lab 2.3 [ 37 ] ; rsl_rl PPO [ 38 , 3 ] | Per-motion policy checkpoint |
| E | Deployable artifact | ONNX export with 11 configuration metadata keys; integrity validation and repair (Sec. 4.1 ) | Validated .onnx |
| F | Onboard runtime | RoboJuDo 1.5.0 LocoMimicPipeline [ 39 ] ; unitree_cpp ; Jetson | Live operation: walking dance, gamepad-switched |
| Ver. | Single change introduced | Sim (final) | Hardware outcome |
|---|---|---|---|
| v1 | Baseline training on the raw retargeted motion ( 30k iters) | — | Early falls (order of ten seconds); pipeline confirmed end-to-end |
| v2 | Corrected domain randomization (an over-randomized intermediate had regressed—a useful negative result) | reward 40.3 | Completed twice on a foam mat with hand-steadying; not free-standing |
| v3 | Estimator-drift-aware observation noise (Sec. 4.2 ) | reward 35.4; anchor err 0.52–0.62 m (by design) | 22 s rope-free on hard floor; visibly reduced drift-chasing; falls at single-support sections |
| v4 | Amplitude curriculum, “grounded” variant (Sec. 4.3 a) | reward | Completed but visibly unstable |
| v5 | Amplitude curriculum, “in-place” variant (Sec. 4.3 a) | reward 44.05; ep. len. 500/500; timeout 99.8% | Three consecutive full 43 s rope-free completions on hard floor; fell only at the abrupt original ending (caught by hand) |
| v6 | Engineered ending appended (Sec. 4.3 b); motion now 2,298 frames / 46 s | reward 43.6; timeout 99.6% | Full completion including graceful ending and automatic handoff, hands-free, hard floor. Foam mat still causes falls (OOD ground compliance). With the dexterous hands mounted (not modeled in training): completes with visibly marginal stability |
| Run (2026) | Version | Iterations | Reward | Ep. len. | Timeout | Anchor err. [m] | Joint vel. err. [rad/s] |
|---|---|---|---|---|---|---|---|
| 07-02 18-19 | v1 (base) | 0–29,999 | 38.59 | 493.2 | 98.3% | 0.147 | 6.15 |
| 07-03 18-05 | v1 cont. | 29,999–59,998 | 38.16 | 493.9 | 98.4% | 0.153 | 6.23 |
| 07-06 17-03 | v2 (discarded) | 59,998–74,997 | 36.96 | 493.9 | 98.3% | 0.162 | 6.52 |
| 07-08 12-35 | v2 | 59,998–74,997 | 40.04 | 494.0 | 98.5% | 0.132 | 5.68 |
| 07-09 17-48 | v3 (drift noise) | 74,997–84,996 | 34.87 | 487.8 | 97.6% | 0.534 | 6.29 |
| 07-10 18-40 | v4 | 84,996–90,995 | 41.71 | 497.3 | 99.6% | 0.120 | 4.69 |
| Observation | Measured behaviour |
|---|---|
| Entry to user control | Accepted from FSM 1 only; sent from 501 or 4 (even pre-armed) it returns success and does nothing |
| Damp transition | FSM reports 1 after 120–140 ms and “settled” after 180–210 ms (from ready-stand); 20–140 ms from balance stand |
| Engagement latency | External gains visible on the actuator echo ms after the switch, arm motion after ms |
| Return to internal control | Mode walkrun lands in FSM 802; last returns to the pre-entry state (damp) and is unusable; a subsequent SetFsmId(501) restores balance stand |
| Service release path | After ReleaseMode no external stream drives the motors; SelectMode returns error 7002 until a power cycle |
| Knee sag in the window | to on the loaded knee, recovered by the stiff catch |
| Metric | v1 | v2 | v3 | v5 | v6 | v8 | v10 | v11 |
|---|---|---|---|---|---|---|---|---|
| Mean reward (final block) | 38.59 | 40.04 | 34.87 | 43.79 | 44.10 | 44.57 | 44.57 | 43.78 |
| Mean episode length / 500 | 493.2 | 494.0 | 487.8 | 497.8 | 497.3 | 496.5 | 495.8 | 492.8 |
| Timeout (success) fraction (%) | 98.3 | 98.5 | 97.6 | 99.7 | 99.7 | 99.7 | 99.7 | 99.7 |
| Joint-velocity tracking error (rad/s) | 6.15 | 5.68 | 6.29 | 3.84 | 3.69 | 3.47 | 3.43 | 4.16 |
| Joint-position tracking error (rad) | 0.58 | 0.55 | 0.61 | 0.56 | 0.54 | 0.53 | 0.53 | 0.56 |
| Anchor position error (m) | 0.147 | 0.132 | 0.534 | 0.108 | 0.108 | 0.106 | 0.101 | 0.129 |
| Ver. | Trials | Full | Partial | Falls | Sim succ. |
|---|---|---|---|---|---|
| v1 | 3 | 0 | 0 | 3 | 98.3% |
| v2 | 2 | 2 | 0 | 0 | 98.5% |
| v3 | 1 | 0 | 1 | 0 | 97.6% |
| v4 | 1 | 1 | 0 | 0 | 99.6% |
| v5 | 3 | 3 | 0 | 0 | 99.7% |
| v6 | 1 | 1 | 0 | 0 | 99.7% |
| Design | Outcome |
|---|---|
| Hold at walking ankle gains | collapse (ankle 80/20 below critical stiffness) |
| Stiff catch + 1 s pose ramp (v5) | engaged 103 ms, 3 ∘ sag; forward lurch m |
| 2 s ramp + 2 s blend (v6) | tilt watchdog trip at pitch, 1.9 s into the ramp |
| Statue + 3 s crossfade (final) | engaged 45 / 121 / 141 ms; sag / / ; no lurch |
| Exit: walkrun SetFsmId(501) | factory balance stand restored, joystick-drivable |
| Remote trigger ( /awa/dance , /awa/finish ) | full cycle; refusals only when FSM |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
|---|---|
| Parallel environments | 2048 |
| Physics / control rate | 200 Hz / 50 Hz (decimation 4) |
| Episode length | 500 control steps (10 s) |
| Actor MLP | , ELU |
| Critic input | 286-dim privileged |
| Observation normalisation | empirical running mean/std |
| Term | Weight | Kernel / params |
|---|---|---|
| anchor_pos | 0.500 | std=0.3 |
| anchor_ori | 0.500 | std=0.4 |
| body_pos | 1.000 | std=0.3 |
| body_ori | 1.000 | std=0.4 |
| body_lin_vel | 1.000 | std=1.0 |
| body_ang_vel | 1.000 | std=3.14 |
| Quantity | Range / rule |
|---|---|
| Ground friction / restitution | 0.3–1.2 / 0–0.5 (per episode) |
| Joint default-position offset | rad |
| Base centre-of-mass offset | m in , , (startup) |
| Per-link mass scale | – (startup) |
| Actuator stiffness / damping scale | – (startup) |
| External push interval | every 3–6 s |
| Ver. | Run (2026) | Parent ckpt | Reference motion | Configuration change vs. parent |
|---|---|---|---|---|
| v1 | 07-02 18:19 | — (scratch) | awa_odori (2,148 fr) | friction, restitution, joint-offset , CoM , pushes every 0.5–2 s; no actuator-gain or link-mass randomisation |
| v1 ′ | 07-03 18:05 | 29,999 | same | actuator-gain scale 0.9–1.1, link-mass scale 0.9–1.1 (continued to 60k) |
| v2 † | 07-06 17:03 | 59,998 | same | joint-offset widened to , CoM x .. (over-randomised; regressed, discarded) |
| v2 | 07-08 12:35 | 59,998 | same | corrected DR: gains 0.8–1.2, mass 0.85–1.15, joint-offset , CoM (x,y,z), pushes every 3–6 s |
| v3 | 07-09 17:48 | 74,997 (v2) | same | per-episode constant bias on anchor pos ( ), anchor ori ( ), base lin. vel. ( ) ( NoiseModelWithAdditiveBias ) |
| v4 | 07-10 18:40 | 84,996 (v3) | awa_odori_ground | bias model removed (plain uniform noise); grounded amplitude curriculum |