Learning whole-body musculoskeletal control from human motion is important for developing human-like robots, assistive devices, and rehabilitation systems. This task presents two main challenges, namely reference trajectories that are kinematically plausible but difficult to reproduce through muscle-driven dynamics, and a highly redundant 700-dimensional action space in which unstructured exploration can easily disrupt the target motion. In this paper, we propose a motion-to-policy pipeline that exploits muscle redundancy in two complementary stages. First, muscle-aware retargeting (MAR) performs bounded adaptation of kinematic trajectories subject to hard semantic, anatomical, and contact constraints. We motivate the constraint bounds by analyzing how reference-trajectory artifacts affect muscle-driven tracking. Second, null-space exploration (NSE) estimates a one-control-frame mapping from muscle excitations to generalized forces and injects bounded, anatomically structured perturbations along its null space into the actions of a value-guided flow policy. On a 24-clip motion-capture benchmark, MAR and NSE together solve every reference, reach the full horizon with half the samples of the baseline pipeline, and lower joint-center error by 44% without late collapse. To the best of our knowledge, this is the first end-to-end pipeline that transforms monocular video into a trained whole-body imitation policy controlling 700 muscles.
Figures & tables
Fig. 1: Pipeline from MoCap or video (GVHMR) through faithful IK and grounding, MAR with bounded edits under muscle-motivated gates, and QFlex imitation with NSE on the behavior actions to a 700-muscle policy. Redundancy is used twice, to pick a realizable reference and to explore without disturbing the task.
Fig. 2: Null directions in the excitation box of three muscles, with an exaggerated margin m (green band). With only ut,k within the margin of its lower bound, the cone Tm(ut) is the half-space dk≥0 above the gray plane ∂Tm through ut . Its intersection with the null space Nε(Bt) is a half-plane. For the sample ξt , Dykstra’s iterates (dashed) alternate between ∂Tm and Nε(Bt) and converge to the boundary direction dt , whereas one projection takes the opposite sample ξt′=−ξt to dt′ inside the half-plane.
ID
Action
Category
S01
S02
S03
A03
Single-leg balance
static
5.20
4.53
6.57
A05
Squats
static
5.28
5.35
7.13
A08
Walking
locomotion
7.04
7.28
6.23
A09
Jogging
locomotion
6.22
6.93
5.41
A13
Side-stepping
locomotion
6.35
6.06
5.91
A14
Jump in place
dynamic
4.21
2.43
4.05
TABLE I: Benchmark clips with categories and durations (s).
Fig. 3: (a) NSE versus pure QFlex on the built-in walking task with the same reference and reward. Lines are 500k-bin means of episode return over 50 M samples averaged over five seeds, and bands are p10–p90 ranges of the per-episode values pooled across seeds. (b) Benchmark learning curves over the first 50 M samples, with the axis compressed between 10 and 40 M. Episode return is normalized per reference and averaged over the 24 references, shown as 1M-bin means with faint 50k-step traces. Triangles mark the mean first full completion, hollow for pure QFlex and filled for NSE. Blue denotes BA and orange MAR, while dashed lines denote pure QFlex and solid lines NSE.
Method
Success
Rollout
1st full
Return
MPJPE
RMSE
qpos MAE
qpos RMSE
(%)
success (%)
(M)
(b/t)
(b/t)
(b/t)
(b/t)
(b/t)
BA + Pure ‡
87.5
76.6
16.1
18039 / †
95.4 / †
129.9 / †
0.217 / †
0.294 / †
BA + NSE ‡
95.8
87.8
13.0
19281 / 18243
74.2 / 78.1
97.0 / 102.0
0.210 / 0.219
0.281 / 0.289
MAR + Pure
100
91.5
11.0
20231 / 19091
56.5 / 62.3
72.6 / 80.4
0.194 / 0.206
0.259 / 0.275
MAR + NSE
100
98.5
8.1
20690 / 19853
53.7 / 58.5
69.3 / 75.5
0.190 / 0.196
0.252 / 0.262
Success is the share of references reaching a full-horizon trajectory, 1st full the samples to the first full-horizon trajectory (M), and b/t the best checkpoint / the mean over the final 5 M samples. Rollout success is the share of 100 rollouts of the best checkpoint per reference that reach the full horizon with actions sampled from the learned source, a0=tanh(μ+σ⊙z) with z∼N(0,I) , and transported through the 20-step flow, with no noise added to the final action. MPJPE and RMSE are joint-center errors (mm), and qpos MAE and RMSE joint-angle errors (rad). † collapsed (trailing return below 50% of best) on 6/24 references including the 3 never solved, so trailing values are undefined. ‡ best-checkpoint values include forced-continuation rollouts on the references never solved.
TABLE II: 2×2 results on the 24-clip benchmark (means over references).
Fig. 4: Video to policy on the tennis clip. Rows show the MAR + NSE rollout overlaid on the video and the rollout alone, sampled about once per second over the 10.38 s clip.
Metric
BA
MAR
Joint-center RMS (mm)
6.55
6.50
Joint-center P95 (mm)
26.20
25.94
Joint-center max (mm)
43.89
40.11
Static-feasible frames (%)
0.34
3.64
Support frames without near-ground endpoint
1030
0
Residual root force, median (N)
327
110
TABLE III: Reference-level comparison. Kinematic errors are relative to the faithful reference, static feasibility uses bounded activations with admissible contact forces, and residual is the minimum norm of the unmet generalized force (body weight 590 N).
ε
0
0.01
0.03
0.1
0.3
1
3
BA
0.34
0.87
1.78
10.4
23.2
59.5
90.8
MAR
3.64
4.38
6.46
22.9
48.2
77.3
95.9
TABLE IV: Static-feasible frames (%) within residual tolerance ε .
Institute for AI Industry Research (AIR), Tsinghua University · The University of Hong Kong (HKU) · Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University +2