Organizations: Zhejiang University · Westlake University · Shanghai Jiao Tong University · Shanghai Innovation Institute · Beijing Institute of Technology · Huazhong University of Science and Technology · Northwest Polytechnical University · Xmov · Fudan University
In human-robot collaboration, natural language lets people direct a humanoid's whole-body movements without programming joint trajectories. A common approach generates a kinematic reference from an instruction and relies on a tracking controller to execute it. However, as the controller maintains balance under contact constraints, the executed motion can depart from the reference in ways that change the action it conveys. Our central insight is that semantic alignment must be evaluated directly on the complete motion the robot physically executes. Building on this insight, we introduce SemPhys, a framework that trains motion generators using semantic feedback from simulated executions. A differentiable world model and a fixed tracking controller predict a short segment of a recorded execution. SemPhys inserts this prediction at its original position, scores the complete trajectory against the instruction, and backpropagates through the predicted segment. A paired anchoring loss stabilizes adaptation by keeping each generated reference close to its counterpart from the initial generator. On BONES-SEED, SemPhys largely preserves instruction retrieval accuracy from reference to execution and achieves the highest retrieval accuracy on executed motions among the evaluated methods.
Figures & tables
Figure 2: Overview of SemPhys training via execution-space semantic feedback. (a) Generate & Execute: A DiT generator synthesizes a reference motion from text and an LLM event plan, tracked by a frozen controller in simulation. (b) Surrogate Learning: A learned differentiable world model and the frozen controller predict a short closed-loop window ( H=32 at 50 Hz) from a recorded state. Spliced into the full episode, this window enables a frozen matcher to backpropagate semantic gradients to the generator’s projection heads (orange). Prior anchoring and physical shaping regularize updates; snowflakes denote frozen modules. Icons adapted from Smashicons/Flaticon.
Objective Metrics
User Study
Method
MM ↓
R@1 ↑
R@2 ↑
R@3 ↑
FID ↓
Fail (‰) ↓
Motion ↑
Semantics ↑
TextOp [ 35 ] (CoRL ’26)
1.279 [-1pt] ± 0.0012
17.72 [-1pt] ± 0.19
26.73 [-1pt] ± 0.33
33.37 [-1pt] ± 0.34
0.3404 [-1pt] ± 0.0010
9.53 [-1pt] ± 1.15
1.68 [-1pt] ± 0.40
1.40 [-1pt] ± 0.59
TeXedo [ 3 ] (arXiv ’26)
1.250 [-1pt] ± 0.0012
22.58 [-1pt] ± 0.13
30.27 [-1pt] ± 0.10
36.03 [-1pt] ± 0.10
0.3037 [-1pt] ± 0.0008
0.46 [-1pt] ± 0.08
1.58 [-1pt] ± 0.43
1.40 [-1pt] ± 0.58
MoMask [ 10 ] (CVPR ’24)
1.180 [-1pt] ± 0.0004
36.61 [-1pt] ± 0.11
44.46 [-1pt] ± 0.07
48.99 [-1pt] ± 0.14
0.3521 [-1pt] ± 0.0002
1.81 [-1pt] ± 0.75
1.84 [-1pt] ± 0.46
1.86 [-1pt] ± 0.50
ECHO [ 18 ] (arXiv ’26)
1.177 [-1pt] ± 0.0010
26.36 [-1pt] ± 0.23
39.36 [-1pt] ± 0.21
47.72 [-1pt] ± 0.24
0.2242 [-1pt] ± 0.0013
2.09 [-1pt] ± 0.16
1.64 [-1pt] ± 0.46
1.60 [-1pt] ± 0.63
OMG [ 16 ] (RSS Workshop ’26)
1.060 [-1pt] ± 0.0015
50.78 [-1pt] ± 0.21
64.67 [-1pt] ± 0.22
71.48 [-1pt] ± 0.29
0.1441 [-1pt] ± 0.0013
2.24 [-1pt] ± 0.65
1.98 [-1pt] ± 0.48
1.82 [-1pt] ± 0.52
Table 1: Instruction following on BONES-SEED after execution under a shared G1 controller. MM and R@ k measure text–motion matching distance and retrieval accuracy (%). FID measures feature-distribution distance. Fail is the sustained-failure rate per thousand upright tasks. Objective columns report mean ± SD across three generation seeds.
Figure 3: Semantic fidelity before and after execution. (a) At generation seed 42, SemPhys scores 80.63% R@1 on its references and 79.59% on their executions. Lines connect the reference (open) and executed (filled) scores for each method. (b) Executed R@1 and sustained failure (log scale) under a shared frozen controller and nominal dynamics. Retrieval covers the test split, and failure is measured on upright tasks.
Figure 4: Executed text–motion matching across eight motion groups. SemPhys (green stars) achieves the lowest matching distance in seven groups. Each group compares all eight methods at generation seed 42. Lower values indicate closer text–motion matching.
Figure 5: Retrieval and stability under changes in robot mass. (a) Means and ranges of executed R@1 and sustained failure across three mass scales ( 0.8× , 0.9× , and 1.1× ) on the full test set. (b) The mean R@1 drop for SemPhys is 0.25 percentage points relative to nominal execution. Negative changes on either axis in (b) indicate improvement.
Figure 6: Preserving the low reach and the carrying pose through execution. The instructions are (a) “character picks up big light object with two hands from low height up behind” and (b) “character turns back over right shoulder then walks forward while holding medium light object with two hands”. Translucent blue poses show the generated reference and solid robots show its execution at five matched times. The methods share a controller and a display scale within each example. Each trajectory’s initial heading is aligned for display. The carrying example illustrates whole-body posture.
Figure 7: Supervision on executed motion and paired anchoring improve retrieval. Each bar shows the relative change 100(m/mfull−1) from the full objective. Replacing semantic supervision with GT trajectory reconstruction gives 56.93% executed R@1 versus 79.30% for the full objective, a relative change of −28.2% . No post-training denotes the shared first-stage generator.
Figure 8: Motion context during training. Frames from one G1 simulation illustrate three inputs to the semantic scorer. All rows share the same 32-step differentiable window. Blank regions indicate omitted context. Scores give executed R@1 on the BONES-SEED test split (mean ± SD across three training seeds).
Figure 9: Testing the surrogate gradient by replaying the updated motion. (a) Agreement between the predicted and measured directions of change on pairs above replay tolerance. (b) Fraction of finite trials in which the full simulator replay improves matching distance. Error bars show 95% caption-bootstrap intervals.
H
R@1 (%) ↑
MM ↓
Fail (‰) ↓
32
79.31±0.16
0.8453±0.0015
0.23±0.08
64
80.11±0.32
0.8332±0.0021
0.15±0.15
96
79.81±0.20
0.8320±0.0002
0.31±0.20
Table 2: Sensitivity to gradient window length. Executed BONES-SEED test scores by window length H in control steps (default 32). Mean ± SD across three paired training seeds, with generation seed 42.
Text-to-motion models generate plausible human motion but do not model a robot's dynamics; whole-body tracking controllers execute robot references reliably but cannot replan an infeasible one. Recent language-to-humanoid systems bridge this gap by training. We measure how much of the gap closes with no training at all, by putting the deployment controller itself in the loop. Sample-simulate-select (S3) draws N motions per prompt from a frozen text-to-motion model, retargets each to a Unitree G1 by direction-matching inverse kinematics, rolls all of them out under full rigid-body dynamics with the pretrained SONIC tracking policy, and keeps the candidate the policy executed best. Because the verifier is the deterministic simulator itself, S3 attains the any-of-N ceiling by construction; what we measure is where that ceiling lies and what falls short of it. On 200 stratified HumanML3D test prompts with N=8, upright execution rises from 83.5% to 89.5% and hardware-gate passes from 33 to 85; on the complete test split (4,184 prompts) it rises from 80.5% to 89.5%. A kinematic verifier that predicts falls well (AUROC 0.90) recovers only a quarter of this gain: ranking a prompt's own candidates is harder than classifying the population. What selection cannot fix is one class, prompts that lower the pelvis, which a generator trained on retargeted robot data does execute. We further score the semantic fidelity of the executed motion with the standard text-motion evaluator, with a real-mocap control that attributes the loss to the robot projection, ablate the retargeter against GMR (complementary failures: the any-of-8 ceiling rises to 95.0% over both), and execute all 177 gate-selected clips on the real G1: every one completes standing, with hardware tracking error matching simulation (r=0.94).
Raphael Memmesheimer, Sven Behnke
Autonomous Intelligent Systems, Computer Science Institute VI, Lamarr Institute for Machine Learning and Artificial Intelligence, and Center for Robotics, University of Bonn, Bonn, Germany
While generative models have become effective at producing human-like motions from text, transferring these motions to humanoid robots for physical execution remains challenging. Existing pipelines are often limited by retargeting, where kinematic quality is undermined by physical infeasibility, contact-transition errors, and the high cost of real-world dynamical data. We present a unified latent-driven framework that bridges natural language and whole-body humanoid locomotion through a retarget-free, physics-optimized pipeline. Rather than treating generation and control as separate stages, our key insight is to couple them bidirectionally under physical constraints.We introduce a Physical Plausibility Optimization (PP-Opt) module as the coupling interface. In the forward direction, PP-Opt refines a teacher-student distillation policy with a plausibility-centric reward to suppress artifacts such as floating, skating, and penetration. In the backward direction, it converts reward-optimized simulation rollouts into high-quality explicit motion data, which is used to fine-tune the motion generator toward a more physically plausible latent distribution. This bidirectional design forms a self-improving cycle: the generator learns a physically grounded latent space, while the controller learns to execute latent-conditioned behaviors with dynamical integrity.Extensive experiments on the Unitree G1 humanoid show that our bidirectional optimization improves tracking accuracy and success rates. Across IsaacLab and MuJoCo, the implicit latent-driven pipeline consistently outperforms conventional explicit retargeting baselines in both precision and stability. By coupling diffusion-based motion generation with physical plausibility optimization, our framework provides a practical path toward deployable text-guided humanoid intelligence.
Humanoid robots are expected to execute agile and expressive whole-body motions in real-world settings. Existing text-to-motion generation models are predominantly trained on captured human motion datasets, whose priors assume human biomechanics, actuation, mass distribution, and contact strategies. When such motions are directly retargeted to humanoid robots, the resulting trajectories may satisfy geometric constraints (e.g., joint limits and pose continuity) and appear kinematically reasonable. However, they frequently violate the physical feasibility required for real-world execution. To address these issues, we present PhyGile, a unified framework that closes the loop between robot-native motion generation and General Motion Tracking (GMT). PhyGile performs physics-prefix-guided robot-native motion generation at inference time, directly generating robot-native motions in a 262-dimensional skeletal space with physics-guided prefixes, thereby eliminating inference-time retargeting artifacts and reducing generation-execution discrepancies. Before physics-prefix adaptation, we train the GMT controller with a curriculum-based mixture-of-experts scheme, followed by post-training on unlabeled motion data to improve robustness over large-scale robot motions. During physics-prefix adaptation, the GMT controller is further fine-tuned with generated objectives under physics-derived prefixes, enabling agile and stable execution of complex motions on real robots. Extensive offline and real-robot experiments demonstrate that PhyGile expands the frontier of text-driven humanoid control, enabling stable tracking of agile, highly difficult whole-body motions that go well beyond walking and low-dynamic motions typically achieved by prior methods.
Jiacheng Bao, Haoran Yang, Yucheng Xin +7
Northwestern Polytechnical University · Shanghai AI Laboratory · University of Science and Technology of China +4