SemPhys: Semantic-Physical Co-Learning for Humanoid Motion Generation via Differentiable World Models
Organizations: Zhejiang University · Westlake University · Shanghai Jiao Tong University · Shanghai Innovation Institute ·
Abstract
In human-robot collaboration, natural language lets people direct a humanoid's whole-body movements without programming joint trajectories. A common approach generates a kinematic reference from an instruction and relies on a tracking controller to execute it. However, as the controller maintains balance under contact constraints, the executed motion can depart from the reference in ways that change the action it conveys. Our central insight is that semantic alignment must be evaluated directly on the complete motion the robot physically executes. Building on this insight, we introduce SemPhys, a framework that trains motion generators using semantic feedback from simulated executions. A differentiable world model and a fixed tracking controller predict a short segment of a recorded execution. SemPhys inserts this prediction at its original position, scores the complete trajectory against the instruction, and backpropagates through the predicted segment. A paired anchoring loss stabilizes adaptation by keeping each generated reference close to its counterpart from the initial generator. On BONES-SEED, SemPhys largely preserves instruction retrieval accuracy from reference to execution and achieves the highest retrieval accuracy on executed motions among the evaluated methods.
Figures & tables
| Objective Metrics | User Study | |||||||
|---|---|---|---|---|---|---|---|---|
| Method | MM | R@1 | R@2 | R@3 | FID | Fail (‰) | Motion | Semantics |
| TextOp [ 35 ] (CoRL ’26) | 1.279 [-1pt] 0.0012 | 17.72 [-1pt] 0.19 | 26.73 [-1pt] 0.33 | 33.37 [-1pt] 0.34 | 0.3404 [-1pt] 0.0010 | 9.53 [-1pt] 1.15 | 1.68 [-1pt] 0.40 | 1.40 [-1pt] 0.59 |
| TeXedo [ 3 ] (arXiv ’26) | 1.250 [-1pt] 0.0012 | 22.58 [-1pt] 0.13 | 30.27 [-1pt] 0.10 | 36.03 [-1pt] 0.10 | 0.3037 [-1pt] 0.0008 | 0.46 [-1pt] 0.08 | 1.58 [-1pt] 0.43 | 1.40 [-1pt] 0.58 |
| MoMask [ 10 ] (CVPR ’24) | 1.180 [-1pt] 0.0004 | 36.61 [-1pt] 0.11 | 44.46 [-1pt] 0.07 | 48.99 [-1pt] 0.14 | 0.3521 [-1pt] 0.0002 | 1.81 [-1pt] 0.75 | 1.84 [-1pt] 0.46 | 1.86 [-1pt] 0.50 |
| ECHO [ 18 ] (arXiv ’26) | 1.177 [-1pt] 0.0010 | 26.36 [-1pt] 0.23 | 39.36 [-1pt] 0.21 | 47.72 [-1pt] 0.24 | 0.2242 [-1pt] 0.0013 | 2.09 [-1pt] 0.16 | 1.64 [-1pt] 0.46 | 1.60 [-1pt] 0.63 |
| OMG [ 16 ] (RSS Workshop ’26) | 1.060 [-1pt] 0.0015 | 50.78 [-1pt] 0.21 | 64.67 [-1pt] 0.22 | 71.48 [-1pt] 0.29 | 0.1441 [-1pt] 0.0013 | 2.24 [-1pt] 0.65 | 1.98 [-1pt] 0.48 | 1.82 [-1pt] 0.52 |
| R@1 (%) | MM | Fail (‰) | |
|---|---|---|---|
| 32 | |||
| 64 | |||
| 96 |
Human Motion Generation
Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.
See all 309 papers