Currently, robots can learn whole-body loco-manipulation through scale: the strongest policies have billions of parameters and still need task-specific fine-tuning. Humans have limited processing capacity, yet learn and act efficiently through a body schema that organizes what they see and how they move. We introduce Body Schema Policy (BSP), an imitation learning policy that builds such a schema into the vision-action interface. Its latent body schema places joint and link tokens on the robot's kinematic tree, where learned screw motions carry features between links. Through body-queried vision, each body part retrieves its own visual evidence, which conditions the denoising of its actions. With this inductive bias, BSP returns an action chunk faster than any policy we compare. Compact, body-specific visual readouts also emerge without spatial supervision: manipulation commands depend strongly on task objects, while locomotion commands respond more broadly to the scene. On nine simulated Unitree G1 whole-body tasks, BSP matches the Ψ0 humanoid foundation model with 75× fewer parameters and nearly doubles Diffusion Policy's success. The same holds on a real G1 and a Fetch mobile manipulator.
Figures & tables
Figure 2: Body Schema Policy architecture. The state st , denoising step k , and current action hypothesis xt(k) initialize joint ( J ), link ( L ), and global ( g ) tokens. Each of the N neural screw blocks transports parent-link features to child links (Eq. 3 ), then updates the corresponding joint tokens. Periodic blocks add body self-attention over links and the global token and body-queried cross-attention, which retrieves image features with body tokens as queries Q and visual tokens as keys and values K , V ; the last such block is drawn. The final schema supplies one condition cn per action coordinate to a shared temporal decoder fϕ , which predicts action noise.
Figure 3: Evaluation tasks. Top left: the two real-world G1 tasks, each in three phases. Purple arrows mark the motion. Bottom left: the three ManiSkill-HAB tasks on the Fetch mobile manipulator, in third-person views. Right: the nine SIMPLE tasks in egocentric views.
Table 1: Nine representative whole-body humanoid tasks on SIMPLE : successes on each of the three evaluation levels (L0/L1/L2), 10 fixed episodes per task per level. Params: the whole model as deployed. Time: ms per action chunk on an RTX 6000 Ada. Policy type: IL, VLA, WAM. Bold = best, underline = second best; tied totals share a mark.
Figure 4: Forward kinematics beyond the training range: one test configuration of the G1 (top) and of Fetch (bottom). Each link is drawn displaced by its pose error and colored by that error on a log scale. Fetch is drawn in its base frame, with each model’s base translation error removed; the color keeps the full error.
Figure 5: Denoising from noise on Handover tasks. (a) Each row is one denoiser’s clean-action estimate after the given number of DDPM iterations, out of 100, drawn in the head-camera view; last column: the demonstration. (b) The joint-command error of the next action at every iteration: the mean absolute difference from the demonstration over the 31 joint commands of the hands, arms, and waist; dots mark the iterations shown in (a). Rodri.: Rodrigues Network.
Table 2: ManiSkill-HAB SetTable subtasks with the Fetch mobile manipulator. Success rate (%), mean ± standard deviation over three training seeds; 1,008 episodes per seed. Params: the whole model as deployed. Policy type: IL, VLA.
Table 3: Real-world G1 tasks: successes in 10 trials per task. Params: the whole model as deployed. Time: ms per action chunk on an RTX 5090. Policy type: IL, VLA.
Latent body schema
Body-queried vision
DINOv2
LPBT
XMBP
HAND
Total
✓
✓
✓
23
28
24
75
✓
✗
✓
18
24
18
60
✓
✗
✗
10
8
13
31
✗
✗
✓
5
0
19
24
✗
✗
✗
6
0
14
20
Table 4: Ablation studies. We study the effects of the latent body schema, body-queried vision, and the DINOv2 encoder on 3 tasks in SIMPLE; successes out of 30 per task, one seed each. ✗ replaces them, in column order, with DP’s U-Net, one pooled vector, and a ResNet-18 trained from scratch.
Figure 6: What three parts read over one LPBT episode (a) and one XMBP episode (b). Sensitivity of BSP ’s right-thumb, waist-yaw and base-velocity commands to the patch tokens the denoiser reads, at 15 frames of each episode; each heat map is scaled to its own maximum.
Vision-based whole-body loco-manipulation on humanoid robots is challenging due to partial observability, contact-rich dynamics, and the difficulty of learning long-horizon behaviors from high-dimensional visual inputs. We present DreamMimic, a framework that distills privileged teacher policies into vision-based humanoid controllers via world-model-assisted distillation. Instead of using a Dreamer-style RSSM for planning, we repurpose it to learn predictive latent dynamics that serve as both a representation space and an action-conditioned multi-step supervision signal, while exposing compact predictive features to the student policy to reduce long-term drift. Beyond standard reconstruction objectives for proprioceptive and visual observations, we add auxiliary prediction heads for privileged state, contact, object state, and reward estimation. These heads provide additional supervision related to agent--object interaction and task progress, encouraging the latent representation to retain signals that are useful for contact-rich loco-manipulation. We further introduce Performance-Conditioned Guidance (PCG), a reward-driven adaptive distillation schedule that computes performance scores for both teacher and student to dynamically balance guidance and exploration. PCG prevents both premature teacher annealing and excessive teacher interference in challenging visual settings. Experiments on OMOMO and BEHAVE show improved tracking-based loco-manipulation performance over strong vision-based baselines, without exposing online privileged interaction states to the student at deployment. Qualitative simulations further examine morphology and simulator changes. These results suggest that world models can provide a useful mechanism for stabilizing visual policy distillation in contact-rich humanoid behaviors.
Whole-body humanoid loco-manipulation requires coordinating the robot's entire kinematic chain. However, most existing systems typically decouple the upper and lower bodies into separate controllers, limiting such coordination and yielding behaviors similar to those of a wheeled dual-arm platform. In this paper, we ask what it takes to build a whole-body native vision-language-action (VLA) model that maps language and pixels directly to all of the humanoid's degrees of freedom. We conduct a systematic empirical study organized as a roadmap of one-variable-at-a-time experiments across three phases: whole-body teleoperation, VLA model design, and heterogeneous co-training. Our study yields several intriguing findings: a joint-based whole-body teleoperation interface outperforms alternatives that only partially expose the humanoid's degrees of freedom; a VLA pretrained on static and wheeled dual-arm platforms transfers surprisingly well to a humanoid's full action space; and co-training with HuMI, the humanoid analog of UMI, extends the policy to new objects and instructions without additional whole-body teleoperation on those targets. Following this roadmap yields OpenHLM, an open-source recipe for whole-body humanoid loco-manipulation. In a challenging long-horizon task that spans a wide vertical range of the humanoid, OpenHLM outperforms two state-of-the-art humanoid VLA baselines (GR00T N1.6 and Ψ0) using less than half the total demonstration time. Our code, training data, and model checkpoints are available at [https://openhlm-project.github.io/].
Yingdong Hu, Haodong Zhu, Boyuan Zheng +6
Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI
A general-purpose vision-language model can understand a task goal without knowing how a particular robot's motion and functional parts produce the intended effect. We introduce KnowBody, a harness that makes these action-relevant body relations explicit, queryable, and revisable while keeping the model weights frozen. Initialized from one off-task trajectory, a partial body model guides action selection and the interpretation of past interactions. New evidence refines the model, and knowledge dependent on revised body estimates is rechecked before reuse. Across 32 fixed-budget trials on four real-robot tasks, initialized KnowBody achieves 75% completion versus 25% for the native harness and requires fewer planner rounds on successful trials in tasks completed by both. With persistent updates enabled, planner rounds decrease by 29-53% from the first to the fifth recorded success.
Zeyu Lou, Yanhong Zeng, Yong Wang +1
Nanjing University · Ant Group · Zhejiang University