DreamMimic: Learning Visuomotor Whole-Body Loco-Manipulation via World Model

Jie Yin; Xingyu Lai

cs.ROAug 23, 2026

DreamMimic: Learning Visuomotor Whole-Body Loco-Manipulation via World Model

Authors: Jie Yin, Xingyu Lai

Organizations: Independent · Tsinghua University

Abstract

Vision-based whole-body loco-manipulation on humanoid robots is challenging due to partial observability, contact-rich dynamics, and the difficulty of learning long-horizon behaviors from high-dimensional visual inputs. We present DreamMimic, a framework that distills privileged teacher policies into vision-based humanoid controllers via world-model-assisted distillation. Instead of using a Dreamer-style RSSM for planning, we repurpose it to learn predictive latent dynamics that serve as both a representation space and an action-conditioned multi-step supervision signal, while exposing compact predictive features to the student policy to reduce long-term drift. Beyond standard reconstruction objectives for proprioceptive and visual observations, we add auxiliary prediction heads for privileged state, contact, object state, and reward estimation. These heads provide additional supervision related to agent--object interaction and task progress, encouraging the latent representation to retain signals that are useful for contact-rich loco-manipulation. We further introduce Performance-Conditioned Guidance (PCG), a reward-driven adaptive distillation schedule that computes performance scores for both teacher and student to dynamically balance guidance and exploration. PCG prevents both premature teacher annealing and excessive teacher interference in challenging visual settings. Experiments on OMOMO and BEHAVE show improved tracking-based loco-manipulation performance over strong vision-based baselines, without exposing online privileged interaction states to the student at deployment. Qualitative simulations further examine morphology and simulator changes. These results suggest that world models can provide a useful mechanism for stabilizing visual policy distillation in contact-rich humanoid behaviors.

Figures & tables

Fig. 1 : Overview of the DreamMimic framework: Multiple specialist teachers trained with privileged simulation observations are consolidated into a unified privileged teacher, which guides a vision-based student via Performance-Conditioned Guidance (PCG). PCG adaptively balances teacher supervision and student exploration according to their relative performance. The student operates on non-privileged proprioception, a compact goal condition, and world-model features inferred from depth and segmentation. The goal condition jointly describes the target object pose and short-horizon robot trajectory cues, while the world model supplies predictive interaction features and action-conditioned multi-step latent supervision. This design enables the policy to exploit both task-level goals and inferred interaction cues while reducing long-horizon drift.

Fig. 2 : DreamMimic supports vision-based humanoid loco-manipulation through world-model-assisted policy distillation. Distilling privileged specialist teachers into a visual student enables whole-body interaction under partial observability (top left) and diverse behaviors (top right). The same framework transfers across morphologies, including SMPL-X and Unitree G1 (bottom left), and sustains long-horizon contact-rich loco-manipulation (bottom right).

Fig. 3 : Overview of the world model framework. A Dreamer-style world model jointly learns predictive latent dynamics from visual and proprioceptive observations and provides action-conditioned multi-step latent targets to stabilize policy learning.

Method	Distill.	SMPL-X on OMOMO				SMPL-X on OMOMO (w x5)
Method	Distill.	Succ. ↑	Time ↑	$E_{r}$ ↓	$E_{o}$ ↓	Succ. ↑	Time ↑	$E_{r}$ ↓	$E_{o}$ ↓
InterMimic [ 35 ]	Teacher	100.0	190.51	9.4	6.8	68.6	157.96	11.7	15.0
ResNet-18 [ 8 ] + policy	RL	0.0	18.53	19.8	-	0.0	18.51	20.0	-
Dreamer (single-stage)	RL	0.0	28.73	25.6	-	0.0	28.22	25.9	-
ResNet-18 [ 8 ] + policy	DAgger	66.7	163.22	8.9	10.1	25.5	105.98	11.3	16.0
ResNet-18 [ 8 ] + policy	DAgger+RL	72.6	169.49	7.8	9.7	29.4	102.92	10.5	15.7

TABLE I : Comparison of visuomotor policies on SMPL-X in OMOMO dataset.

Method	Distill.	SMPL-X on BEHAVE				SMPL-X on BEHAVE (w x2)
Method	Distill.	Succ. ↑	Time ↑	$E_{r}$ ↓	$E_{o}$ ↓	Succ. ↑	Time ↑	$E_{r}$ ↓	$E_{o}$ ↓
InterMimic [ 35 ]	Teacher	100.0	304.45	9.7	10.4	90.1	303.82	10.3	12.3
ResNet-18 [ 8 ] + policy	RL	0.0	30.52	20.9	18.6	0.0	29.11	23.5	19.1
Dreamer (single-stage)	RL	0.0	35.22	18.1	17.8	0.0	32.63	19.1	18.3
ResNet-18 [ 8 ] + policy	DAgger+RL(Naive Annealing)	26.6	130.42	17.7	16.2	18.2	102.33	18.8	18.6
DreamMimic	DAgger+RL(Naive Annealing)	72.7	275.21	15.2	14.9	63.6	266.86	16.8	15.4

TABLE II : Comparison of visuomotor policies on SMPL-X in BEHAVE dataset.

Fig. 4 : Qualitative comparison on contact-rich loco-manipulation. DreamMimic maintains stable interaction and balance over long horizons, while the baseline often fails to sustain contact and collapses.

Fig. 5 : Qualitative comparison on the Unitree G1 humanoid in Isaac Gym. DreamMimic generates stable pushing behavior with consistent balance and object interaction, whereas the Dreamer baseline becomes unstable during contact and falls.

Fig. 6 : Qualitative test in Isaac Sim. DreamMimic performs stable whole-body interactions with a wooden chair and a large table.

Explore similar work

Sep 16, 2026

ViLoMan: Learning Visual-Proprioceptive Whole-Body Loco-Manipulation Skills for Humanoid Robots

Humanoid loco-manipulation requires adaptive whole-body coordination to seamlessly integrate locomotion and physical interaction. Despite recent advances, learning autonomous loco-manipulation remains challenging due to the scarcity of diverse, physically executable robot-object interaction data and the difficulty of learning unified whole-body control directly from onboard observations. We present ViLoMan, a scalable framework for autonomous humanoid loco-manipulation. ViLoMan first transforms partial kinematic demonstrations of human-object interactions into complete, physically executable robot trajectories. It then leverages these trajectories within a teacher-student distillation framework to learn a unified policy that maps egocentric depth observations and proprioceptive measurements directly to joint-level whole-body actions. During deployment, the policy requires neither reference motions nor intermediate commands. We evaluate ViLoMan on door-closing tasks across diverse door configurations and robot initial conditions in both simulation and the real world. Experimental results demonstrate that a single policy enables a Unitree G1 humanoid to complete the full task using only onboard depth sensing and proprioception, while generalizing robustly across task variations and transferring effectively from simulation to reality. Project page: viloman-anonymous.pages.dev.

Zejie Tian, Ruibing Hou, Bingpeng Ma +2

State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, China.

Humanoid Loco-Manipulation Humanoid Robot Control Visuomotor Policy Learning Whole-Body Robotic Manipulation Robot Imitation Learning

Ask PDF

Aug 6, 2026

$ω$ -0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation

Humanoid household tasks often require concurrent loco-manipulation, where the robot must move, adjust posture, maintain balance, and manipulate objects as a single coordinated behavior. Yet existing humanoid policies typically decompose locomotion and manipulation, while recent world-action models remain either arm-centric or video-centered. We present

ω

-0, a latent predictive whole-body world-action model for real-world humanoid concurrent loco-manipulation. Given a language instruction, current visual observation, and robot proprioceptive state,

ω

-0 directly predicts controller-compatible whole-body action latents for real-robot execution. Rather than reconstructing future videos,

ω

-0 learns compact future observation embeddings as a lightweight predictive objective, coupling latent visual foresight with diffusion-based whole-body action generation. The model supports egocentric RGB, exocentric RGB, and exocentric depth inputs, and leverages controller-based simulation replay to ground human/public visual-motion priors into robot-executable action latents. We further collect

ω

-HOME, a 40+ hour real-world household humanoid dataset with synchronized multi-view observations, whole-body SMPL motions, robot states, and action latents. Real-world experiments on 11 household tasks demonstrate that a single

ω

-0 model can produce smooth manipulate-while-moving behaviors and consistently outperform representative imitation learning, VLA, humanoid, and WAM baselines.

Zhe Li, Zhenzhe Zhang, Yangyang Wei +8

MARS Lab, NTU · BAAI · PKU +1

Humanoid Loco-Manipulation Language-Conditioned Robot Manipulation Humanoid Robot Control World Action Models Latent World Models

Ask PDF

Aug 7, 2026

LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation

Long-horizon humanoid loco-manipulation requires composing versatile whole-body skills and reliable high-level decision making. Existing methods often coordinate pretrained skills with scripted planners, finite-state machines or task-specific model-free policies, restricting their ability to handle complex task sequences. To address this limitation, we propose LUCID, a hierarchical model-based reinforcement learning framework that plans over reusable skills through imagined rollouts of a learned dynamics model. LUCID first trains a structured latent-conditioned low-level policy via adversarial imitation and then freezes it while jointly learning a high-level policy and macro-dynamics world model. The world model predicts the temporally extended state transitions induced by latent decisions, enabling high-level policy optimization through imagined rollouts. We evaluate our framework across various simulated multi-object rearrangement scenarios. Experimental results show that LUCID improves the full-task success and partial-completion rates compared to prior baseline methods, demonstrating its effectiveness in complex sequential loco-manipulation tasks.

Cheng Guo, Mingzhe Ni, Angelo Cangelosi +1

Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy

Humanoid Loco-Manipulation Latent Action Learning Robot Skill Learning Model-Based RL

Ask PDF

Abstract

Explore similar work

ViLoMan: Learning Visual-Proprioceptive Whole-Body Loco-Manipulation Skills for Humanoid Robots

ωωω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation

LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation

$ω$ -0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation