Coordinating Humanoid Loco-Manipulation through Action-Conditioned Latent Foresight

Ju Dong; Xinyu Xiao; Yiming Zhou; Tianshuo Yang; Xuerong Liu; Lei Zhang; Jian Chen; Haocheng Zhao

cs.RONov 3, 2026

Coordinating Humanoid Loco-Manipulation through Action-Conditioned Latent Foresight

Authors: Ju Dong, Xinyu Xiao, Yiming Zhou, Tianshuo Yang, Xuerong Liu, Lei Zhang, Jian Chen, Haocheng Zhao,

Organizations: TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, University of Hamburg, Hamburg, Germany. · Kinetix AI, China. · The University of Hong Kong, Hong Kong, China. · University of Science and Technology of China, Hefei, China. ·

Abstract

Humanoid loco-manipulation requires coordinated body motion and hand interaction over time, and a plausible-looking command sequence can still drop a held object or leave an interaction incomplete. We present action-conditioned latent foresight, a latent world-action model (WAM) that evaluates proposed command sequences before execution. An action generator proposes joint motion-hand command sequences from vision, language, and the robot state. An action-conditioned endpoint predictor estimates the visual features each sequence prefix reaches and measures progress toward a demonstration-derived subgoal, while a joint temporal prior scores temporal consistency given the robot state and preceding commands. The robot executes the first commands of the best-scoring sequence and replans from the next observation. During training, action consistency regularization reconstructs the first demonstrated command from observed and predicted feature changes, encouraging the predictor to retain action-relevant information. On ten tasks in the humanoid loco-manipulation simulation benchmark SIMPLE, our method achieves 89.7% average success, compared with 73.7% for Psi-Zero and 82.0% for DiT4DiT. On seven real-robot tasks, it achieves 85.7% under standard conditions and 65.7% under unseen object configurations, while GR00T N1.7 achieves 70.0% and 48.6%, respectively. Project page: https://ju6276.github.io/Action-Grounded-Foresight/

Figures & tables

Fig. 2: Architecture. $\mathbf{f}$ denotes V-JEPA features. Each prefix $B_{t}^{(i)}$ receives visual progress $V_{i}$ toward the active subgoal $g_{t}$ (front of the stack) and temporal-consistency energy $E_{i}$ . In training, the demonstrated prefix $B_{t}$ replaces sampled prefixes, and its first command $a_{t,0}$ is reconstructed as $\hat{a}_{t,0}$ from $\Delta z$ and from $\Delta\hat{z}$ . Score bars are schematic; snowflakes: frozen weights.

Setting	Value
Real robot	Unitree G1
Egocentric camera	ZED2
Training hardware	$8\times$ NVIDIA A100
Evaluation (sim./real)	$1\times$ NVIDIA RTX 4090D (48 GB)
Onboard deployment test	Jetson Orin NX (16 GB)
Proposal length	Sim.: $H=30$ ; real: $H=40$

TABLE I: Platforms and implementation settings.

Fig. 3: Representative task scenes: the other six real-robot tasks and the ten SIMPLE simulation tasks. Place Basket on Table is shown in Fig. 1 .

Fig. 4: Grasping behavior of our method under standard and unseen object configurations. Rings mark the grasping hand.

Method	XMovePick	BendPick	Handover	Mobile P&P	Grasp	XMoveBendPick	Avg. (%)
	Main SIMPLE Tasks (L0/L1/L2)
$\Psi_{0}$ [ 4 ]	10 \| 10 \| 6	10 \| 10 \| 10	7 \| 7 \| 10	7 \| 5 \| 6	10 \| 10 \| 8	10 \| 9 \| 9	85.6
GR00T N1.6 [ 17 ]	10 \| 10 \| 7	7 \| 7 \| 6	1 \| 3 \| 3	0 \| 0 \| 0	9 \| 9 \| 7	4 \| 4 \| 1	48.9
$\pi_{0.5}$ [ 3 ]	7 \| 5 \| 1	10 \| 10 \| 8	5 \| 4 \| 5	3 \| 3 \| 3	10 \| 10 \| 8	0 \| 0 \| 0	51.1
InternVLA-M1 [ 40 ]	0 \| 0 \| 0	5 \| 5 \| 0	0 \| 0 \| 0	0 \| 0 \| 0	0 \| 0 \| 0	3 \| 5 \| 7	13.9
H-RDT [ 41 ]	0 \| 0 \| 2	0 \| 0 \| 1	0 \| 1 \| 0	0 \| 0 \| 0	0 \| 0 \| 0	0 \| 0 \| 0	2.2

TABLE II: Six main SIMPLE tasks: L0/L1/L2 successes per ten trials. DiT4DiT, Fast-WAM, VLA-JEPA, and Ours are evaluated by us; other rows are from SIMPLE [ 16 ] . Task names follow SIMPLE’s results table: Mobile P&P is LocomotionPickBetweenTables, and BendPick and Grasp (TabletopGrasp) use motion-planning demonstrations. Bold/underline mark best/second-best.

Method	OpenOven	PickAndPlace AndHugContainer	OpenFaucet	OpenTrashCan	Avg. (%) (4 tasks)	Overall (%) (10 tasks)
	Additional SIMPLE Tasks (L0/L1/L2)
$\Psi_{0}$ [ 4 ]	7 \| 5 \| 4	7 \| 6 \| 3	3 \| 3 \| 4	7 \| 9 \| 9	55.8	73.7
DiT4DiT [ 11 ]	10 \| 10 \| 9	7 \| 8 \| 8	5 \| 6 \| 7	6 \| 7 \| 5	73.3	82.0
Fast-WAM [ 12 ]	1 \| 1 \| 0	0 \| 0 \| 0	2 \| 1 \| 0	2 \| 2 \| 1	8.3	27.3
VLA-JEPA [ 23 ]	8 \| 9 \| 7	8 \| 8 \| 7	6 \| 2 \| 4	4 \| 3 \| 4	58.3	49.0
Ours	9 \| 10 \| 10	9 \| 9 \| 8	5 \| 7 \| 5	8 \| 8 \| 9	80.8	89.7

TABLE III: Four additional SIMPLE tasks. Results for

\Psi_{0}

are from SIMPLE [ 16 ] ; we evaluate the other methods. Reporting follows Table II . Overall is the average success rate across all ten tasks.

Method	Dispose of Trash	Collect Toy with Cart	Water Plant	Pack and Stow Power Strip	Load Washer	Retrieve Shirt from Drawer	Place Basket on Table	Avg.
(a) Standard conditions
GR00T N1.7	9/10	6/10	8/10	8/10	2/10	8/10	8/10	70.0
$\Psi_{0}$	2/10	6/10	6/10	4/10	2/10	2/10	1/10	32.9
DiT4DiT	4/10	8/10	8/10	6/10	8/10	4/10	6/10	62.9
Ours	6/10	6/10	10/10	8/10	10/10	10/10	10/10	85.7
(b) Unseen object configurations

TABLE IV: Real-robot success under standard and unseen object configurations. Cells show successes out of ten trials; Avg. is the mean success rate across tasks (%). Bold/underline mark best/second-best.

Fig. 5: Average success on five SIMPLE tasks versus the number of training demonstrations per task.

Fig. 6: Failure cases of our method: (a) OpenFaucet (simulation), (b) Dispose of Trash, and (c) Collect Toy with Cart.

Configuration	Main (6 tasks)	Additional (4 tasks)	Overall (10 tasks)
Feature-shift variant	91.7	73.3	84.3
Endpoint (Ours)	95.6	80.8	89.7

TABLE V: Prediction formulation: closed-loop success on the SIMPLE benchmark (%).

Configuration	Open Oven	PickAnd PlaceAnd Hug Container	Open Faucet	Open TrashCan	Avg. (%)
w/o ACR	7 \| 7 \| 6	7 \| 6 \| 6	3 \| 4 \| 3	6 \| 6 \| 5	55.0
w/ ACR	8 \| 8 \| 8	8 \| 7 \| 7	4 \| 5 \| 4	7 \| 6 \| 7	65.8

TABLE VI: Action consistency regularization (ACR) with visual-only scoring. Counts show successes out of ten trials at L0/L1/L2; Avg. is the success rate over all 120 episodes.

Configuration	Open Oven	PickAnd PlaceAnd Hug Container	Open Faucet	Open TrashCan	Avg. (%)
Visual only	8 \| 8 \| 8	8 \| 7 \| 7	4 \| 5 \| 4	7 \| 6 \| 7	65.8
1st-order smoothness	8 \| 8 \| 8	8 \| 8 \| 7	4 \| 5 \| 4	7 \| 7 \| 7	67.5
2nd-order smoothness	8 \| 9 \| 8	8 \| 7 \| 8	4 \| 6 \| 5	7 \| 7 \| 7	70.0
Factorized prior	8 \| 9 \| 9	8 \| 8 \| 8	4 \| 6 \| 4	8 \| 7 \| 7	71.7
Joint prior (Ours)	9 \| 10 \| 10	9 \| 9 \| 8	5 \| 7 \| 5	8 \| 8 \| 9	80.8

TABLE VII: Temporal-scoring ablation. Counts follow Table VI ; the joint-prior row uses the same four-task results as Table III .

Method	GPU memory (GB) $\downarrow$	Trainable params. (B)	Inference (Hz) $\uparrow$
GR00T N1.6 [ 17 ]	6.8	1.62	15.46
DiT4DiT [ 11 ]	21	2.35	3.8
Fast-WAM [ 12 ]	28	6.7	5.2
Ours	7.2	3.0	9.1

TABLE VIII: Inference cost on SIMPLE. Hz denotes command chunks per second; timing for Ours includes sequence generation, visual prediction, and prior scoring.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Task	Configurations
Dispose of Trash	Tissue packet; tea bottle.
Collect Toy with Cart	Two arrangements of the blue target toy and two neighboring distractors.
Water Plant	Watering can shifted left; shifted right; spout reversed.
Pack and Stow Power Strip	Power strip rotated; repositioned on the tabletop.
Load Washer	Purple or pink T-shirt, placed on either the right or left side of the washer top.
Retrieve Shirt from Drawer	Target T-shirt repositioned; gray T-shirt nearby; pink T-shirt nearby.

Appendix

TABLE S1: Unseen object configurations.

Robotic Loco-Manipulation

See all 41 papers

$ω$ -0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation
Aug 6, 2026Zhe Li, Zhenzhe Zhang, Yangyang Wei +8Humanoid Loco-Manipulation Language-Conditioned Robot Manipulation
MARS Lab, NTU · BAAI · PKU +1
Being-M0.7: A Latent World-Action Model for Humanoid Robots
Oct 8, 2026Junpeng Yue, Boyuan Li, Yuxuan Wang +11Humanoid Loco-Manipulation World Action Models
BeingBeyond Team
PACT-WAM: Predicting Actions and Visual Foresight with Compact Temporal Encoding for Robot Manipulation
Feb 5, 2026Yushan Liu, Jingjing Fan, Shoujie Li +3World Models for Robotics Visuomotor Policy Learning
Tsinghua University · Nanyang Technological University · Xspark AI

Aug 6, 2026J/K move · Enter open · S save

$ω$ -0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation

Zhe Li, Zhenzhe Zhang, Yangyang Wei, Wenjie Zhang+7

MARS Lab, NTU · BAAI · PKU+1

Humanoid Loco-Manipulation Language-Conditioned Robot Manipulation Humanoid Robot Control World Action Models Latent World Models

Open paper AskPDF arXiv

Humanoid household tasks often require concurrent loco-manipulation, where the robot must move, adjust posture, maintain balance, and manipulate objects as a single coordinated behavior. Yet existing humanoid policies typically decompose locomotion and manipulation, while recent world-action models remain either arm-centric or video-centered. We present

ω

-0, a latent predictive whole-body world-action model for real-world humanoid concurrent loco-manipulation. Given a language instruction, current visual observation, and robot proprioceptive state,

ω

-0 directly predicts controller-compatible whole-body action latents for real-robot execution. Rather than reconstructing future videos,

ω

-0 learns compact future observation embeddings as a lightweight predictive objective, coupling latent visual foresight with diffusion-based whole-body action generation. The model supports egocentric RGB, exocentric RGB, and exocentric depth inputs, and leverages controller-based simulation replay to ground human/public visual-motion priors into robot-executable action latents. We further collect

ω

-HOME, a 40+ hour real-world household humanoid dataset with synchronized multi-view observations, whole-body SMPL motions, robot states, and action latents. Real-world experiments on 11 household tasks demonstrate that a single

ω

-0 model can produce smooth manipulate-while-moving behaviors and consistently outperform representative imitation learning, VLA, humanoid, and WAM baselines.

Method	XMovePick	BendPick	Handover	Mobile P&P	Grasp	XMoveBendPick	Avg. (%)
	Main SIMPLE Tasks (L0/L1/L2)
$\Psi_{0}$ [ 4 ]	10 \| 10 \| 6	10 \| 10 \| 10	7 \| 7 \| 10	7 \| 5 \| 6	10 \| 10 \| 8	10 \| 9 \| 9	85.6
GR00T N1.6 [ 17 ]	10 \| 10 \| 7	7 \| 7 \| 6	1 \| 3 \| 3	0 \| 0 \| 0	9 \| 9 \| 7	4 \| 4 \| 1	48.9
$\pi_{0.5}$ [ 3 ]	7 \| 5 \| 1	10 \| 10 \| 8	5 \| 4 \| 5	3 \| 3 \| 3	10 \| 10 \| 8	0 \| 0 \| 0	51.1
InternVLA-M1 [ 40 ]	0 \| 0 \| 0	5 \| 5 \| 0	0 \| 0 \| 0	0 \| 0 \| 0	0 \| 0 \| 0	3 \| 5 \| 7	13.9
H-RDT [ 41 ]	0 \| 0 \| 2	0 \| 0 \| 1	0 \| 1 \| 0	0 \| 0 \| 0	0 \| 0 \| 0	0 \| 0 \| 0	2.2

Configuration	Open Oven	PickAnd PlaceAnd Hug Container	Open Faucet	Open TrashCan	Avg. (%)
Visual only	8 \| 8 \| 8	8 \| 7 \| 7	4 \| 5 \| 4	7 \| 6 \| 7	65.8
1st-order smoothness	8 \| 8 \| 8	8 \| 8 \| 7	4 \| 5 \| 4	7 \| 7 \| 7	67.5
2nd-order smoothness	8 \| 9 \| 8	8 \| 7 \| 8	4 \| 6 \| 5	7 \| 7 \| 7	70.0
Factorized prior	8 \| 9 \| 9	8 \| 8 \| 8	4 \| 6 \| 4	8 \| 7 \| 7	71.7
Joint prior (Ours)	9 \| 10 \| 10	9 \| 9 \| 8	5 \| 7 \| 5	8 \| 8 \| 9	80.8

Abstract

Robotic Loco-Manipulation

Explore similar work

ωωω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation

Being-M0.7: A Latent World-Action Model for Humanoid Robots

PACT-WAM: Predicting Actions and Visual Foresight with Compact Temporal Encoding for Robot Manipulation

$ω$ -0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation