Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation

Zirui Wu; Chen Guo; Tianjian Jiang; Zeren Jiang; Zeming Li; Lingchen Yang; Longteng Duan; Ying Xue

cs.CVOct 15, 2026

Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation

Authors: Zirui Wu, Chen Guo, Tianjian Jiang, Zeren Jiang, Zeming Li, Lingchen Yang, Longteng Duan, Ying Xue,

Organizations: HKUST (GZ) · Primatix · ETH Zurich · Visual Geometry Group, University of Oxford ·

Abstract

We introduce P $^{3}$ -WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes. Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation. Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines. Project Page: https://p3-wam.pages.dev/

Figures & tables

Figure 1: P 3 -WAM unifies perception, prediction, and planning. Perceive recovers hand motion from video; Predict generates video under prescribed motion; Plan jointly generates motion and video guided by text instructions and optional visual goals.

Figure 2: Task modes and unified training. Colored tiles denote observed video

\mathbf{V}

and actions

\mathbf{A}

; gray tiles denote missing inputs. Unified training samples these patterns across clips and time.

Figure 3: P 3 -WAM couples video and action streams through joint attention and shared base weights. The action expert adds low-rank adaptation and directly predicts hand and camera states.

Figure 4: Visual goals specify spatial targets. (a) A goal image specifies how far apart the red blocks should be. (b) Intermediate goal images guide the block arrangement at each stage of a longer task; without them, placement errors accumulate.

		Perceive (mm)			Predict (full frame)
Dataset	Training setting	MPJPE $\downarrow$	RR-MPJPE $\downarrow$	PA-MPJPE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$
EgoDex-MANO	Single mode	52.491	29.170	12.577	23.106	0.8178	0.3090	757.06
	Unified training	47.587	25.637	12.184	23.668	0.8182	0.3096	653.49
ARCTIC	Single mode	227.608	27.335	13.144	12.431	0.5501	0.4940	1144.02
	Unified training	150.501	22.946	12.154	13.431	0.5687	0.4661	1035.38
HOT3D	Single mode	80.656	31.318	12.317	14.917	0.4151	0.5189	1466.68

Table 1: Unified versus single-mode training on Perceive and Predict .

Dataset	Method	MPJPE- p $\downarrow$	RR-MPJPE- p $\downarrow$	PA-MPJPE- p $\downarrow$	F1 $\uparrow$
EgoDex-MANO	Ours	32.92	31.58	13.12	99.96
	HaMeR	112.92	44.35	16.22	91.48
	Dyn-HaMR	115.53	36.66	18.29	98.17
ARCTIC	Ours	45.29	13.08	7.64	99.72
	HaMeR	85.34	28.87	11.51	97.31
	Dyn-HaMR	317.46	48.28	39.40	89.88

Table 3: Presence-aware Perceive reconstruction, measured by camera-space, root-relative (RR), and Procrustes-aligned (PA) MPJPE- p in mm (

\downarrow

), and hand-visibility F1 in percent (

\uparrow

Figure 5: Qualitative Perceive comparisons across EgoDex, ARCTIC, HOT3D, and EgoLive. Red boxes highlight missed hands or local pose errors in baseline predictions.

Figure 6: Qualitative Predict comparisons on EgoDex, HOT3D, ARCTIC, and EgoLive.

		Full frame					Hand region
Dataset	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP-I $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP-I $\uparrow$	IoU $\uparrow$
EgoDex	Ours	24.511	0.8462	0.2679	0.9448	401.26	21.498	0.7265	0.3362	0.9203	0.7082
	Wan2.2-Fun-Control	20.304	0.7570	0.3682	0.9254	719.66	18.016	0.6342	0.4488	0.8923	0.6306
	DreamDojo	19.561	0.7583	0.3781	0.8868	931.96	17.498	0.6285	0.4842	0.8537	0.3005
	Hand2World	13.287	0.6639	0.6656	0.7894	3028.61	12.290	0.5506	0.6580	0.8011	0.5296
	EgoSim	21.331	0.8189	0.2955	0.9242	649.87	19.447	0.6982	0.3647	0.9078	0.6474

Table 4: Action-conditioned video prediction across four datasets. CLIP-I measures image-feature similarity and IoU measures hand-box overlap. Darker greens mark the top three results.

Method	Goal image	Action process	Instruction adherence	Physical interaction	Temporal / identity consistency	Goal transition
Cosmos 3-Nano	✓	2.279	2.703	3.190	2.852	2.080
Cosmos 3-Nano	✗	2.725	2.817	4.146	4.684	2.732
Wan2.2 I2V-A14B	✓	2.647	2.847	3.251	2.741	2.273
Wan2.2 I2V-A14B	✗	3.120	3.119	4.119	4.406	3.178
Ours	✓	3.591	3.739	4.350	4.651	3.841
$\Delta$	–	$+0.471$	$+0.620$	$+0.205$	$-0.033$	$+0.663$

Table 5: VLM comparison of Plan -generated videos with video-generation baselines.

\Delta

compares Ours with the best baseline score for each metric across the four baseline settings.

Figure 7: Goal-conditioned Plan comparisons.

Figure 8: Robot deployment. Each example shows the initial observation, generated human motion, and robot execution after retargeting.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Evaluation	Cam. (mm)	World (mm)	RR (mm)	UV (px)	Depth (mm)
EgoDex	0.868	1.048	0.967	0.323	0.432
ARCTIC	2.600	4.212	2.543	0.882	1.137
HOT3D	1.336	2.088	1.406	0.580	0.691
EgoLive	1.498	1.627	1.573	0.470	0.891

Appendix

Table 6: Multi-domain action-codec reconstruction quality on held-out validation data. All values are dataset means; lower is better.

Panel of Figure 9 — Figure 9: Plan conditioning masks and goal encoding. Both settings observe the initial image; the default also fixes a terminal latent encoded from repeated copies of the goal image. Hand and camera trajectories are predicted for all frames.

	Human			VLM
Dimension	Plan -only	Unified training	Gain [95% CI]	Plan -only	Unified training	Gain [95% CI]
Action process	2.23	3.12	+0.88 [0.37, 1.42]	1.88	2.89	+1.01 [0.73, 1.33]
Instruction adherence	2.18	3.02	+0.83 [0.43, 1.23]	2.27	3.22	+0.94 [0.62, 1.28]
Physical interaction	1.98	2.90	+0.92 [0.53, 1.33]	2.52	3.32	+0.80 [0.46, 1.13]
Temporal / identity	2.05	3.30	+1.25 [0.73, 1.77]	2.15	3.57	+1.42 [1.08, 1.76]
Goal transition	1.72	3.55	+1.83 [1.33, 2.30]	1.98	3.29	+1.30 [1.03, 1.60]

Appendix

Table 7: Human and VLM evaluation of Plan -only and unified training. Human scores use 21 pairs across 15 tasks; VLM scores use 188 pairs across 17 tasks. Scores range from 1 to 5. Unified training gains include 95% bootstrap confidence intervals.

Dimension	1: Severe failure	3: Partial completion	5: Good visible performance
Action process (primary)	No effective action, wrong direction, or arrival through a sudden jump.	Some necessary actions are visible, but key steps are missing or transitions are flawed.	Necessary actions occur clearly in a plausible order; intermediate states support the outcome.
Instruction adherence (primary)	Core instruction is not executed, or the stopping condition is seriously violated.	Some instructions are executed, with visible omissions or constraint violations.	Observable requirements and constraints are met, with the correct stopping point.
Physical interaction	Severe penetration, self-moving or disappearing objects, or invalid interaction.	Clear local problems in contact or shape.	Observed contact, grasping, movement, and shape changes are plausible.
Temporal / identity consistency	Severe jumps or changes in object count or identity.	Local discontinuities or shape drift.	Observed hands, objects, and background remain consistent across frames.
Goal transition	No approach to the goal, or sudden replacement at the end.	Partly natural approach, with clear late corrections or discontinuities.	Natural transition across the final frames, with the required relationships established.

Appendix

Table 8: Shared human/VLM score anchors. Unobservable dimensions are recorded as missing.

Baseline	Goal image	Action process	Instruction adherence
Cosmos 3-Nano	✓	$1.311\;[1.190,1.439]$	$1.036\;[0.874,1.189]$
Cosmos 3-Nano	✗	$0.865\;[0.661,1.060]$	$0.922\;[0.657,1.178]$
Wan2.2 I2V-A14B	✓	$0.944\;[0.743,1.143]$	$0.892\;[0.669,1.107]$
Wan2.2 I2V-A14B	✗	$0.471\;[0.245,0.703]$	$0.620\;[0.364,0.875]$

Appendix

Table 9: Ours-minus-baseline VLM score differences with 95% bootstrap intervals. Positive differences indicate higher scores for Ours.

Figure 10: Interface of the questionnaire. Reviewers see the instruction, initial and goal images, and one anonymous video.

Panel of Figure 11 — Figure 11: Questionnaire response interface. Each dimension has a 1–5 rating and an unobservable option, with optional notes.

		Joint error (mm)			Hand visibility (%)
Dataset	Method	MPJPE $\downarrow$	RR-MPJPE $\downarrow$	PA-MPJPE $\downarrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
EgoDex-MANO	Ours	32.92	31.58	13.12	99.92	100.00	99.96
	HaMeR	106.86	43.49	14.80	84.98	99.06	91.48
	Dyn-HaMR	103.58	33.82	15.05	99.52	96.86	98.17
ARCTIC	Ours	42.31	13.03	7.57	99.93	99.51	99.72
	HaMeR	51.36	27.36	9.42	99.23	95.47	97.31

Appendix

Table 10: Joint reconstruction and hand visibility in Perceive mode. We report camera-space, root-relative (RR), and Procrustes-aligned (PA) joint errors without missing-hand penalties, alongside separately scored hand-in-frame precision, recall, and F1.

	Without presence penalties			With presence penalties
Method	MPJPE	RR-MPJPE	PA-MPJPE	MPJPE- p	RR-MPJPE- p	PA-MPJPE- p
HaMeR	106.23	39.75	11.98	112.30	40.90	13.70
Dyn-HaMR	102.62	24.20	12.26	114.58	27.45	15.86
Ours	30.07	19.02	10.98	30.07	19.02	10.98

Appendix

Table 11: Reconstruction against original EgoDex joint annotations. Errors are in mm (lower is better);

p

denotes presence penalties.

	Frames $2$ – $81$		Frames $2$ – $77$
Image source	MPJPE	RR-MPJPE	MPJPE	RR-MPJPE
EgoDex	4.79	2.31	4.78	2.32
Generated images	4.99	2.76	4.96	2.79

Appendix

Table 12: Agreement between Plan hand trajectories and Perceive reconstructions of the generated videos. MPJPE is in mm (lower is better).

Scale $\gamma$	Full-image PSNR $\uparrow$	SAM3 20% hand PSNR $\uparrow$
0.0	20.644	16.764
0.5	23.726	21.008
1.0	25.327	23.724
1.5	25.271	23.899
2.0	25.035	23.651

Appendix

Table 13: Action-guidance sensitivity for Predict on 20 EgoDex windows. Full-image PSNR is measured on native

848\!\times\!480

frames 2–81. Hand PSNR is measured inside

128\!\times\!128

GT-pose crops using GT-only SAM3 hand pixels with at least 20% crop coverage. Higher is better.

Dataset	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
EgoDex	Ours (Wan2.2)	24.511	0.8462	0.2679
	Ours (Wan2.1)	23.540	0.8262	0.2984
	EgoSim	21.331	0.8189	0.2955
ARCTIC	Ours (Wan2.2)	14.106	0.5785	0.4411
	Ours (Wan2.1)	13.921	0.5903	0.4437
	EgoSim	12.874	0.5545	0.4968

Appendix

Table 14: Predict results with Wan2.2 and Wan2.1 backbones, alongside EgoSim. Full-frame metrics follow Table 4 . Macro averages the four datasets equally. Bold entries denote the best result for each dataset.

Dataset	Method	MPJPE $\downarrow$	RR-MPJPE $\downarrow$	PA-MPJPE $\downarrow$
EgoDex	Ours (Wan2.2)	32.92	31.58	13.12
	Ours (Wan2.1)	30.31	18.79	10.88
ARCTIC	Ours (Wan2.2)	42.47	12.99	7.55
	Ours (Wan2.1)	38.99	13.14	8.36
HOT3D	Ours (Wan2.2)	87.07	20.12	9.42
	Ours (Wan2.1)	109.99	24.86	10.05

Appendix

Table 15: Perceive results with Wan2.2 and Wan2.1 backbones. We report ordinary camera-space, root-relative (RR), and Procrustes-aligned (PA) MANO joint errors in millimeters. Wan2.2 entries evaluate the dense predictions from Table 3 before visibility filtering and mesh matching. Macro averages the four datasets equally.

Figure 12: Qualitative comparison of single-mode and unified training. Perceive : unified training places the reconstructed palm closer to the observed hand. Predict : the table extent and viewpoint more closely match GT. Plan : the narrow support is positioned beside the fixed broad post. The left column shows GT for Perceive and Predict , and the goal-image reference for Plan .

Figure 13: Plan and video-generation baselines without a goal image. All methods receive only the initial image and instruction. Our model maintains the support’s shape during placement and the identities of the puzzle pieces during separation. The close-ups show deformations in the baseline outputs.

World Action Models

World Action Models connect predictions of how scenes change under action to executable policies for robots, vehicles, and other embodied agents.

See all 218 papers

PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation
Oct 2, 2026Chunghyun Park, Beomjun Kim, Seungcheol Park +5Robotic Manipulation Embodied World Models
POSTECH · KAIST · RLWRLD +1
UNITAS: A 3D-Native World Action Model for Embodied Manipulation
Oct 8, 2026Ruixiang Wang, Yongyi Su, Wenlve Zhou +14World Action Models Embodied World Models
The Chinese University of Hong Kong, Shenzhen · DexForce Technology Co., Ltd. · Foshan University +2
TrajWAM: An Efficient Trajectory-Guided World Action Model for Dynamic Robotic Manipulation
Oct 19, 2026Haohan Geng, Sha Zhang, Weishuo Sun +7Dynamic Robotic Manipulation World Action Models
University of Science and Technology of China. · The Chinese University of Hong Kong. · University of Macau. +1

Oct 2, 2026J/K move · Enter open · S save

PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

Chunghyun Park, Beomjun Kim, Seungcheol Park, Heeseung Kwon+4

POSTECH · KAIST · RLWRLD+1

Robotic Manipulation Embodied World Models Robot Imitation Learning

Open paper AskPDF arXiv

World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.

Abstract

World Action Models

Explore similar work

PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

UNITAS: A 3D-Native World Action Model for Embodied Manipulation

TrajWAM: An Efficient Trajectory-Guided World Action Model for Dynamic Robotic Manipulation