Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline. Project page https://dexforesight.github.io/
Figures & tables
Figure 1: Learning and transferring spatial foresight. Demonstrations spanning 60 tasks and five spatial categories train an action-conditioned spatial world model to predict visual dynamics and 3D hand flow. The learned spatial foresight is transferred into a low-level policy, enabling action generation without world-model execution or future decoding at deployment. DexForesight outperforms prior world-action models in both simulation and real-robot evaluations.
Figure 2: DexForesight architecture and query alignment. The spatial world model learns action-conditioned hand motion and visual changes from RGB, offline scene depth, hand geometry, and language. Demonstrated controls condition the world queries through the query adapter and are also supplied to both prediction experts. Queries from the frozen world model supervise policy queries, which condition the action expert.
Figure 3: Real-robot data and training. Recording hours and example frames for the five task categories. The final stage uses 100 hours with depth estimated offline by StereoAnyVideo.
Figure 4: Experimental setup and tasks. Top: the Sharpa North robot with 22-DoF Sharpa Wave hands and the three real-robot tasks. Bottom: the Franka Panda with a 16-DoF Allegro hand and example simulation tasks.
Method
Single-arm
Bimanual
All
π0.5
53.0
13.6
35.1
GR00T N1.7
61.5
34.8
49.4
Fast-WAM
60.2
39.8
50.9
Cosmos3
59.5
37.4
49.5
DexForesight (based on GR00T N1.7)
76.3
41.6
60.5
Table 1: Simulation success rates (%). 100 trials per task.
Method
Screw
Close
Insert
Overall
GR00T N1.7
40.0
35.0
70.0
48.3
QwenVLA
40.0
15.0
35.0
30.0
WLA
50.0
30.0
70.0
50.0
Fast-WAM
15.0
5.0
10.0
10.0
Cosmos3
25.0
15.0
50.0
30.0
DexForesight
50.0
45.0
80.0
58.3
Table 2: Real-robot success rates (%). 20 trials per task. Screw / Close / Insert abbreviate Screw On Lid / Close Box / Insert Battery.
Method
0.13 s
0.27 s
0.54 s
DexForesight
3.92 [3.47, 4.13]
6.33 [6.01, 7.07]
10.01 [9.52, 11.01]
QwenVLA action-only
4.12 [3.83, 4.47]
7.70 [7.32, 8.09]
13.59 [13.01, 14.21]
GR00T N1.7
4.39 [4.12, 4.67]
7.99 [7.66, 8.30]
13.74 [13.14, 14.42]
Zero-motion prior
5.08 [4.59, 5.61]
9.79 [8.93, 10.62]
18.48 [16.99, 19.83]
Table 3: Future hand-motion information in frozen deployed latents. Linear-probe 3D displacement error in mm ( ↓ ); brackets are 95% episode-bootstrap CIs. Features are extracted from the policy alone.
Variant
Single-arm
Bimanual
All
DexForesight
76.3
41.6
60.5
Direct additions to GR00T
DINO future-residual head
63.5
23.8
45.5
Hand-flow head
66.0
37.6
53.1
DINO and hand-flow heads
64.7
31.4
49.5
Depth and AttnRes
59.2
33.2
47.4
Table 4: Simulation ablations (%). Single-arm, bimanual, and overall success rates across 11 DexJoCo tasks.
World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.
Bimanual dexterous tool use remains challenging for robots due to high-dimensional hand configurations and complex hand-tool-object dynamics and contact. Most existing control policies depend on future configuration references provided from demonstrations, while future action-conditioned world models require slow online planning over high-dimensional action sequences. A significant challenge is generating a dynamically consistent future reference trajectory without relying on privileged states from demonstrations or slow counterfactual planning. We propose DexFuture, a hierarchical system that couples a high-level Future-State Visuomotor Target Predictor with a low-level Target-Conditioned Structured Dexterous Policy. Conditioned on egocentric RGB, proprioceptive and geometric history, the high-level predictor constructs structured hand-tool-object visuomotor embeddings and uses a horizon-conditioned transformer to generate a multi-step future target trajectory. Then, the low-level policy tracks them with a target-conditioned per-link transformer. This hierarchy decouples coarse future reference generation from fine-grained action control, and slow long-horizon semantic prediction from high-frequency execution. On OakInk2 bimanual tool-use tasks, DexFuture achieves 90% of the privileged-oracle performance, compared to 7% for a no-reference policy. DexFuture operates at 60 Hz, approximately 250 times faster than DexWM-style Cross-Entropy Method (CEM) planning with a future action-conditioned world model.
Before acting on an object, a robot needs to anticipate how its hand motion will change the physical world and what the resulting scene will look like. Existing approaches typically entangle action-conditioned dynamics, camera motion, and visual synthesis within a video generation model, or represent dynamics as point trajectories without explicit object structure. We introduce DeXplicit, a dexterous manipulation world model that instead predicts how the world evolves explicitly in metric 3D before rendering its visual consequences. Given an initial scene and a future hand trajectory, a lightweight 3D dynamics network predicts object-centric motion and deformation, providing structured and geometrically grounded future world states. These predictions then serve as a strong scaffold for video generation: appearance is transported through the predicted 3D motion and projected into future views, allowing the renderer to focus on completing uncertain and newly visible content rather than implicitly learning world dynamics from pixels. This explicit-3D-first decomposition also enables efficient causal generation, requiring only two denoising steps per block while conditioning on generated history. Across HOT3D, OakInk2, and PhysTwin, DeXplicit reduces moved-point error by 27-40% relative to PointWorld and improves object fidelity IoU+ by 47% on HOT3D under a shared renderer. Our design with two-step causal generation achieves a 57x speedup from our 14B bi-directional configuration in video synthesis.
Adam Lee, Mutian Xu, Yukang Cao +2
S-Lab, Nanyang Technological University · University of California, Berkeley