Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline. Project page https://dexforesight.github.io/
Figures & tables
Figure 1: Learning and transferring spatial foresight. Demonstrations spanning 60 tasks and five spatial categories train an action-conditioned spatial world model to predict visual dynamics and 3D hand flow. The learned spatial foresight is transferred into a low-level policy, enabling action generation without world-model execution or future decoding at deployment. DexForesight outperforms prior world-action models in both simulation and real-robot evaluations.
Figure 2: DexForesight architecture and query alignment. The spatial world model learns action-conditioned hand motion and visual changes from RGB, offline scene depth, hand geometry, and language. Demonstrated controls condition the world queries through the query adapter and are also supplied to both prediction experts. Queries from the frozen world model supervise policy queries, which condition the action expert.
Figure 3: Real-robot data and training. Recording hours and example frames for the five task categories. The final stage uses 100 hours with depth estimated offline by StereoAnyVideo.
Figure 4: Experimental setup and tasks. Top: the Sharpa North robot with 22-DoF Sharpa Wave hands and the three real-robot tasks. Bottom: the Franka Panda with a 16-DoF Allegro hand and example simulation tasks.
Method
Single-arm
Bimanual
All
π0.5
53.0
13.6
35.1
GR00T N1.7
61.5
34.8
49.4
Fast-WAM
60.2
39.8
50.9
Cosmos3
59.5
37.4
49.5
DexForesight (based on GR00T N1.7)
76.3
41.6
60.5
Table 1: Simulation success rates (%). 100 trials per task.
Method
Screw
Close
Insert
Overall
GR00T N1.7
40.0
35.0
70.0
48.3
QwenVLA
40.0
15.0
35.0
30.0
WLA
50.0
30.0
70.0
50.0
Fast-WAM
15.0
5.0
10.0
10.0
Cosmos3
25.0
15.0
50.0
30.0
DexForesight
50.0
45.0
80.0
58.3
Table 2: Real-robot success rates (%). 20 trials per task. Screw / Close / Insert abbreviate Screw On Lid / Close Box / Insert Battery.
Method
0.13 s
0.27 s
0.54 s
DexForesight
3.92 [3.47, 4.13]
6.33 [6.01, 7.07]
10.01 [9.52, 11.01]
QwenVLA action-only
4.12 [3.83, 4.47]
7.70 [7.32, 8.09]
13.59 [13.01, 14.21]
GR00T N1.7
4.39 [4.12, 4.67]
7.99 [7.66, 8.30]
13.74 [13.14, 14.42]
Zero-motion prior
5.08 [4.59, 5.61]
9.79 [8.93, 10.62]
18.48 [16.99, 19.83]
Table 3: Future hand-motion information in frozen deployed latents. Linear-probe 3D displacement error in mm ( ↓ ); brackets are 95% episode-bootstrap CIs. Features are extracted from the policy alone.
Variant
Single-arm
Bimanual
All
DexForesight
76.3
41.6
60.5
Direct additions to GR00T
DINO future-residual head
63.5
23.8
45.5
Hand-flow head
66.0
37.6
53.1
DINO and hand-flow heads
64.7
31.4
49.5
Depth and AttnRes
59.2
33.2
47.4
Table 4: Simulation ablations (%). Single-arm, bimanual, and overall success rates across 11 DexJoCo tasks.