Predicting future scene or object states does not necessarily reveal how robot-object interactions will advance or hinder a task. This challenge is particularly pronounced in dynamic robotic manipulation, where objects continue moving during perception and execution. Effective action generation therefore requires not only modeling object dynamics but also anticipating how future interactions will shape task progress. To this end, we propose PhysVAM, a Physics-oriented Future Value Action Model that represents anticipated interactions through their effects on future task progress. This value-based formulation provides task-directed foresight without reconstructing future visual states. Conditioned on current observations and historical object and robot motion, PhysVAM jointly learns action generation and multi-horizon future-value prediction. The value branch learns horizon-specific latent representations through future task-value prediction from successful and failed trajectories. These future-value representations are integrated into the action policy through gated cross-attention, guiding action generation toward interactions that facilitate task success. Extensive simulation and real-world experiments demonstrate superior task success, with absolute success-rate improvements of 27.0 and 27.5 percentage points over the strongest baselines in simulation and real-world tasks, respectively. Project page: https://metarobo.github.io/physvam.github.io/.
Figures & tables
Figure 1: Overview of PhysVAM. (A) Comparison of visual and value-based future modeling. Visual prediction may suffer from geometric inconsistencies and lacks explicit task-progress supervision. In contrast, PhysVAM predicts multi-horizon future values to guide action generation. The h=0 special case illustrates how favorable current-progress estimates may still precede unsuccessful interactions. (B) Multi-horizon value predictions decrease following a missed grasp and increase during subsequent recovery and successful lifting, reflecting changes in anticipated task progress. (C) PhysVAM achieves the highest success rates among the compared methods in both simulation and real-world tasks.
Figure 2: Framework of proposed PhysVAM. Historical object and robot motion provide complementary temporal context, while ValueDiT learns multi-horizon, task-directed future representations. These latent features guide ActionDiT through gated cross-attention to guide action generation toward task success.
Methods
Moderate-speed
High-speed
Average
Avocado
Beer
Potato
Apple
Lemon
Can
D-Bottle
W-Bottle
Egg
Cup
SR (%) ↑
Time (s) ↓
FastWAM [ 3 ]
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
11.67 ± 8.50
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
1.67 ± 2.36
3.33 ± 2.36
0.00 ± 0.00
1.67 ± 0.47
11.59 ± 0.16
DynamicWAM [ 4 ]
6.67 ± 6.24
5.00 ± 4.08
0.00 ± 0.00
5.00 ± 0.00
1.67 ± 2.36
0.00 ± 0.00
25.00 ± 4.08
8.33 ± 2.36
1.67 ± 2.36
0.00 ± 0.00
5.33 ± 0.62
10.96 ± 0.10
InternVLA-A1.5 [ 13 ]
3.33 ± 2.36
0.00 ± 0.00
0.00 ± 0.00
3.33 ± 2.36
0.00 ± 0.00
0.00 ± 0.00
18.33 ± 2.36
0.00 ± 0.00
15.00 ± 0.00
0.00 ± 0.00
4.00 ± 0.82
10.82 ± 0.03
DynamicVLA [ 2 ]
23.33 ± 2.36
10.00 ± 7.07
5.00 ± 4.08
8.33 ± 2.36
16.67 ± 2.36
1.67 ± 2.36
83.33 ± 10.27
10.00 ± 7.07
6.67 ± 6.24
8.33 ± 8.50
17.33 ± 0.24
8.92 ± 0.09
PUMA [ 5 ]
10.00 ± 0.00
65.00 ± 8.16
41.67 ± 10.27
11.67 ± 8.50
85.00 ± 7.07
1.67 ± 2.36
58.33 ± 2.36
10.00 ± 7.07
0.00 ± 0.00
35.00 ± 4.08
31.83 ± 1.70
9.56 ± 0.15
Table 1: Main comparison results and transferability of proposed value guidance . “+” denotes augmentation with our value design. Best and second-best results are shown in bold and underlined , respectively.
Model Variant
SR (%) ↑
Time (s) ↓
(a) Value branch
Without Value Branch
50.50 ± 1.08
9.65 ± 0.02
(b) Value representations
Scalar Conditioning
40.83 ± 1.43
9.02 ± 0.16
Binary Conditioning
48.50 ± 2.55
8.85 ± 0.08
(c) Multi-horizon value prediction
Table 2: Ablations of value-guided action generation. We examine the value branch, conditioning representations, horizon-specific query targets, and future progress prediction.
Model Variant
SR (%) ↑
Time (s) ↓
(a) Object motion representation
Without Object Motion History
17.17 ± 2.39
9.65 ± 0.02
RGB History
22.33 ± 1.55
9.27 ± 0.26
Object Position History
30.67 ± 2.01
9.53 ± 0.03
Object Velocity History (Oracle)
34.00 ± 3.27
9.00 ± 0.10
Object Velocity History (Estimated)
33.67 ± 0.85
8.93 ± 0.04
Table 3: Ablations of object and robot motion histories . All historical inputs span four frames.
Figure 3: Effect of Object and Robot Motion History Length.
Figure 4: Real-world experimental setup . Our platform uses a single AgileX PiPER arm and includes multiple targets that undergo linear, circular, and compound rolling dynamic manipulation tasks.
Figure 5: Object motion estimation for real-world tasks. We employ FoundationPose to estimate and track object poses from RGB-D observations using an initial mask and an object mesh. Four-frame position histories and their derived velocities jointly form the object motion inputs to PhysVAM.
Figure 6: Real-world experimental results for DynamicVLA, PUMA, and PhysVAM under multiple scenes.
Window
Lift ( n=119 )
Miss ( n=101 )
Gap
±8
+0.103±0.279
−0.150±0.240
+0.253
±16
+0.254±0.370
−0.185±0.237
+0.439
Table 4: Closure-aligned future-value changes in simulation. Mean ± standard deviation across 220 valid grasp attempts from 200 evaluation trajectories, including retries.
Figure 7: Closure-aligned future-value predictions for Lift and Miss attempts in simulation. Representative attempts are collected in the same environment using the same policy, with similar object–gripper positions at closure. RGB observations and 4-step-ahead task-value predictions illustrate the progress gains and setbacks associated with the two interaction outcomes. The vertical reference line marks gripper closure.
Figure 8: Interaction-sensitive future-value predictions in simulation. RGB observations and 4-step-ahead task-value predictions illustrate (a) a missed grasp, (b) re-approach after a setback, and (c) successful grasping and lifting. Each row shows a different trajectory. Dashed lines mark the corresponding interaction events, and yellow circles highlight the target objects.
Methods
Moderate-speed
High-speed
Average
Avocado
Beer
Potato
Apple
Lemon
Can
D-Bottle
W-Bottle
Egg
Cup
SR (%) ↑
Time (s) ↓
Main (3-CAM)
60.00 ± 10.80
41.67 ± 10.27
65.00 ± 7.07
45.00 ± 4.08
88.33 ± 6.24
36.67 ± 10.27
86.67 ± 8.50
93.33 ± 4.71
28.33 ± 8.50
43.33 ± 4.71
58.83 ± 1.65
8.23 ± 0.13
Reweighted (3-CAM)
66.00 ± 4.08
49.33 ± 8.50
61.00 ± 4.08
42.67 ± 2.36
84.33 ± 6.24
37.67 ± 4.71
89.33 ± 4.71
91.00 ± 10.80
29.33 ± 2.36
64.33 ± 6.24
61.50 ± 1.08
8.88 ± 0.57
Main (2-CAM)
55.00 ± 4.08
46.67 ± 10.27
46.67 ± 4.71
25.00 ± 10.80
45.00 ± 4.08
8.33 ± 4.71
93.33 ± 6.24
86.67 ± 6.24
28.33 ± 6.24
65.00 ± 4.08
50.00 ± 2.55
9.05 ± 0.10
Table 5: Effects of value-guided action reweighting and multi-view observations. Main (3-CAM) denotes the default setting, Reweighted applies value-guided action weights, and Main (2-CAM) uses two camera views. Best results are shown in bold.
Figure 9: Real-world future-value predictions during turntable manipulation. RGB observations and 4-step-ahead task-value predictions illustrate progress gains during successful lifting and a setback when the cup tips over. Reported value changes are measured around the corresponding lift or tip-over event.
Figure 10: Effect of Hyper λV .
Methods
Moderate-speed
High-speed
Average
Avocado
Beer
Potato
Apple
Lemon
Can
D-Bottle
W-Bottle
Egg
Cup
SR (%) ↑
Time (s) ↓
(a) Value branch
Without Value Branch
48.33 ± 6.24
36.67 ± 6.24
56.67 ± 2.36
41.67 ± 2.36
75.00 ± 4.08
40.00 ± 12.25
85.00 ± 7.07
71.67 ± 9.43
16.67 ± 4.71
33.33 ± 2.36
50.50 ± 1.08
9.65 ± 0.02
(b) Value representations
Scalar Conditioning
53.33 ± 20.14
30.00 ± 14.72
45.00 ± 8.16
36.67 ± 8.50
46.67 ± 6.24
8.33 ± 6.24
73.33 ± 2.36
71.67 ± 9.43
21.67 ± 4.71
21.67 ± 10.27
40.83 ± 1.43
9.02 ± 0.16
Binary Conditioning
68.33 ± 2.36
25.00 ± 7.07
46.67 ± 10.27
30.00 ± 7.07
53.33 ± 6.24
40.00 ± 4.08
83.33 ± 9.43
80.00 ± 4.08
13.33 ± 2.36
45.00 ± 14.72
48.50 ± 2.55
8.85 ± 0.08
Table 6: Detailed results of value-guided action generation ablations. Per-object success rates across different value branch designs, conditioning representations, and prediction objectives. Best results are shown in bold.
Methods
Moderate-speed
High-speed
Average
Avocado
Beer
Potato
Apple
Lemon
Can
D-Bottle
W-Bottle
Egg
Cup
SR (%) ↑
Time (s) ↓
(a) Object motion representation
Without Object Motion History
0.00 ± 0.00
18.33 ± 4.71
1.67 ± 2.36
43.33 ± 11.79
0.00 ± 0.00
0.00 ± 0.00
78.33 ± 10.27
6.67 ± 4.71
10.00 ± 0.00
13.33 ± 4.71
17.17 ± 2.39
9.65 ± 0.02
RGB History
5.00 ± 0.00
23.33 ± 10.27
15.00 ± 4.08
18.33 ± 6.24
6.67 ± 4.71
0.00 ± 0.00
91.67 ± 4.71
18.33 ± 12.47
21.67 ± 8.50
23.33 ± 6.24
22.33 ± 1.55
9.27 ± 0.26
Object Position History
30.00 ± 4.08
23.33 ± 4.71
15.00 ± 8.16
35.00 ± 4.08
33.33 ± 6.24
1.67 ± 2.36
85.00 ± 4.08
16.67 ± 6.24
13.33 ± 8.50
53.33 ± 12.47
30.67 ± 2.01
9.53 ± 0.03
Object Velocity History (Oracle)
6.67 ± 2.36
21.67 ± 2.36
41.67 ± 14.34
36.67 ± 2.36
26.67 ± 8.50
3.33 ± 2.36
88.33 ± 2.36
76.67 ± 6.24
23.33 ± 4.71
15.00 ± 4.08
34.00 ± 3.27
9.00 ± 0.10
Table 7: Detailed results of object and robot motion history ablations. Per-object success rates across different motion representations. All historical inputs span four frames. Best results are shown in bold.
Scenario
Object
DynamicVLA
PUMA
PhysVAM
Conveyor
Orange
30.0 (6/20)
35.0 (7/20)
70.0 (14/20)
Cube
50.0 (10/20)
60.0 (12/20)
75.0 (15/20)
Avg.
40.0
47.5
72.5
Turntable
Bread
25.0 (5/20)
35.0 (7/20)
60.0 (12/20)
Cup
20.0 (4/20)
30.0 (6/20)
70.0 (14/20)
Avg.
22.5
32.5
65.0
Table 8: Detailed per-object real-world manipulation results. Success rates (SR) are reported with successful trials and total trials in parentheses. Overall Avg. is computed by equally weighting the three dynamic manipulation scenarios. Best results are shown in bold.
Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.
Vision-Language-Action (VLA) models have demonstrated promising capabilities in robotic manipulation by integrating visual perception, language understanding, and action generation. Recent advances have further incorporated future prediction to enhance the understanding of environmental dynamics and task progression. However, existing methods often model short-term dynamics and long-term task objectives separately, while directly incorporating predicted futures into action policies without sufficiently considering their relevance to current action requirements. To address these limitations, we propose HiFOR, a hierarchical foresight framework with action-guided world retrieval. Specifically, Hierarchical Foresight Prediction jointly predicts short-term state evolution, semantic milestones, and final goal states through a Hierarchical Dependency Mask (HDM), establishing dependencies across temporal horizons. Building upon these predictions, Structure-Aware Latent World Modeling organizes future latent representations along the horizon, view, and spatial dimensions to facilitate fine-grained retrieval. Furthermore, Action-Guided Factorized World Retrieval adaptively retrieves relevant future information from the structured latent world based on evolving action representations during denoising, providing targeted guidance for action generation. Extensive experiments on two simulation benchmarks and six real-world robotic manipulation tasks demonstrate the effectiveness of our approach.
Rongxing Ding, Hongyu Qu, Kehong Wu +4
Nanjing University of Science and Technology · National University of Singapore · Shenyang Institute of Automation, Chinese Academy of Sciences +1
Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.
Zhe Tao, Feiran Wang, Gaowen Liu +2
University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco
Dynamic object manipulation requires robots to anticipate target motion while replanning fast enough to avoid executing outdated actions. Existing approaches either rely on fast reactive control without an explicit future target state or encode anticipated motion only implicitly. World Action Models (WAMs) offer a promising alternative by coupling future visual prediction with action generation, but scene-level video objectives may underrepresent target dynamics that are critical for manipulation, while iterative video-action denoising introduces substantial latency. We present TrajWAM, a trajectory-guided world action model for efficient dynamic robotic manipulation. Our key insight is to explicitly ground future visual imagination in predicted target-object motion and use the resulting future representation to condition action generation, while preserving this anticipation pathway after acceleration. Given a short history of object masks, a history-conditioned motion predictor forecasts future masks encoding the target object's trajectory. A lightweight Trajectory ControlNet then steers a pretrained video model toward trajectory-consistent futures. The resulting future representation conditions an action expert to predict executable action chunks. To enable one-step inference without breaking this interface, we introduce a decoupled distillation scheme that combines distribution matching for the video backbone, Trajectory ControlNet realignment, and action rollout distillation to support accurate one-step action prediction conditioned on the compressed future representation. Comprehensive evaluations in simulation and the real world show that TrajWAM achieves state-of-the-art performance on dynamic object manipulation tasks while enabling efficient one-step video-action inference. Project page: https://kevingeng3.github.io/TrajWAM/.
Haohan Geng, Sha Zhang, Weishuo Sun +7
University of Science and Technology of China. · The Chinese University of Hong Kong. · University of Macau. +1