Physics-Aware Object-Centric SE(3) Forecasting for Real-Time Proactive Dynamic Grasping

Yirui Zhou; Hongli Xu; Ying Zheng; Yi Wang; Lap-Pui Chau

cs.ROOct 13, 2026

Physics-Aware Object-Centric SE(3) Forecasting for Real-Time Proactive Dynamic Grasping

Authors: Yirui Zhou, Hongli Xu, Ying Zheng, Yi Wang, Lap-Pui Chau

Organizations: The Hong Kong Polytechnic University · Technical University of Munich

Abstract

Proactive dynamic grasping requires robots to predict the future state at which physical contact will occur, rather than reacting to the currently observed object pose. But what future must the robot predict, and at what level of abstraction? We argue that contact-relevant rigid-body state in SE(3) is a compact task-sufficient abstraction for this problem: it directly determines where and when the robot should establish contact, without requiring visually complete future scenes. This insight leads to a lightweight object-centric forecast-then-select framework that predicts future rigid-body trajectories from short pose histories and uses a reachability-aware interception selector to choose feasible contact states under kinematic and timing constraints. Trained with randomized physics simulation over primitive shapes, the forecaster transfers to real objects once their metric geometry and SE(3) state histories are recovered. Experiments across simulation and real-world dynamic grasping tasks show that our framework improves the average success rate from $24.5\%$ to $62.0\%$ in simulation, and from $15.3\%$ to $67.3\%$ in real-world catching, conveyor, and turntable interception. The SE(3) forecaster runs in $15\,\mathrm{ms}$ , orders of magnitude faster than dense visual future prediction, enabling real-time contact decision making. Our results suggest that effective proactive grasping does not require imagining the entire future scene. Instead, predicting the compact physical future that determines contact provides an effective and practical abstraction for real-time dynamic grasping.

Figures & tables

Figure 1: Left: Reactive policies act on the current object pose and fail under fast dynamics. Video world models predict pixel-level futures that are slow and physically imprecise. Our method predicts contact-relevant SE(3) state — compact, fast, and action-guided. Right: Real-world dynamic grasping tasks including parabolic catching, conveyor interception, and turntable grasping.

Figure 2: Overview of PAF-Grasp. The Neural SE(3) forecaster is trained on Genesis-generated primitive-object trajectories to learn reusable rigid-body dynamics. At deployment, an anchor RGB-D observation is used offline to recover object-specific geometry and an approximate inertia prior, while the online RGB-D stream continuously updates the object’s SE(3) history for closed-loop forecasting and interception. The forecaster encodes recent motion history, estimates initial velocities, and integrates a Newton–Euler-structured neural ODE to predict future rigid-body trajectories. The forecast-guided interception selector then propagates task-specific contact frames along the predicted trajectory and selects a reachable end-effector target under timing and kinematic constraints.

Panel of Figure 3 — Figure 3: Real-world success rates across three dynamic grasping tasks and a representative reactive-tracking failure on conveyor interception.

Method	Catch			Conveyor			Turntable			Ramp			Avg. SR $\uparrow$
Method	SR $\uparrow$	PL $\downarrow$	TCT $\downarrow$	SR $\uparrow$	PL $\downarrow$	TCT $\downarrow$	SR $\uparrow$	PL $\downarrow$	TCT $\downarrow$	SR $\uparrow$	PL $\downarrow$	TCT $\downarrow$	Avg. SR $\uparrow$
$\pi_{0}$ -FAST [ 43 ]	8	0.16	1.32	16	0.46	2.34	8	0.33	1.96	8	0.25	3.03	10
$\pi_{0}$ [ 6 ]	6	0.12	0.56	50	0.44	2.3	16	0.34	2.23	26	0.18	1.99	24.5
$\pi_{0.5}$ [ 7 ]	11	0.11	0.55	46.0	0.44	2.36	16	0.33	2.10	24	0.18	2.09	24.3
VLASH [ 9 ]	6	0.11	0.54	32	0.45	2.74	26	0.43	2.4	12	0.25	2.60	19
GR00T N1.7 [ 10 ]	4	0.06	0.47	24	0.47	2.92	12	0.46	2.43	2	0.18	2.01	10.5

Table 1: Overall dynamic grasping performance. Ours uses SE(3) forecasting and reachability-aware contact-time selection. SR is over all trials; PL and TCT are over successful trials only.

Figure 6: Fast-WAM (right) failure: the gripper reaches the cup but misses the rotating handle.

Task	Pred.	Pos-FDE $\downarrow$	Rot-FDE $\downarrow$	SR $\uparrow$
Toss	Classical	0.203	12.99 ∘	36%
Toss	NODE	0.202	10.64 ∘	56%
Conveyor	Classical	0.076	10.48 ∘	78%
Conveyor	NODE	0.057	0.55 ∘	78%
Turntable	Classical	0.049	10.60 ∘	54%
Turntable	NODE	0.059	3.94 ∘	62%

Table 3: Task-aware classical predictors vs. NODE under matched noisy SE(3) inputs and the same selector. Pos-FDE: positional final displacement error; Rot-FDE: final rotation error.

Method	Pos-FDE $\downarrow$	Rot-FDE $\downarrow$
AR	0.044	12.693
NODE w/o $I^{B}$	0.023	11.241
NODE w/ $I^{B}$	0.023	4.745

Table 4: Toss ablation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Motion type	Observation horizon $K$	Prediction horizon $H$
Conveyor	Linear	$4$	$20$
Turntable	Rotation	$4$	$20$
Ramp sliding	Sliding	$4$	$16$
Parabolic catching	Projectile	$4$	$12$

Appendix

Table 6: Task-specific observation and prediction horizons.

Component	Setting
ODE solver	Dopri5, rtol $=10^{-5}$ , atol $=10^{-6}$
Context / dynamics dim.	$32/64$
MLP hidden dims.	$[64,128,128,64]$
Velocity estimator	GRU, hidden dim. $64$ , 1 layer, dropout $0.1$
Optimization	AdamW, lr $10^{-4}$ , wd $10^{-5}$ , cosine decay
Batch size / epochs	$512/3000$

Appendix

Table 7: Training configuration for the Neural SE(3) forecaster.

Component	Setting
Input representation	SE(3) pose history; no RGB/depth/velocity
Transformer dim.	$d_{\mathrm{model}}=128$
Attention heads	$8$
Encoder / decoder layers	$4/4$
Feedforward dim.	$512$
Dropout	$0.1$

Appendix

Table 8: Configuration for the autoregressive Transformer forecaster baseline.

Figure 8: Real-world experimental setup for dynamic grasping.

Method	mIoU $\uparrow$	Time $\downarrow$
CogVideoX	0.0036	180 s
Wan 2.2	0.0225	41 s
Veo 3.1	0.0334	71 s
Kling 3.0	0.0466	30 s
Ours	0.2762	15 ms

Appendix

Table 9: Contact-relevant future prediction quality.

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 309 papers

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies
Oct 3, 2026Zhe Tao, Feiran Wang, Gaowen Liu +2World Models for Robotics Vision-Language-Action Models
University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco
DynaMOMA: Instantaneous Prediction of Grasp Poses for Mobile Manipulation of Dynamic Objects
Jun 24, 2026Zhinan Yu, Junyan Xu, Jiazhao Zhang +9RL for Robotics Robotic Grasping
National University of Defense Technology · CFCS, Peking University · Defense Innovation Institute, Academy of Military Sciences +2
Exp-Force: Experience-Conditioned Pre-Grasp Force Selection with Vision-Language Models
Mar 9, 2026Siqi Shang, Minchao Huang, Bill Fan +1Robotic Grasping VLMs for Robotics
The University of Texas at Austin, TX, USA.

Oct 3, 2026J/K move · Enter open · S save

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

Zhe Tao, Feiran Wang, Gaowen Liu, Ramana Rao Kompella+1

University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco

World Models for Robotics Vision-Language-Action Models Language-Conditioned Robot Manipulation 3D Scene Understanding

Open paper AskPDF arXiv

Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.

Abstract

Robotic Grasping

Explore similar work

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

DynaMOMA: Instantaneous Prediction of Grasp Poses for Mobile Manipulation of Dynamic Objects

Exp-Force: Experience-Conditioned Pre-Grasp Force Selection with Vision-Language Models