Proactive dynamic grasping requires robots to predict the future state at which physical contact will occur, rather than reacting to the currently observed object pose. But what future must the robot predict, and at what level of abstraction? We argue that contact-relevant rigid-body state in SE(3) is a compact task-sufficient abstraction for this problem: it directly determines where and when the robot should establish contact, without requiring visually complete future scenes. This insight leads to a lightweight object-centric forecast-then-select framework that predicts future rigid-body trajectories from short pose histories and uses a reachability-aware interception selector to choose feasible contact states under kinematic and timing constraints. Trained with randomized physics simulation over primitive shapes, the forecaster transfers to real objects once their metric geometry and SE(3) state histories are recovered. Experiments across simulation and real-world dynamic grasping tasks show that our framework improves the average success rate from 24.5% to 62.0% in simulation, and from 15.3% to 67.3% in real-world catching, conveyor, and turntable interception. The SE(3) forecaster runs in 15ms, orders of magnitude faster than dense visual future prediction, enabling real-time contact decision making. Our results suggest that effective proactive grasping does not require imagining the entire future scene. Instead, predicting the compact physical future that determines contact provides an effective and practical abstraction for real-time dynamic grasping.
Figures & tables
Figure 1: Left: Reactive policies act on the current object pose and fail under fast dynamics. Video world models predict pixel-level futures that are slow and physically imprecise. Our method predicts contact-relevant SE(3) state — compact, fast, and action-guided. Right: Real-world dynamic grasping tasks including parabolic catching, conveyor interception, and turntable grasping.
Figure 2: Overview of PAF-Grasp. The Neural SE(3) forecaster is trained on Genesis-generated primitive-object trajectories to learn reusable rigid-body dynamics. At deployment, an anchor RGB-D observation is used offline to recover object-specific geometry and an approximate inertia prior, while the online RGB-D stream continuously updates the object’s SE(3) history for closed-loop forecasting and interception. The forecaster encodes recent motion history, estimates initial velocities, and integrates a Newton–Euler-structured neural ODE to predict future rigid-body trajectories. The forecast-guided interception selector then propagates task-specific contact frames along the predicted trajectory and selects a reachable end-effector target under timing and kinematic constraints.
Figure 3: Real-world success rates across three dynamic grasping tasks and a representative reactive-tracking failure on conveyor interception.
Method
Catch
Conveyor
Turntable
Ramp
Avg. SR ↑
SR ↑
PL ↓
TCT ↓
SR ↑
PL ↓
TCT ↓
SR ↑
PL ↓
TCT ↓
SR ↑
PL ↓
TCT ↓
π0 -FAST [ 43 ]
8
0.16
1.32
16
0.46
2.34
8
0.33
1.96
8
0.25
3.03
10
π0 [ 6 ]
6
0.12
0.56
50
0.44
2.3
16
0.34
2.23
26
0.18
1.99
24.5
π0.5 [ 7 ]
11
0.11
0.55
46.0
0.44
2.36
16
0.33
2.10
24
0.18
2.09
24.3
VLASH [ 9 ]
6
0.11
0.54
32
0.45
2.74
26
0.43
2.4
12
0.25
2.60
19
GR00T N1.7 [ 10 ]
4
0.06
0.47
24
0.47
2.92
12
0.46
2.43
2
0.18
2.01
10.5
Table 1: Overall dynamic grasping performance. Ours uses SE(3) forecasting and reachability-aware contact-time selection. SR is over all trials; PL and TCT are over successful trials only.
Figure 5
Figure 6: Fast-WAM (right) failure: the gripper reaches the cup but misses the rotating handle.
Task
Pred.
Pos-FDE ↓
Rot-FDE ↓
SR ↑
Toss
Classical
0.203
12.99 ∘
36%
NODE
0.202
10.64 ∘
56%
Conveyor
Classical
0.076
10.48 ∘
78%
NODE
0.057
0.55 ∘
78%
Turntable
Classical
0.049
10.60 ∘
54%
NODE
0.059
3.94 ∘
62%
Table 3: Task-aware classical predictors vs. NODE under matched noisy SE(3) inputs and the same selector. Pos-FDE: positional final displacement error; Rot-FDE: final rotation error.
Method
Pos-FDE ↓
Rot-FDE ↓
AR
0.044
12.693
NODE w/o IB
0.023
11.241
NODE w/ IB
0.023
4.745
Table 4: Toss ablation.
Table 9
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Motion type
Observation horizon K
Prediction horizon H
Conveyor
Linear
4
20
Turntable
Rotation
4
20
Ramp sliding
Sliding
4
16
Parabolic catching
Projectile
4
12
Appendix
Table 6: Task-specific observation and prediction horizons.
Component
Setting
ODE solver
Dopri5, rtol =10−5 , atol =10−6
Context / dynamics dim.
32/64
MLP hidden dims.
[64,128,128,64]
Velocity estimator
GRU, hidden dim. 64 , 1 layer, dropout 0.1
Optimization
AdamW, lr 10−4 , wd 10−5 , cosine decay
Batch size / epochs
512/3000
Appendix
Table 7: Training configuration for the Neural SE(3) forecaster.
Component
Setting
Input representation
SE(3) pose history; no RGB/depth/velocity
Transformer dim.
dmodel=128
Attention heads
8
Encoder / decoder layers
4/4
Feedforward dim.
512
Dropout
0.1
Appendix
Table 8: Configuration for the autoregressive Transformer forecaster baseline.
Figure 8: Real-world experimental setup for dynamic grasping.
Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.
Zhe Tao, Feiran Wang, Gaowen Liu +2
University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco
Mobile manipulation is a fundamental robotics task and has advanced rapidly in recent years, enabling robots to navigate, reach, and interact with objects in complex environments. However, mobile manipulation of dynamic objects remains highly challenging, as robots must coordinate the mobile base and arm while adapting to continuously evolving target poses. A key challenge lies in predicting temporally consistent short-horizon grasp trajectories from dynamic observations. In this work, we propose \ours{}, a dynamic mobile manipulation framework that couples instantaneous grasp trajectory prediction with whole-body control policy. Our predictor uses an anchor-based diffusion model to generate temporally consistent short-horizon grasp trajectories conditioned on historical observations. The predicted trajectories are then encoded as compact features and fed to a whole-body reinforcement learning policy, which controls the mobile manipulator for dynamic grasping. We further introduce a anticipation-guided reward that equips the policy with an anticipatory grasping horizon by adaptively shifting the target from the current grasp observation to the instantaneously predicted grasp trajectory. Through extensive experiments in Isaac Gym simulation, we show that our method achieves strong performance in mobile manipulation of dynamic objects across diverse settings and grasping metrics. Furthermore, our predictor and policy demonstrate strong generalizability in real-world experiments.
Zhinan Yu, Junyan Xu, Jiazhao Zhang +9
National University of Defense Technology · CFCS, Peking University · Defense Innovation Institute, Academy of Military Sciences +2
Accurate pre-contact grasp force selection is critical for safe and reliable robotic manipulation. Adaptive controllers regulate force after contact but still require a reasonable initial estimate. Starting a grasp with too little force requires reactive adjustment, while starting a grasp with too high a force risks damaging fragile objects. This trade-off is particularly challenging for compliant grippers, whose contact mechanics are difficult to model analytically. We propose Exp-Force, an experience-conditioned framework that predicts the minimum feasible grasping force from a single RGB image. The method retrieves a small set of relevant prior grasping experiences and conditions a vision-language model on these examples for in-context inference, without analytic contact models or manually designed heuristics. On 129 object instances, ExpForce achieves a best-case MAE of 0.43 N, reducing error by 72% over zero-shot inference. In real-world tests on 30 unseen objects, it improves appropriate force selection rate from 63% to 87%. These results demonstrate that Exp-Force enables reliable and generalizable pre-grasp force selection by leveraging prior interaction experiences. http://expforcesubmission.github.io/Exp-Force-Website/