Robotic manipulation policies are advancing rapidly with increasing reliance on vision-language models for end-to-end decision making. However, reliable deployment remains challenging because many policies lack explicit mechanisms for predicting task outcomes and evaluating whether generated actions will achieve desired final states, causing execution errors to accumulate during long-horizon manipulation. We present Robot-GST, a geometry-aware spatio-temporal behaviour representation and evaluation framework that constructs a Gaussian-SAM robotic environment for real-to-sim policy verification and improves the reliability of real-world manipulation deployment. Our approach constructs a high-fidelity robotic environment from RGB-D observations using 3D Gaussian Splatting and SAM3D, enabling ``simulation and evaluation before acting''. It integrates visual observations and language instructions with spatio-temporal reasoning for long-horizon task planning using large vision-language models. To bridge high-level planning and real-world execution, we introduce Gaussian-aware final-state estimation through geometric sampling and state-based trajectory planning. Before execution, candidate action sequences are simulated and evaluated in the Gaussian-SAM environment to filter infeasible behaviours. We validate our approach on representative manipulation tasks involving rigid, soft, and deformable objects, including cube placing, toy packing, and duck rearrangement, demonstrating that geometry-aware spatio-temporal reasoning and state-aware execution improve manipulation reliability across different object categories. Our results suggest that combining geometry-aware reconstruction with high-quality rendering and simulation provides a scalable approach for evaluating robotic manipulation behaviours. Website: https://robot-gst.github.io
Figures & tables
Fig. 2: 3DGS reconstruction of robotic scenes. Multi-view RGB-D observations are processed using four complementary encoders: SAM3D for object-level segmentation, DINOv3 for semantic correspondence, CLIP for vision–language alignment, and depth cues for geometric structure. The resulting features enable high-fidelity 3D scene reconstruction.
Methods
PSNR[db] ↑
SSIM ↑
LPIPS(vgg) ↓
GScream [ 33 ]
17.82
0.590
0.56
VR-GS [ 34 ]
24.13
0.833
0.32
Decoupled GS [ 35 ]
27.32
0.905
0.30
POGS [ 36 ]
19.66
0.869
0.098
D3DGS [ 37 ]
12.05
0.399
0.23
Embodied GS [ 38 ]
14.95
0.389
0.36
TABLE I: Average rendering performance on robotic scene reconstruction . Our method achieves competitive performance across rendering metrics.
Fig. 3: GS reconstruction rendering examples : Our method achieves the best GS reconstruction performance compared with POGS and Embodied GS.
Fig. 4: Geometry-aware spatio-temporal planning and policy evaluation . Top: vision–language-conditioned keypoint-affordance representation and task planning in a 3D coordinate frame; bottom: policy evaluation in the constructed simulator using reconstructed robotic scenes.
Fig. 5: Simulation and real-world experiments for rigid and soft-object manipulation . Rows 1 and 3 show simulation results, while Rows 2 and 4 show real-world results for cube placing and sloth packing, respectively.
Fig. 6: Policy evaluation and replanning . A robot policy (orange) fails (the sloth is not fully packed into the basket), triggering replanning (green), which the simulator evaluates. If successful, the policy is rolled out on the robot.
Without replanning
Replanning
Task
sim
real
sim
real
Place cube into a box
80.0%
80.0%
100.0%
90.0%
Package sloth into a basket
60.0%
40.0%
100.0%
80.0%
Arrange ducks into a basket
60.0%
46.0%
80.0%
70.0%
Average
66.7%
55.3%
93.3%
80.0%
TABLE II: Success rate of tasks with/without replanning in real and simulation environments.
Fig. 7: Left: Correlation of success rate in simulation and real-world experiments ; Right: Success rate of each control step (subtask) measured on the physical robot over 10 trials for toy duck rearrangement.
Fig. 8: Simulation and real-world experiments for duck toy rearrangement . Left → Right shows results for ducks 1–5. “Simulation” and “Real-world” show results from policy simulation and policy rollout on the physical robot.
Robotic manipulation requires reasoning about future spatial-temporal interactions and geometric constraints, yet existing Vision-Language-Action (VLA) policies often leave predictive representation weakly coupled with action execution, causing failures in tasks requiring precise spatial-temporal coordination. We propose STARRY, a world-model-enhanced action-generation policy that aligns spatial-temporal prediction and action generation by jointly denoising future spatial-temporal latents and actions through a unified diffusion process. To bridge 2D visual tokens and 3D metric control, STARRY introduces Geometry-Aware Selective Attention Modulation (GASAM), which converts predicted depth and end-effector geometry into token-aligned weights for selective action-attention modulation. On RoboTwin 2.0, STARRY achieves 93.82% / 93.30% average success under Clean and Randomized settings across 50 bimanual tasks. Real-world experiments show that STARRY improves average success from 42.5% to 70.8% compared with π0.5. These results demonstrate the effectiveness of action-centric spatial-temporal world modeling for spatially and temporally demanding robotic manipulation.
Yuxuan Tian, Yurun Jin, Bin Yu +5
Beijing Institute of Technology · Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence +4
We introduce Forecast-aware Gaussian Splatting (Forecast-GS), a predictive 3D representation framework for language-conditioned robotic manipulation. While recent manipulation systems have made progress by grounding language instructions into robot affordances, value maps, or relational keypoint constraints, they usually reason over the current scene and do not explicitly model the task-completed state. This limitation is critical when success depends on satisfying spatial and semantic goals under partial observations, where the robot must evaluate whether a candidate action leads to a feasible task-consistent outcome. We validate Forecast-GS on real-world pick-and-place manipulation tasks, including Cutter-to-Box, Apple-to-Bowl, and Sponge-to-Tray. For each task, we conduct 25 real-world trials under varied initial object configurations using the same robot platform and sensing setup. Forecast-GS with automatic candidate selection achieves success rates of 21/25, 23/25, and 16/25 on the three tasks, respectively, outperforming the ReKep baseline, which achieves 15/25, 19/25, and 10/25. A diagnostic human-assisted setting further improves success rates to 23/25, 24/25, and 19/25, suggesting that candidate generation is effective while automatic ranking remains imperfect. These results suggest that explicitly forecasting task-completed 3D states enables more reliable action evaluation, while the gap between automatic and human-assisted selection indicates that robust final-state ranking remains an important challenge for fully autonomous manipulation. Overall, Forecast-GS provides an interpretable bridge between language understanding, 3D perception, and robotic manipulation planning.
Kaixin Jia, Jiacheng Xu
KTH Royal Institute of Technology, Stockholm, Sweden.
Manipulation policies must know which objects matter and where they are, yet the pretrained backbones that current robot foundation models build on, from language in vision-language-action models (VLAs) to video generation in world-action models (WAMs), do not directly require this metric grounding, leaving it to be learned implicitly from robot demonstrations. We propose Grounded Action Models (GAMs), a new paradigm of robot foundation models built with 3D grounding. GAM can be conditioned using language, points, or box prompts, which are first transformed into a shared object-centric representation of the selected objects. This representation captures target-focused visual features and metric object geometry, which is mixed with robot state history through a multi-stream transformer to predict action chunks. Although GAMs can be run autonomously, they can also serve as a low-level controller that a high-level planner controls using its various input modalities, allowing for long-horizon and memory-dependent manipulation. On RoboTwin 2.0, GAM achieves an average success rate of 55.3% across 50 tasks (vs. 52.0% for Spatial Forcing), including 47.6% under scene randomization (vs. 30.4% for Abot-M0), with its action policy trained only on clean-scene demonstrations. On LIBERO-PRO, it achieves a state-of-the-art average success rate of 61% (vs. 53% for π0.5) across 16 perturbation settings, with the largest gains when targets are relocated or newly designated. On two real robots, GAM retains 17/20 successes under visual shift on a bimanual YAM versus 4/20 for π0.5, while its composition with a Molmo2 planner on a Franka achieves 64.7% ID and 49.8% OOD step completion on long-horizon and memory-dependent tasks.
Gehao Zhang, Weikai Huang, Shailesh Shailesh +3
Northwestern University · University of Washington · National University of Singapore +1