GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation
Authors: Yuqing Jiang, Zijian Zhang, Weitao Zhou, Jiawei Wang, Junjie He, Lei Yang, Haifang Qing, Si Liu, +3 more
Organizations: University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Tuojing Intelligence · Tsinghua University · University of Science and Technology of China · The Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University · Beihang University · Carnegie Mellon University · The University of Hong Kong
Vision-Language-Action (VLA) policies have advanced language-conditioned robotic manipulation, yet action-imitation objectives provide only weak supervision for metric 3D structure and short-horizon physical evolution. Geometry-enhanced policies mainly improve current-scene grounding, whereas predictive policies often model future dynamics in RGB or latent spaces and may incur substantial deployment cost. GaussianDream demonstrates that training-time current Gaussian reconstruction and future Gaussian prediction provide effective 3D supervision, but its dense VGGT/TGE-based prefix jointly carries state, dynamics, and action-conditioning information. We present \textbf{\methodname}, a compact, policy-native extension that inserts \textbf{World State Tokens} and \textbf{World Prediction Tokens} directly into the VLA backbone. A training-only \textbf{World Representation Head} decodes these tokens into a Current World and coupled Future Prediction over shared Gaussian primitives, while static--dynamic factorization preserves persistent structure and focuses residual motion on interaction-relevant regions. At inference, the head, renderer, auxiliary objectives, and VGGT/TGE pathway are removed, leaving only 20 world tokens without online Gaussian decoding or rollout. \method achieves \textbf{98.6%} on LIBERO and \textbf{87.8%} on LIBERO-Plus, with clear gains under Camera and Layout shifts. Real-robot experiments further improve average success from 29.2% to 52.5% over reproduced π0.5 while maintaining efficient closed-loop control.
Figures & tables
Figure 1: Comparison of robotic policy paradigms. Reactive VLAs map observations and language instructions directly to actions but rely mainly on implicit visual representations. Geometry-enhanced policies improve current-scene spatial grounding, whereas predictive and world-model policies introduce temporal supervision through future modeling. GaussianDream++ combines current-world reconstruction and future prediction within a compact, policy-native Gaussian representation, providing structured 3D supervision during training without requiring runtime world decoding or future rollout.
Figure 2: Overview of GaussianDream++. World State Tokens and World Prediction Tokens are inserted directly into the PaliGemma prefix and contextualized with visual and language tokens. The resulting representation directly conditions the Action Expert. During training, a lightweight World Representation Head decodes the same token representations into the Current World and coupled Future Prediction for dense Gaussian supervision. Future observations are used only to construct supervision targets and never enter the policy forward pass. At deployment, the World Representation Head, Gaussian renderer, and all auxiliary supervision branches are removed, while the world tokens remain in the policy.
(a) Standard LIBERO
Method
Spatial
Object
Goal
Long
Average
Published results
π0 ( Black et al., 2024 )
96.8
98.8
95.8
85.2
94.1
π0.5 ( Physical Intelligence et al., 2025 )
97.8
98.8
97.6
92.4
96.7
GeoPredict ( Qian et al., 2025 )
98.0
98.2
95.7
94.0
96.5
QDepth-VLA ( Li et al., 2025b )
97.6
96.6
95.2
90.0
94.9
Table 1: Main results on LIBERO and LIBERO-Plus. We report task success rate (%). (a) Standard LIBERO results across the four task suites. (b) Zero-shot robustness on LIBERO-Plus. Published results are included for contextual comparison, whereas direct improvement claims are based on the matched GaussianDream-family blocks. LIBERO-Plus Overall follows the official task-level aggregation rather than the unweighted mean of the seven shifts. Best results within each evaluation block are shown in bold .
Figure 3: Real-world tasks and distribution shifts. Bowl-Proximity evaluates fine-grained object-relative placement, whereas Eggplant-to-Pink-Plate evaluates long-horizon manipulation with semantic distractors. Each task is tested under the nominal Standard setting, a Layout shift that changes object-relative configurations, and a Camera shift that relocates the global camera while retaining the same task goal.
Method
Standard
Layout
Camera
Overall
Task 1: Bowl-Proximity
π0.5 reproduced
7/20 (35.0)
4/20 (20.0)
4/20 (20.0)
15/60 (25.0)
GaussianDream++
11/20 (55.0)
10/20 (50.0)
7/20 (35.0)
28/60 (46.7)
Task 2: Eggplant-to-Pink-Plate
π0.5 reproduced
9/20 (45.0)
6/20 (30.0)
5/20 (25.0)
20/60 (33.3)
GaussianDream++
14/20 (70.0)
11/20 (55.0)
10/20 (50.0)
35/60 (58.3)
Table 2: Real-world manipulation results. Each task–condition pair contains 20 trials. Entries report successful trials followed by success rate (%) in parentheses. The Overall column pools the three conditions for each task, while the final group pools both tasks.
Figure 4: Qualitative analysis of GaussianDream++. We show Ground Truth, Current Reconstruction decoded from World State Tokens, and Future Prediction decoded jointly from World State Tokens and World Prediction Tokens. Current Reconstruction recovers the dominant robot configuration, workspace geometry, and object layout, whereas Future Prediction preserves the same coarse scene structure while representing short-horizon changes. The objective is not photorealistic future generation, but an action-relevant physical representation shaped by Gaussian world supervision.
Method
Runtime VGGT/TGE
World Tokens
Runtime Head
Latency
Matched local deployment
π0.5 reproduced
No
0
No
286 ms
GaussianDream++
No
20
No
330 ms
Reported reference
GaussianDream ( Zhang et al., 2026 ) †
Yes
1024
No
531 ms
Table 3: Inference-path and latency comparison. Latency is reported per action chunk. The first block is measured using our matched local deployment pipeline. The GaussianDream value marked by † is reported by its original supplementary evaluation and is included only as a reference because the measurement environment may differ. Runtime Head indicates whether explicit Gaussian decoding or rendering is executed during deployment.
Variant
LIBERO
LIBERO-Plus
Spatial
Object
Goal
Long
Avg.
Camera
Robot
Lang.
Light
BG
Noise
Layout
Overall
π0.5 reproduced
98.8
98.2
98.0
92.4
96.9
73.2
77.9
84.3
96.3
95.5
89.9
87.7
85.5
Structural ablations
World Tokens w/o Gaussian Supervision
98.8
99.0
97.2
95.0
97.5
75.0
73.5
85.0
97.5
96.5
90.1
88.0
86.3
Current World only
99.0
99.2
98.0
95.4
97.9
77.0
73.2
86.2
98.6
97.7
91.5
88.8
86.9
Current + Future, uncoupled
99.0
99.4
98.2
95.8
98.1
78.0
73.0
86.8
98.8
98.0
92.5
89.2
87.2
Table 4: Ablation studies of GaussianDream++. We report task success rate (%) on the four LIBERO suites and the seven LIBERO-Plus distribution shifts. Structural ablations isolate token capacity, Current World supervision, Future Prediction, coupled decoding, and static consistency. Supervision ablations remove one Gaussian-world objective from the full model. LIBERO-Plus Overall follows the official task-level aggregation.
Vision-language-action (VLA) policies have advanced language-conditioned robotic manipulation by transferring semantic priors from pretrained vision-language models to action generation. However, standard action-imitation learning often lacks sufficient modeling of explicit 3D spatial information, dense geometric supervision, and future environment evolution, all critical for precise robotic interaction. To address this, we propose \textbf{GaussianDream}, a feed-forward 3D Gaussian world-model plug-in. Specifically, we introduce learnable GaussianDream Queries in the encoder, enabling the model to capture current-frame 3D spatial structure and short-horizon future evolution. During training, the latent GaussianDream prefix is processed by a static reconstruction head and a future prediction head to produce current 3D Gaussian scene states and future Gaussian evolution states. The current branch is supervised by RGB rendering and depth, while the future branch uses future RGB, depth, and pseudo 3D scene-flow signals. During inference, GaussianDream discards all auxiliary heads and retains only the learned prefix to condition action generation, without test-time Gaussian reconstruction or future prediction. Experimental results demonstrate that GaussianDream achieves state-of-the-art performance across multiple robotic manipulation benchmarks, reaching \textbf{98.4%} on LIBERO, \textbf{54.8%} on RoboCasa Human-50, and \textbf{50.0%} on real-robot tasks. Compared with existing 3D-enhanced VLA methods, GaussianDream achieves strong accuracy while providing higher inference efficiency than video-based world-model approaches.
Zijian Zhang, Yuqing Jiang, Qian Cheng +6
Tuojing Intelligence · University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences +5
Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.
Zhe Tao, Feiran Wang, Gaowen Liu +2
University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco
We present GeomVLA, a Vision-Language-Action (VLA) model that unifies perception, latent scene motion prediction, and action generation within a shared robot-centric 3D coordinate frame. Our approach lifts pretrained VLM features into spatially grounded 3D scene tokens using depth and camera calibration, while retaining the semantic representations learned during VLM pretraining. We further introduce a 3D Scene Trajectory Denoiser, a task-conditioned module that learns a latent representation of how scene points are expected to move in 3D. Rather than executing the predicted trajectory as an open-loop plan, GeomVLA extracts intermediate motion tokens from the trajectory denoiser and uses them to condition a 3D flow-based action denoiser through geometry-aware attention. GeomVLA achieves state-of-the-art performance on CALVIN, competitive performance on LIBERO and RoboTwin2.0, and outperforms strong baselines in real-world manipulation settings without robot-action pretraining. Extensive ablations show that future-motion reasoning alone is insufficient: the primary gains are associated with maintaining geometric consistency among scene representation, motion prediction, and robot actions throughout the perception-to-action pipeline.