GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation

Yuqing Jiang; Zijian Zhang; Weitao Zhou; Jiawei Wang; Junjie He; Lei Yang; Haifang Qing; Si Liu

cs.ROAug 26, 2026

GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation

Authors: Yuqing Jiang, Zijian Zhang, Weitao Zhou, Jiawei Wang, Junjie He, Lei Yang, Haifang Qing, Si Liu, +3 more

Organizations: University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Tuojing Intelligence · Tsinghua University · University of Science and Technology of China · The Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University · Beihang University · Carnegie Mellon University · The University of Hong Kong

Abstract

Vision-Language-Action (VLA) policies have advanced language-conditioned robotic manipulation, yet action-imitation objectives provide only weak supervision for metric 3D structure and short-horizon physical evolution. Geometry-enhanced policies mainly improve current-scene grounding, whereas predictive policies often model future dynamics in RGB or latent spaces and may incur substantial deployment cost. GaussianDream demonstrates that training-time current Gaussian reconstruction and future Gaussian prediction provide effective 3D supervision, but its dense VGGT/TGE-based prefix jointly carries state, dynamics, and action-conditioning information. We present \textbf{\methodname}, a compact, policy-native extension that inserts \textbf{World State Tokens} and \textbf{World Prediction Tokens} directly into the VLA backbone. A training-only \textbf{World Representation Head} decodes these tokens into a Current World and coupled Future Prediction over shared Gaussian primitives, while static--dynamic factorization preserves persistent structure and focuses residual motion on interaction-relevant regions. At inference, the head, renderer, auxiliary objectives, and VGGT/TGE pathway are removed, leaving only 20 world tokens without online Gaussian decoding or rollout. \method achieves \textbf{98.6%} on LIBERO and \textbf{87.8%} on LIBERO-Plus, with clear gains under Camera and Layout shifts. Real-robot experiments further improve average success from 29.2% to 52.5% over reproduced $π_{0.5}$ while maintaining efficient closed-loop control.

Figures & tables

Figure 1: Comparison of robotic policy paradigms. Reactive VLAs map observations and language instructions directly to actions but rely mainly on implicit visual representations. Geometry-enhanced policies improve current-scene spatial grounding, whereas predictive and world-model policies introduce temporal supervision through future modeling. GaussianDream++ combines current-world reconstruction and future prediction within a compact, policy-native Gaussian representation, providing structured 3D supervision during training without requiring runtime world decoding or future rollout.

Figure 2: Overview of GaussianDream++. World State Tokens and World Prediction Tokens are inserted directly into the PaliGemma prefix and contextualized with visual and language tokens. The resulting representation directly conditions the Action Expert. During training, a lightweight World Representation Head decodes the same token representations into the Current World and coupled Future Prediction for dense Gaussian supervision. Future observations are used only to construct supervision targets and never enter the policy forward pass. At deployment, the World Representation Head, Gaussian renderer, and all auxiliary supervision branches are removed, while the world tokens remain in the policy.

(a) Standard LIBERO
Method	Spatial	Object	Goal	Long	Average
Published results
$\pi_{0}$ ( Black et al., 2024 )	96.8	98.8	95.8	85.2	94.1
$\pi_{0.5}$ ( Physical Intelligence et al., 2025 )	97.8	98.8	97.6	92.4	96.7
GeoPredict ( Qian et al., 2025 )	98.0	98.2	95.7	94.0	96.5
QDepth-VLA ( Li et al., 2025b )	97.6	96.6	95.2	90.0	94.9

Table 1: Main results on LIBERO and LIBERO-Plus. We report task success rate (%). (a) Standard LIBERO results across the four task suites. (b) Zero-shot robustness on LIBERO-Plus. Published results are included for contextual comparison, whereas direct improvement claims are based on the matched GaussianDream-family blocks. LIBERO-Plus Overall follows the official task-level aggregation rather than the unweighted mean of the seven shifts. Best results within each evaluation block are shown in bold .

Figure 3: Real-world tasks and distribution shifts. Bowl-Proximity evaluates fine-grained object-relative placement, whereas Eggplant-to-Pink-Plate evaluates long-horizon manipulation with semantic distractors. Each task is tested under the nominal Standard setting, a Layout shift that changes object-relative configurations, and a Camera shift that relocates the global camera while retaining the same task goal.

Method	Standard	Layout	Camera	Overall
Task 1: Bowl-Proximity
$\pi_{0.5}$ reproduced	7/20 (35.0)	4/20 (20.0)	4/20 (20.0)	15/60 (25.0)
GaussianDream++	11/20 (55.0)	10/20 (50.0)	7/20 (35.0)	28/60 (46.7)
Task 2: Eggplant-to-Pink-Plate
$\pi_{0.5}$ reproduced	9/20 (45.0)	6/20 (30.0)	5/20 (25.0)	20/60 (33.3)
GaussianDream++	14/20 (70.0)	11/20 (55.0)	10/20 (50.0)	35/60 (58.3)

Table 2: Real-world manipulation results. Each task–condition pair contains 20 trials. Entries report successful trials followed by success rate (%) in parentheses. The Overall column pools the three conditions for each task, while the final group pools both tasks.

Figure 4: Qualitative analysis of GaussianDream++. We show Ground Truth, Current Reconstruction decoded from World State Tokens, and Future Prediction decoded jointly from World State Tokens and World Prediction Tokens. Current Reconstruction recovers the dominant robot configuration, workspace geometry, and object layout, whereas Future Prediction preserves the same coarse scene structure while representing short-horizon changes. The objective is not photorealistic future generation, but an action-relevant physical representation shaped by Gaussian world supervision.

Method	Runtime VGGT/TGE	World Tokens	Runtime Head	Latency
Matched local deployment
$\pi_{0.5}$ reproduced	No	0	No	286 ms
GaussianDream++	No	20	No	330 ms
Reported reference
GaussianDream ( Zhang et al., 2026 ) †	Yes	1024	No	531 ms

Table 3: Inference-path and latency comparison. Latency is reported per action chunk. The first block is measured using our matched local deployment pipeline. The GaussianDream value marked by

\dagger

is reported by its original supplementary evaluation and is included only as a reference because the measurement environment may differ. Runtime Head indicates whether explicit Gaussian decoding or rendering is executed during deployment.

Variant	LIBERO					LIBERO-Plus
Variant	Spatial	Object	Goal	Long	Avg.	Camera	Robot	Lang.	Light	BG	Noise	Layout	Overall
$\pi_{0.5}$ reproduced	98.8	98.2	98.0	92.4	96.9	73.2	77.9	84.3	96.3	95.5	89.9	87.7	85.5
Structural ablations
World Tokens w/o Gaussian Supervision	98.8	99.0	97.2	95.0	97.5	75.0	73.5	85.0	97.5	96.5	90.1	88.0	86.3
Current World only	99.0	99.2	98.0	95.4	97.9	77.0	73.2	86.2	98.6	97.7	91.5	88.8	86.9
Current + Future, uncoupled	99.0	99.4	98.2	95.8	98.1	78.0	73.0	86.8	98.8	98.0	92.5	89.2	87.2

Table 4: Ablation studies of GaussianDream++. We report task success rate (%) on the four LIBERO suites and the seven LIBERO-Plus distribution shifts. Structural ablations isolate token capacity, Current World supervision, Future Prediction, coupled decoding, and static consistency. Supervision ablations remove one Gaussian-world objective from the full model. LIBERO-Plus Overall follows the official task-level aggregation.

Explore similar work

GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation
May 20, 2026Zijian Zhang, Yuqing Jiang, Qian Cheng +6World Models for Robotics Robotic Manipulation
Tuojing Intelligence · University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences +5
ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies
Oct 3, 2026Zhe Tao, Feiran Wang, Gaowen Liu +2World Models for Robotics Vision-Language-Action Models
University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco
GeomVLA: Unifying Scene, Motion, and Action in 3D
Sep 12, 2026Ziyin Xiong, Nikolaos Gkanatsios, Moritz Reuss +1Robotic Manipulation 3D Scene Representation
Carnegie Mellon University · NVIDIA

May 20, 2026, cs.ROJ/K move · Enter open · S save

GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation

Zijian Zhang, Yuqing Jiang, Qian Cheng, Xiaofan Li+5

Tuojing Intelligence · University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences+5

World Models for Robotics Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-language-action (VLA) policies have advanced language-conditioned robotic manipulation by transferring semantic priors from pretrained vision-language models to action generation. However, standard action-imitation learning often lacks sufficient modeling of explicit 3D spatial information, dense geometric supervision, and future environment evolution, all critical for precise robotic interaction. To address this, we propose \textbf{GaussianDream}, a feed-forward 3D Gaussian world-model plug-in. Specifically, we introduce learnable GaussianDream Queries in the encoder, enabling the model to capture current-frame 3D spatial structure and short-horizon future evolution. During training, the latent GaussianDream prefix is processed by a static reconstruction head and a future prediction head to produce current 3D Gaussian scene states and future Gaussian evolution states. The current branch is supervised by RGB rendering and depth, while the future branch uses future RGB, depth, and pseudo 3D scene-flow signals. During inference, GaussianDream discards all auxiliary heads and retains only the learned prefix to condition action generation, without test-time Gaussian reconstruction or future prediction. Experimental results demonstrate that GaussianDream achieves state-of-the-art performance across multiple robotic manipulation benchmarks, reaching \textbf{98.4%} on LIBERO, \textbf{54.8%} on RoboCasa Human-50, and \textbf{50.0%} on real-robot tasks. Compared with existing 3D-enhanced VLA methods, GaussianDream achieves strong accuracy while providing higher inference efficiency than video-based world-model approaches.

Abstract

Explore similar work

GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

GeomVLA: Unifying Scene, Motion, and Action in 3D