cs.ROAug 25, 2026

GaussianWAM: Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models

Authors: Zijian Zhang, Yuqing Jiang, Weitao Zhou, Minglei Li, Jinhao Zhang, Yao Mu, Xiaofan Li, Hao Zhao, 

Organizations: University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Tuojing Intelligence · Tsinghua University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

World-Action Models (WAMs) jointly learn future visual prediction and action generation, using video dynamics as a representation-learning signal for robotic manipulation. However, their video latents are primarily optimized for visual prediction and are not explicitly encouraged to preserve cross-view geometric structure or spatially localized, object-relevant semantics. We propose GaussianWAM, a training-time representation-enhancement framework that organizes geometric and semantic supervision through a 3D Gaussian field. Given synchronized multi-view observations, frozen geometry and vision foundation models provide depth, camera parameters, and dense semantic features. GaussianWAM binds these heterogeneous signals to shared Gaussian primitives and renders spatially aligned semantic, depth, and coverage targets, which are distilled into the current-observation representations of the WAM. All teacher models, Gaussian components, and auxiliary prediction heads are removed after training, leaving the original WAM inference path without additional modules or forward computation. On LIBERO-Plus, GaussianWAM improves FastWAM from 52.05% to 71.29% and Cosmos Policy from 71.52% to 77.30%. Direct CLIP and VGGT distillation already establishes a strong FastWAM baseline of 69.37%, while Gaussian-field unification further improves it to 71.29%, supporting the benefit of spatially organizing heterogeneous teacher signals. GaussianWAM also improves performance on standard LIBERO and shows positive transfer trends on RoboTwin and real-world manipulation. These results suggest that training-time Gaussian distillation provides a practical way to inject geometry- and semantics-related supervision into WAM representations without changing their deployment architecture.

Figures & tables

Figure 1: Comparison of WAM enhancement paradigms. (a) Conventional WAMs learn world-action representations primarily in 2D video or latent space. (b) Previous geometry- or semantics-aware WAMs introduce dedicated spatial enhancement modules or structured representations into the modeling pipeline. (c) GaussianWAM instead uses a training-time 3D Gaussian teacher to inject geometry-, semantics-, and coverage-aware supervision into the original WAM representations, without modifying the inference backbone.
Figure 2: Overview of GaussianWAM. (a) A 3D Gaussian teacher is constructed from synchronized multi-view observations and rendered into semantic, depth, and alpha targets, together with a validity mask, which are cached for training. (b) In FastWAM-style dual-expert MoT models, Gaussian supervision is applied to current-observation video representations. (c) In Cosmos-Policy-style unified Transformers, the same supervision is applied to current-observation visual tokens in the shared backbone. (d) Lightweight semantic, depth, and alpha heads distill the cached Gaussian targets into WAM visual representations. All Gaussian-related modules are removed at inference, preserving the original WAM forward path and computational cost.
MethodTypeEmb. PT.LIBEROCameraRobotLang.LightBGNoiseLayoutOverall
Vision-Language-Action Models
UniVLA ( Bu et al., 2025 )VLAYes95.201.8046.2069.6069.0090.7021.2031.9043.90
π0\pi_{0} ( Black et al., 2024 )VLAYes94.2013.806.0058.8085.0090.7079.0068.9054.60
π0\pi_{0} -FAST ( Pertsch et al., 2025 )VLAYes85.5065.1021.6061.0073.2097.7074.4068.8064.20
OpenVLA-OFT ( Kim et al., 2025 )VLAYes97.1056.4031.9079.5088.7097.3075.8074.2070.00
X-VLA ( Zheng et al., 2025 )VLAYes98.1023.4089.7075.7088.2096.0062.7071.8070.46
Table 1: Main results on LIBERO and LIBERO-Plus. We report success rate (%). All models are trained on LIBERO and evaluated on both LIBERO and LIBERO-Plus, where LIBERO-Plus serves as a zero-shot evaluation under distribution shifts. Emb. PT. indicates large-scale embodied pretraining. Our FastWAM baseline and its GaussianWAM variant are trained for 70k optimization steps, while the Cosmos Policy baseline and its GaussianWAM variant are trained for 5k iterations.
MethodCleanRandom
DP34.070.33
ACT34.204.00
DP359.873.80
FastWAM70.500.70
FastWAM + GaussianWAM70.501.60
Cosmos Policy34.407.10
Table 2: Results on the 15-task RoboTwin 2.0 Clean-to-Random benchmark. Clean and Random denote the clean and randomized evaluation settings, respectively. We report average success rate (%).
Figure 3: Real-world evaluation and success rates. The left panel shows two bimanual tasks under standard, layout, camera, and visual-noise variations. The right panel shows success rates for π0.5\pi_{0.5} , FastWAM, and GaussianWAM under the four settings. Each task uses 100 demonstrations and 20 evaluation trials per setting.
Table 6
Figure 7
More in this topic

World Action Models

World Action Models connect predictions of how scenes change under action to executable policies for robots, vehicles, and other embodied agents.

See all 223 papers

Explore similar work

CardsList
Aug 26, 2026

GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation

Vision-Language-Action (VLA) policies have advanced language-conditioned robotic manipulation, yet action-imitation objectives provide only weak supervision for metric 3D structure and short-horizon physical evolution. Geometry-enhanced policies mainly improve current-scene grounding, whereas predictive policies often model future dynamics in RGB or latent spaces and may incur substantial deployment cost. GaussianDream demonstrates that training-time current Gaussian reconstruction and future Gaussian prediction provide effective 3D supervision, but its dense VGGT/TGE-based prefix jointly carries state, dynamics, and action-conditioning information. We present \methodname, a compact, policy-native extension that inserts World State Tokens and World Prediction Tokens directly into the VLA backbone. A training-only World Representation Head decodes these tokens into a Current World and coupled Future Prediction over shared Gaussian primitives, while static--dynamic factorization preserves persistent structure and focuses residual motion on interaction-relevant regions. At inference, the head, renderer, auxiliary objectives, and VGGT/TGE pathway are removed, leaving only 20 world tokens without online Gaussian decoding or rollout. \method achieves 98.6% on LIBERO and 87.8% on LIBERO-Plus, with clear gains under Camera and Layout shifts. Real-robot experiments further improve average success from 29.2% to 52.5% over reproduced π0.5π_{0.5} while maintaining efficient closed-loop control.

Yuqing Jiang, Zijian Zhang, Weitao Zhou +8

University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Tuojing Intelligence +7

World Models for RoboticsRobotic ManipulationVision-Language-Action ModelsEfficient VLA Models3D Gaussian Splatting
AskPDF
May 20, 2026

GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation

Vision-language-action (VLA) policies have advanced language-conditioned robotic manipulation by transferring semantic priors from pretrained vision-language models to action generation. However, standard action-imitation learning often lacks sufficient modeling of explicit 3D spatial information, dense geometric supervision, and future environment evolution, all critical for precise robotic interaction. To address this, we propose GaussianDream, a feed-forward 3D Gaussian world-model plug-in. Specifically, we introduce learnable GaussianDream Queries in the encoder, enabling the model to capture current-frame 3D spatial structure and short-horizon future evolution. During training, the latent GaussianDream prefix is processed by a static reconstruction head and a future prediction head to produce current 3D Gaussian scene states and future Gaussian evolution states. The current branch is supervised by RGB rendering and depth, while the future branch uses future RGB, depth, and pseudo 3D scene-flow signals. During inference, GaussianDream discards all auxiliary heads and retains only the learned prefix to condition action generation, without test-time Gaussian reconstruction or future prediction. Experimental results demonstrate that GaussianDream achieves state-of-the-art performance across multiple robotic manipulation benchmarks, reaching 98.4% on LIBERO, 54.8% on RoboCasa Human-50, and 50.0% on real-robot tasks. Compared with existing 3D-enhanced VLA methods, GaussianDream achieves strong accuracy while providing higher inference efficiency than video-based world-model approaches.

Zijian Zhang, Yuqing Jiang, Qian Cheng +6

Tuojing Intelligence · University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences +5

World Models for RoboticsRobotic ManipulationVision-Language-Action Models
AskPDF
Oct 27, 2026

GAWM: A Compact Latent World-Action Model for Multi-Task Robot Manipulation

World-action models (WAMs) incorporate the prediction of future observations into robot policy learning. However, most WAMs build on billion-parameter vision-language or video-generation backbones, which incur high training cost and inference latency. In this letter, we present GAWM (Geometry-Aware World Model), a compact multi-task WAM that predicts future observations in the latent space of a frozen DINOv3 encoder. GAWM compresses the features of each camera view into a small set of tokens. A lightweight predictor, conditioned on the task rather than on actions, forecasts these tokens at two future horizons. The forecast is supervised with the dense patch features of future frames, which requires it to encode the spatial layout of the future scene. An action decoder directly regresses a chunk of future actions from the current and predicted tokens. With fewer than 100M parameters and no robot or video pretraining, GAWM outperforms billion-parameter WAMs on RoboTwin 2.0 and performs comparably to billion-parameter VLAs on LIBERO. On a dual-arm robot, it achieves lower action-prediction error on held-out demonstrations than a fine-tuned billion-parameter VLA, with substantially lower inference latency. Ablation studies show that the latent world model substantially improves performance when training data or training time is limited. Code is available at https://github.com/gaoxiang12/ga-wm.

Yang Liu, Xiaoli Liu, Liuzhao Li +3

Genisom AI, Beijing, China

World Action ModelsLatent World ModelsMulti-Task Robotic ManipulationEfficient Inference for World Action ModelsData-Efficient Robot Learning
AskPDF