CoGaussian: Open-Vocabulary Gaussian-based Collaborative 3D Scene Understanding

Om Shivam Verma; Shaya Farahmand; Lingjun Zhao; Lu Gan

cs.CVOct 20, 2026

CoGaussian: Open-Vocabulary Gaussian-based Collaborative 3D Scene Understanding

Authors: Om Shivam Verma, Shaya Farahmand, Lingjun Zhao, Lu Gan

Organizations: Lunar Lab, Georgia Institute of Technology, Atlanta.

Abstract

Collaborative perception enables agents to retrieve the scene semantics and geometry beyond the field of view of any individual agent. This broader scene understanding enhances transportation safety and improves efficiency in multi-robot systems. However, existing collaborative semantic occupancy methods rely on dense 3D voxel annotations and are limited to recognizing objects within a predefined set of semantic classes. To overcome these limitations, we present CoGaussian, an open-vocabulary Gaussian-based collaborative semantic occupancy prediction framework. CoGaussian represents a scene as a set of foundation model-aligned sparse 3D Gaussians, which serve as a compact representation for communicating and fusing geometric and semantic information across multiple agents. On the Semantic-OPV2V benchmark, our method outperforms existing open-vocabulary single-agent approaches, highlighting the effectiveness of open-vocabulary collaborative scene understanding.

Figures & tables

Fig. 1: Qualitative comparison of three variants of CoGaussian with ground truth. The ego vehicle is marked as E. Another agent is in the front-right (not visible here), partially completing the front-right of the ego vehicle in collaborative predictions. Class legends are shown in Table I .

Fig. 2: Overview. CoGaussian starts with randomly initialized embeddings. A transformer layer uses multiscale features of images to update the embeddings, followed by a Head Layer that predicts Gaussians. During Joint Training, all agents predict ego Gaussians which are communicated with each other and aggregated. This set of Gaussians is used for training. In the Fusion pipeline, the neighbors predict and send Gaussian embeddings to the ego agent. Radius-gated Attention uses these neighbor Gaussian embeddings to update ego embeddings. The Fusion Head predicts ego Gaussians, and pre-trained Head from Joint Training predicts neighbor Gaussians. These are aggregated to form the final prediction used for both training and inference.

3D Scene Understanding

3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.

See all 295 papers

Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images
Jul 2, 2026Hu Zhu, Bohan Li, Xianda Guo +53D Gaussian Splatting Open-Vocabulary 3D Segmentation
Department of Computing, The Hong Kong Polytechnic University, Hong Kong, SAR, China · Eastern Institute of Technology, Ningbo, China · Shanghai Jiao Tong University, Shanghai, China +4
OpenGaFF: Open-Vocabulary Gaussian Feature Field with Codebook Attention
May 7, 2026Kunyi Li, Michael Niemeyer, Sen Wang +33D Gaussian Splatting Open-Vocabulary 3D Segmentation
1Technical University of Munich · 3Munich Center for Machine Learning · 2Google +1
Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes
Feb 26, 2026Changqing Zhou, Yueru Luo, Han Zhang +23D Occupancy Prediction Semantic Occupancy Mapping
The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen

Jul 2, 2026J/K move · Enter open · S save

Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images

Hu Zhu, Bohan Li, Xianda Guo, Yanlun Peng+4

Department of Computing, The Hong Kong Polytechnic University, Hong Kong, SAR, China · Eastern Institute of Technology, Ningbo, China · Shanghai Jiao Tong University, Shanghai, China+4

3D Gaussian Splatting Open-Vocabulary 3D Segmentation Sparse-View 3D Reconstruction Semantic Occupancy Mapping 3D Occupancy Prediction 3D Scene Understanding

Open paper AskPDF arXiv

Comprehensive 3D scene understanding from sparse, unposed images requires a model to recover renderable geometry, open-vocabulary semantics, and free/occupied 3D space without relying on external camera calibration. Recent feed-forward Gaussian methods improve pose-free reconstruction and semantic rendering, but their Gaussian primitives are mainly optimized through image-space objectives and remain weakly constrained in unobserved regions. We propose COVScene, a pose-free semantic Gaussian framework that couples renderable Gaussian primitives with a dense semantic occupancy field through differentiable volumetric lifting. Instead of converting Gaussians to voxels only at evaluation time, COVScene lifts the predicted semantic Gaussians inside the training computation graph, so volumetric regularization provides gradients to Gaussian opacity, geometry, and semantic features. The framework combines a semantic-aware Geometry Transformer, multi-task Gaussian decoding, geometric foundation distillation, and occupancy entropy regularization to support novel view synthesis, open-vocabulary semantic querying, and semantic occupancy prediction within a single representation. Experiments on ScanNet and ScanNet++ show that COVScene maintains competitive rendering quality, improves open-vocabulary segmentation, and achieves stronger semantic occupancy prediction than the self-supervised baseline without direct voxel-level supervision.

Abstract

3D Scene Understanding

Explore similar work

Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images

OpenGaFF: Open-Vocabulary Gaussian Feature Field with Codebook Attention

Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes