EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding

Seungjun Lee; Zihan Wang; Yunsong Wang; Gim Hee Lee

cs.CVMar 4, 2026

EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding

Authors: Seungjun Lee, Zihan Wang, Yunsong Wang, Gim Hee Lee

Organizations: National University of Singapore

Abstract

Understanding a 3D scene immediately with its exploration is essential for embodied tasks, where an agent must construct and comprehend the 3D scene in an online and nearly real-time manner. In this study, we propose EmbodiedSplat, an online feed-forward 3DGS for open-vocabulary scene understanding that enables simultaneous online 3D reconstruction and 3D semantic understanding from the streaming images. Unlike existing open-vocabulary 3DGS methods which are typically restricted to either offline or per-scene optimization setting, our objectives are two-fold: 1) Reconstructs the semantic-embedded 3DGS of the entire scene from over 300 streaming images in an online manner. 2) Highly generalizable to novel scenes with feed-forward design and supports nearly real-time 3D semantic reconstruction when combined with real-time 2D models. To achieve these objectives, we propose an Online Sparse Coefficients Field with a CLIP Global Codebook where it binds the 2D CLIP embeddings to each 3D Gaussian while minimizing memory consumption and preserving the full semantic generalizability of CLIP. Furthermore, we generate 3D geometric-aware CLIP features by aggregating the partial point cloud of 3DGS through 3D U-Net to compensate the 3D geometric prior to 2D-oriented language embeddings. Extensive experiments on diverse indoor datasets, including ScanNet, ScanNet++, and Replica, demonstrate both the effectiveness and efficiency of our method. Check out our project page in https://0nandon.github.io/EmbodiedSplat/.

Figures & tables

Figure 2: Overall framework of our EmbodiedSplat. We endow feed-forward 3DGS with semantic understanding capabilities by binding the two types of CLIP features: 1) 2D Semantic Features are attached to each Gaussian via Sparse Coefficient Field with CLIP Global Codebook , effectively reducing memory consumption while preserving semantic generalizability of CLIP. 2) 3D Geometric-aware Features are produced by aggregating the feature point cloud of 3DGS through 3D U-Net [ 10 ] and temporal-aware memory adapter [ 75 ] . These two types of features enable mutual compensation between semantic and 3D geometry, which results in superior understanding capabilities compared to the existing baselines.

Method	Search Domain	ScanNet [ 12 ]						ScanNet200 [ 56 ]		ScanNet++ [ 77 ]		Scene-reconstruction Time (363 images)	Per-Scene / Generalizable	on / off
		10 classes		15 classes		19 classes		70 classes		20 classes
		mIoU	mACC	mIoU	mACC	mIoU	mACC	mIoU	mACC	mIoU	mACC
LangSplat [ 53 ]	2D	6.52	20.11	3.25	13.16	1.34	7.44	0.72	4.39	2.21	10.18	$\sim$ 6 hr	Per-Scene	Offline
LEGaussians [ 59 ]		6.79	18.13	4.13	15.49	2.53	5.67	1.39	5.45	2.93	9.34	$\sim$ 6 hr		Offline
Online-LangSplat [ 31 ]		7.13	21.56	3.89	14.52	3.45	8.97	2.45	4.12	4.51	11.34	5.4 min (1.12 FPS)		Online
OpenGaussian [ 72 ]	3D	29.50	44.61	23.74	39.14	22.52	35.02	15.15	25.66	25.65	37.03	$\sim$ 2.5 hr	Per-Scene	Offline

Table 1: Quantitative comparisons on 3D Semantic Segmentation across ScanNet [ 12 ] , ScanNet200 [ 56 ] and ScanNet++ [ 77 ] . We compare the performance of our EmbodiedSplat with existing semantic 3DGS methods on 3D semantic segmentation. Our EmbodiedSplat achieves the best performance across all benchmarks while maintaining the shortest reconstruction time. The best results are in bold while the second best results are underscored .

Figure 3: Quantitative comparisons on 3D semantic segmentation.

Method	ScanNet++ [ 77 ] ➜ ScanNet [ 12 ]		ScanNet [ 12 ] ➜ ScanNet++ [ 77 ]		ScanNet [ 12 ] ➜ Replica [ 60 ]
	19 classes		20 classes		48 classes
	mIoU	mACC	mIoU	mACC	mIoU	mACC
OpenGaussian [ 72 ]	22.52	35.02	25.65	37.03	11.59	18.12
Occam’s LGS [ 9 ]	30.49	57.91	34.08	61.19	16.19	30.14
Dr. Splat [ 30 ]	28.38	55.85	39.85	58.34	14.47	27.12
InstanceGaussian [ 43 ]	26.57	48.63	29.98	47.47	12.45	18.34

Table 2: Quantitative comparisons on cross-domain 3D semantic segmentation across ScanNet [ 12 ] , ScanNet++ [ 77 ] and Replica [ 60 ] .

2D Features $\mathbf{s}^{T}_{g}$	3D Features $\hat{\mathbf{g}}^{T}_{g}$	ScanNet [ 12 ]			ScanNet200 [ 56 ]	ScanNet++ [ 77 ]
		10 classes	15 classes	19 classes	70 classes	20 classes
		mIoU
✓	✘	48.93	48.38	45.09	29.56	40.86
✘	✓	49.24	48.78	45.39	30.36	41.36
✓	✓	49.81	49.23	46.22	31.16	41.93

Table 3: Ablations on 3D CLIP features

\hat{\mathbf{g}}^{T}_{g}

Cosine similarity	Time (ms)	Complexity	Note
Per-Gaussian	14.35	$O(MD)$	$M$ = 3.2M, $K$ = 8.7K, $D$ = 768, $L$ = 6
Codebook-based ( Ours )	1.18	$O(KD+M(L-1))$	$M$ = 3.2M, $K$ = 8.7K, $D$ = 768, $L$ = 6

Table 4: Ablations on codebook-based cosine similarity

Methods	Type / Feature dimension	Size (MB)	pretraining	information loss
LangSplat [ 53 ]	Auto-encoder / 3	30	✓	✓
Dr. Splat [ 30 ]	PQ Index / 130	173	✓	✓
Occam’s LGS [ 9 ]	✘ / 512	2295	✘	✘
EmbodiedSplat	CLIP Global Codebook / 10	148	✘	✘

Table 5: Comparisons on memory size for semantic features.

Dataset	Classes	Metric	$L=2$	$L=4$	$L=6$	$L=11$
ScanNet [ 12 ]	19 classes	mIoU	44.38	45.01	45.09	45.08

Table 6: Ablations on cache size

L

Figure 5: Online 3D reasoning for class “ Bed ”.

Dataset	Scene ID	Num of multi-view images	Num of keyframes
ScanNet [ 12 ]	scene0000_01	5920	363
	scene0046_00	2480	280
	scene0079_00	1196	119
	scene0158_00	1920	83
	scene0316_00	770	45
	scene0389_00	1415	205

Table 7: Configurations of the three benchmarks [ 12 , 77 , 60 ] .

Figure 6: Depth visualizations with RGB-D inputs.

Figure 7: Toy example of sparse coefficient field initialization at $i$ th pixel. 1) We first add index to each instance-level mask. Index starts from 47 since the latest index in global codebook from previous time step is 46. 2) Instance-level features extracted from current frame $\mathbf{I}^{t}$ are appended into codebook with attached indices, producing updated codebook $\mathbf{C}^{t}$ . 3) We initialize the index cache $\mathbf{I}^{t}_{l}(i)$ and weight cache $\Omega^{t}_{l}(i)$ of $i$ -th local Gaussian aligned with $i$ -th pixel. Since $i$ -th pixel belongs to instance ‘53’, index ‘53’ is added to first entry of $\mathbf{I}^{t}_{l}(i)$ : $\mathbf{I}^{t}_{l}(i,0)=53$ . Corresponding confidence value $\omega^{t}_{l}(i)$ is further inserted into first entry of $\Omega^{t}_{l}(i)$ .

Figure 8: Multi-view inconsistent masks from FastSam [ 83 ] .

2D VLM Configurations	Contextual Information	Speed
SAM [ 34 , 53 ] + CLIP [ 54 ]	✘	23220 ms
FastSAM [ 83 ] + CLIP [ 54 ]	✘	31.5 ms
FastSAM [ 83 ] + OpenSeg [ 22 ] ( Ours )	✓	991.3 ms
FastSAM [ 83 ] + Mask-Adapter [ 45 ] ( Ours )	✓	43.3 ms

Table 8: Comparisons on different 2D VLM configurations.

Method	Venue	Generalizable	Online	Whole-Scene
VoteSplat [ 28 ]	ICCV’25	✘	✘	✓
LUDVIG [ 47 ]	ICCV’25	✘	✘	✓
CF 3 [ 38 ]	ICCV’25	✘	✘	✓
LSM [ 19 ]	NeurIPS’24	✓	✘	✘
OVGaussian [ 7 ]	arXiv’24	✓	✘	✘
SLGaussian [ 6 ]	arXiv’24	✓	✘	✘

Table 9: Comparison between EmbodiedSplat and additional baselines.

Method	Inputs	ScanNet [ 12 ]						ScanNet200 [ 56 ]		Online / Offline
		10 classes		15 classes		19 classes		70 classes
		mIoU	mACC	mIoU	mACC	mIoU	mACC	mIoU	mACC
Occam’s LGS [ 9 ]	RGB	42.14	70.28	35.04	63.71	30.49	57.91	20.32	40.49	Offline
Dr. Splat [ 30 ]	RGB	39.21	66.66	31.84	60.58	28.38	55.85	19.29	33.84	Offline
LUDVIG [ 47 ]	RGB	41.11	68.34	33.73	62.90	29.34	56.98	21.23	39.87	Offline
CF 3 [ 38 ]	RGB	38.14	65.13	30.13	59.13	26.34	52.43	18.23	30.11	Offline

Table 10: Additional comparisons on 3D Semantic Segmentation in ScanNet [ 12 ] and ScanNet200 [ 56 ] datasets.

Method	Search Domain	ScanNet [ 12 ]
		10 classes		15 classes		19 classes
		mIoU	mACC	mIoU	mACC	mIoU	mACC
LangSplat [ 53 ]	2D	45.83	73.12	42.89	69.34	44.15	70.45
Occam’s LGS [ 30 ]	3D	41.13	73.34	40.21	66.82	39.11	64.34
Dr. Splat [ 30 ]	3D	40.11	71.62	38.45	66.11	39.67	65.72
EmbodiedSplat	3D	47.44	76.95	44.11	70.12	43.75	68.16

Table 11: Quantitative results on 2D-rendered semantic segmentation in ScanNet [ 12 ] dataset.

Method	iPSNR $\uparrow$	ePSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\delta<1.1\uparrow$	Type
pixelSplat [ 5 ]	15.54	13.47	0.557	0.608	0.023	Offline
MVSplat [ 8 ]	16.51	13.67	0.591	0.541	0.323	Offline
PixelGaussian [ 20 ]	16.33	13.40	0.601	0.549	0.282	Offline
FreeSplat++ [ 66 ]	23.29	19.44	0.771	0.320	0.904	Offline
EmbodiedSplat	22.78	19.14	0.738	0.367	0.885	Online

Table 12: Whole Scene Reconstruction resutls on ScanNet [ 12 ] . iPSNR and ePSNR detnoes PSNR on the interpolated views and extrapolated views, respectively.

Method	iPSNR $\uparrow$	ePSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\delta<1.1\uparrow$	Type
pixelSplat [ 5 ]	10.70	10.37	0.497	0.663	0.000	Offline
MVSplat [ 8 ]	11.10	10.62	0.497	0.648	0.028	Offline
PixelGaussian [ 20 ]	10.78	10.44	0.529	0.639	0.012	Offline
FreeSplat++ [ 66 ]	22.63	19.51	0.829	0.261	0.890	Offline
EmbodiedSplat	21.54	18.62	0.780	0.330	0.925	Online

Table 13: Whole Scene Reconstruction resutls on ScanNet++ [ 77 ] . iPSNR and ePSNR detnoes PSNR on the interpolated views and extrapolated views, respectively.

Scene	Gaussians Num	Codebook Size	Total Size (MB)	Compression Ratio
scene0000_01	3.2M	8.7K	148	$\times$ 63 efficient
scene0046_00	2.4M	5.3K	106	$\times$ 65 efficient
scene0079_00	1.5M	2.8K	64	$\times$ 67 efficient
scene0158_00	1.1M	1.8K	48	$\times$ 68 efficient
scene0316_00	0.6M	0.6K	23	$\times$ 70 efficient
scene0389_00	1.9M	2.8K	82	$\times$ 69 efficient

Table 14: Ablations on memory efficiency of sparse coefficient field in ScanNet [ 12 ] dataset.

Figure 9: Toy example of online fusion algorithm with sparse coefficient field. 1) Sparse coefficient field of paired Gaussians $(i,m_{i})$ are fused by our online fusion Algorithm. 1 . The local Gaussians ( Green ) which do not have valid match with global Gaussians are just appended to the global set without update. 2) Line 1 of Algorithm. 1 : First entry of local index, $\mathbf{I}^{t}_{i}(i,0)$ is inserted to the last entry of global index cache $\mathbf{I}^{t-1}_{g}(m_{i},-1)$ . In the above example, index 53 is appended to $\mathbf{I}^{t-1}_{g}(m_{i})$ , such that $\mathbf{I}^{t-1}_{g}(m_{i},-1)\leftarrow 53$ . 3-4) Lines 3-4 of Algorithm. 1 : Both the local weight cache $\Omega^{t}_{l}(i)$ and global weight cache $\Omega^{t-1}_{g}(m_{i})$ are updated based on the confidence-weighted average. Specifically, all of the entries in $\Omega^{t-1}_{g}(m_{i})$ are multiplied by 0.75, while the first entry of local weight cache $\Omega^{t}_{l}(i,0)$ is scaled by 0.25. Scaled local weight value $0.25\cdot\Omega^{t}_{l}(i,0)$ is then inserted to the last entry of global weight cache, such that: $\Omega^{t-1}_{g}(m_{i},-1)\leftarrow 0.25\cdot\Omega^{t}_{l}(i,0)$ . Since both the index cache and weight cache of local Gaussian $\bar{\Theta}^{t}_{l}(i)$ are incorporated into global gaussian $\bar{\Theta}^{t-1}_{g}(m_{i})$ from the previous stages, $\bar{\Theta}^{t-1}_{g}(m_{i})$ becomes new $m_{i}$ -th global Gaussian at step $t$ : $\bar{\Theta}^{t}_{g}(m_{i})\leftarrow\bar{\Theta}^{t-1}_{g}(m_{i})$ . $i$ -th local Gaussian is simply discarded. 5-6) Lines 5-6 of Algorithm. 1 : We sort both the global weight cache and index cache based on the weight values $\Omega^{t}_{g}(m_{i})$ in descending order. Then, we keep first $L-1=5$ entires and discard the last values by overwriting them with zero: $\Omega^{t}_{g}(m_{i},-1)\leftarrow 0,\;\mathbf{I}^{t}_{g}(m_{i},-1)\leftarrow 0$ . This keeps each cache size fixed at $L$ along the exploration, effectively improving the memory efficiency.

Panel of Figure 10 — Figure 10: More qualitative comparisons on 3D semantic segmentaiton - (1)

Panel of Figure 11 — Figure 11: More qualitative comparisons on 3D semantic segmentaiton - (2)

Panel of Figure 12 — Figure 12: Qualitative results on 2D-rendered object search of our EmbodiedSplat.

Panel of Figure 13 — Figure 13: Qualitative results on novel-view synthesis and depth rendering of our EmbodiedSplat.

Explore similar work

Aug 7, 2026

InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding

Feed-forward 3D Gaussian Splatting (3DGS) enables efficient and generalizable 3D reconstruction, but current feed-forward 3DGS methods for scene understanding remain largely category-oriented. In contrast, instance-aware 3DGS methods typically rely on per-scene optimization and often decouple reconstruction from instance and semantic learning, limiting reciprocal interactions among them. We present InstanceSplat, a unified feed-forward 3DGS framework for generalizable 3D reconstruction and instance-aware scene understanding from pose-free multi-view images. In a single forward pass, InstanceSplat constructs an instance-aware Gaussian representation that jointly encodes appearance, geometry, instance identity, and language-aligned semantics. Shared 3D Gaussians ground instance identities across views, producing renderable and cross-view-consistent instance features. To allow reconstruction and scene understanding to benefit from each other, we further design an instance-centric learning strategy that connects reconstruction, instance learning, and semantic learning through shared instance structure. Specifically, instance cues guide reconstruction, language-aligned semantics strengthen the discrimination of confusing same-category instances, and instance regions aggregate semantic evidence into coherent object-level predictions. Experiments on novel-view synthesis, instance segmentation, and open-vocabulary semantic understanding under varying input-view settings and on an unseen dataset demonstrate state-of-the-art performance, practical efficiency, and strong generalization.

Minchao Jiang, Xiaoxuan Ma, Shunyu Jia +3

1Shanghai Jiao Tong University · 2Eastern Institute of Technology, Ningbo · 3Carnegie Mellon University +2

3D Gaussian Splatting Open-Vocabulary 3D Segmentation Feed-Forward 3D Reconstruction 3D Instance Segmentation 3D Scene Understanding

Ask PDF

May 6, 2026

Ilov3Splat: Instance-Level Open-Vocabulary 3D Scene Understanding in Gaussian Splatting

We introduce Ilov3Splat, a novel framework for instance-level open-vocabulary 3D scene understanding built on 3D Gaussian Splatting (3D-GS). Most prior work depends on 2D rendering-based matching or point-level semantic association, which undermines cross-view consistency, lacks coherent instance-level reasoning, and limits precision in downstream 3D tasks. To address these limitations, our method jointly optimizes scene geometry and semantic representations by augmenting Gaussian splats with view-consistent feature fields. Specifically, we leverage multi-resolution hash embedding to efficiently encode language-aligned CLIP features, enabling dense and coherent language grounding in 3D space. We further train an instance feature field using contrastive loss over SAM masks, supporting fine-grained object distinction across views. At inference time, CLIP-encoded queries are matched against the learned features, followed by two-stage 3D clustering to retrieve relevant Gaussian groups. This enables our framework to identify arbitrary objects in 3D scenes based on natural language descriptions, without requiring category supervision or manual annotations. Experiments on standard benchmarks demonstrate that Ilov3Splat outperforms prior open-vocabulary 3D-GS methods in both object selection and instance segmentation, offering a flexible and accurate solution for language-driven 3D scene understanding. Project page: https://csiro-robotics.github.io/Ilov3Splat.

Binh Long Nguyen, Kien Nguyen, Sridha Sridharan +2

School of Electrical Engineering and Robotics, Queensland University of Technology (QUT), Brisbane, QLD 4000, Australia · CSIRO Robotics, CSIRO, Brisbane, QLD 4069, Australia

3D Gaussian Splatting Language-Embedded 3DGS Open-Vocabulary 3D Segmentation 3D Instance Segmentation 3D Scene Understanding 3D Visual Grounding

Ask PDF

Mar 15, 2026

S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction

Existing offline feed-forward methods for joint scene understanding and reconstruction on long image streams often repeatedly perform global computation over an ever-growing set of past observations, causing runtime and GPU memory to increase rapidly with sequence length and limiting scalability. We propose Streaming Semantic Gaussian Splatting (S2GS), a strictly causal, incremental 3D Gaussian semantic field framework: it does not leverage future frames and continuously updates scene geometry, appearance, and instance-level semantics without reprocessing historical frames, enabling scalable online joint reconstruction and understanding. S2GS adopts a geometry-semantic decoupled dual-backbone design: the geometry branch performs causal modeling to drive incremental Gaussian updates, while the semantic branch leverages a 2D foundation vision model and a query-driven decoder to predict segmentation masks and identity embeddings, further stabilized by query-level contrastive alignment and lightweight online association with an instance memory. Experiments show that S2GS matches or outperforms strong offline baselines on joint reconstruction-and-understanding benchmarks, while significantly improving long-horizon scalability: it processes 1,000+ frames with much slower growth in runtime and GPU memory, whereas offline global-processing baselines typically run out of memory at around 80 frames under the same setting.

Renhe Zhang, Yuyang Tan, Jingyu Gong +4

East China Normal University · Shanghai Artificial Intelligence Laboratory

Streaming 3D Reconstruction 3D Scene Understanding 3D Gaussian Splatting 3D Instance Segmentation

Ask PDF