AOV-GS: Active Open-Vocabulary Semantic Mapping with 3D Gaussian Splatting

Rodion Shkokov; Tatiana Zemskova; Dmitry Yudin

cs.CVOct 21, 2026

AOV-GS: Active Open-Vocabulary Semantic Mapping with 3D Gaussian Splatting

Authors: Rodion Shkokov, Tatiana Zemskova, Dmitry Yudin

Organizations: Moscow Institute of Physics and Technology (MIPT), Dolgoprudny, Russia · AIRI, Moscow, Russia

Abstract

Robots deployed in human-centered environments need maps that support both geometric navigation and open-vocabulary semantic queries beyond a fixed label set. A key challenge is enriching such maps with semantic information without relying on manually annotated segmentation masks or closed-set semantic categories. Existing geometry-driven active mapping maximizes coverage but treats semantics as a passive by-product of the collected trajectory, whereas active semantic mapping typically depends on supervised semantic segmentation. We present AOV-GS, an active 3D Gaussian mapping system that guides exploration using frozen foundation models. During coarse exploration, AOV-GS re-ranks geometrically promising candidate views according to CLIP mask-embedding novelty computed from SAM instance segmentation, biasing exploration toward semantically informative regions. Frozen SAM and CLIP provide instance masks and language-aligned embeddings throughout mapping, while a lightweight scene-specific language field is fitted offline directly to the collected CLIP embeddings rather than manually annotated masks, enabling open-vocabulary text-query segmentation on novel views. In a controlled comparison using the same SAM-CLIP perception stack, AOV-GS improves average language-field mIoU over geometry-only active planning by 1.3 percentage points on Replica (34.6% on eight scenes) and by 1.8 points on Matterport3D (23.1% on five scenes). With CorrCLIP, AOV-GS reaches 35.6% on Replica, 2.0 points above the matched Geometry-based + CorrCLIP baseline. Novel-view PSNR remains comparable to the geometry-only baseline on both datasets while depth L1 improves on Replica. Code is available at https://github.com/OptimusFaber/AOV-GS.

Figures & tables

Figure 1: AOV-GS system overview. Online loop : posed RGB-D updates a Gaussian map; the planner selects NBVs; CLIP mask embeddings are extracted in parallel on keyframes. Offline stage : a language field is trained on Gaussians and queried by text.

Figure 2: Stage-0 NBV on office0 : candidate poses, geometric scores, and geometry-only vs. semantic-aware argmax.

Figure 3: SAM–CLIP extraction on keyframes with optional CorrCLIP post-processing; embeddings feed $\mathcal{B}$ and offline language-field training.

Figure 4: Open-vocabulary text query via multi-scale cosine-similarity maps and thresholding.

Figure 5: Language-field masks on novel views for queries door, sofa, floor, and blanket. AOV-GS improves large-object coverage ( office3 /sofa, room0 /floor); narrow objects such as the office0 door remain difficult for all methods.

Figure 6: Exploration routes on office0 : AOV-GS covers the central workspace more thoroughly than the geometry-only planner.

Method	Steps $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	L1 (cm) $\downarrow$
Passive	1999	27.45	0.949	0.149	6.98
Geometry-based	914	31.47	0.964	0.131	1.26
ActiveSGM	868	32.11	0.970	0.110	1.23
AOV-GS (Ours)	956	31.19	0.964	0.123	1.18
AOV-GS + CorrCLIP (Ours)	952	31.56	0.964	0.128	1.14

Table 3: Novel-view rendering after refinement on Replica (eight-scene average, single run).

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mIoU (%)
Geometry-based	$\textbf{31.62}{\pm}2.63$	$\textbf{0.965}{\pm}0.014$	$0.128{\pm}0.033$	$28.8{\pm}1.4$
AOV-GS (Ours)	$31.53{\pm}2.44$	$\textbf{0.965}{\pm}0.014$	$\textbf{0.126}{\pm}0.027$	$\textbf{33.6}{\pm}1.1$

Table 4: Five-seed stability on Replica (seeds 0–4). Rendering: eight-scene average; mIoU: office0 only.

Figure 7: Language-field masks on MP3D novel views for gZ6f7 /seating, HxpKQ /chair, YmJkq /seating, and Gdvg /bed (GT in green; Geometry-based in orange; AOV-GS in yellow).

Method	Steps $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	L1 (cm) $\downarrow$
Geometry-based	3496	26.19	0.928	0.233	2.80
ActiveSGM	3821	26.08	0.917	0.259	3.76
AOV-GS (Ours)	3472	26.25	0.931	0.232	2.83
AOV-GS + CorrCLIP (Ours)	3454	26.27	0.932	0.233	2.79

Table 6: Novel-view rendering on Matterport3D (five-scene average, single run); each scene uses the better offline evaluation of the stage-1 and final-refinement outputs.

Semantic Mapping

Semantic mapping links observations and data to meaningful concepts, categories, entities, and relations for machine reasoning and action.

See all 93 papers

Oct 7, 2026

ActiveLang: Active Open-Vocabulary 3D Mapping with Semantic-Uncertainty-Guided Exploration

As robots increasingly assist humans with diverse tasks, they need both geometric and semantic understanding of their surroundings. Moreover, robots often operate in unfamiliar environments and take on new tasks without knowing the relevant concepts ahead of time. This motivates language-annotated 3D maps that support open-vocabulary scene understanding and human-robot interaction. We introduce ActiveLang, an autonomous system for active open-vocabulary 3D mapping with semantic-uncertainty-guided exploration. ActiveLang performs online language-feature adaptation on a compact dual-Gaussian representation to jointly reconstruct scene geometry, appearance, and open-vocabulary semantics with modest memory overhead. Its planner efficiently selects informative viewpoints, enabling effective mapping with fewer observations and lower computational cost. Experiments on Replica and ScanNet++ demonstrate substantial improvements in 2D and 3D open-vocabulary segmentation over both online and offline baselines, highlighting that actively exploring scenes builds language-annotated 3D maps more efficiently.

Liyan Chen, Hairong Yin, Huangying Zhan +3

Stevens Institute of Technology · Purdue University · Goertek Alpha Labs

3D Semantic Mapping 3D Mapping 3D Scene Representation Open-Vocabulary 3D Segmentation 3D Reconstruction Active Perception+1

Ask PDF

Oct 6, 2026

OpenSplatGraph: From Dense Semantic Maps to Structured Scene Graphs for Open-Vocabulary Robot Perception

Dense 3D mapping with semantic understanding is essential for robotic perception in complex environments. Recent 3D Gaussian Splatting-based mapping approaches enable high-fidelity geometry and efficient open-vocabulary perception, but typically represent semantics as unstructured feature fields that limit object-centric reasoning. In contrast, 3D scene graphs explicitly model objects and their relationships for structured reasoning, but are commonly constructed from sparse geometric representations that do not fully exploit dense semantic maps. In this work, we present OpenSplatGraph, a unified framework that constructs persistent 3D scene graphs directly from an online Gaussian-based open-vocabulary semantic map. The proposed framework augments the dense semantic map with a reliability-aware semantic field that maintains lightweight observation statistics for confidence-aware, query-conditioned object extraction. Extracted object instances are associated with persistent graph nodes, allowing object attributes and relationships to be incrementally updated across observations and queries. By tightly coupling dense semantic mapping with persistent object-centric representations, our framework supports both language-guided object grounding and structured relational reasoning while preserving the geometric fidelity of Gaussian-based mapping. Comprehensive evaluations on standard 3D scene understanding benchmarks and real-world robotic experiments demonstrate that OpenSplatGraph achieves competitive performance for online open-vocabulary perception and downstream robotic tasks. Project page: https://csiro-robotics.github.io/OpenSplatGraph.

Binh Long Nguyen, Kien Nguyen, Clinton Fookes +1

School of Electrical Engineering and Robotics, Queensland University of Technology (QUT), Brisbane, QLD 4000, Australia · CSIRO Robotics, CSIRO, Brisbane, QLD 4069, Australia

Language-Embedded 3DGS Open-Vocabulary 3D Segmentation 3D SGG 3D Visual Grounding 3D Semantic Mapping

Ask PDF

Jun 24, 2026

RoboAtlas: Contextual Active SLAM

We present RoboAtlas, a contextual Active SLAM framework that adaptively balances geometric exploration and semantic reasoning using a scalable 3D semantic mapping system, OpenRoboVox. RoboAtlas integrates frontier exploration, global semantic-map reasoning, and egocentric VLM-based reasoning through a contextual multi-armed bandit that transitions from exploration to semantically guided navigation as scene understanding improves. We evaluate the system in simulation and on a Unitree Go2 robot in large-scale real-world environments exceeding 1800 m2 with approx. 30k mapped semantic instances, achieving a 100% task success rate. On the GOAT-Bench "Val Unseen" benchmark, RoboAtlas achieves state-of-the-art performance with highest reported success rate (SR) of 90.6%, using GPT-4o, improving over the strongest prior baseline by 17.8 percentage points in SR. Using the much smaller Qwen2.5-VL-7B model, it still achieves 88.8% SR, outperforming all baselines using GPT-4o in SR, and revealing the importance of the information gained by our semantic mapping framework over simply replacing the underlying foundation model. The results demonstrate that grounding foundation models with large-scale 3D semantic maps enables robust and efficient contextual Active SLAM.

Alexander Schperberg, Shivam K. Panda, Abraham P. Vinod +3

Mitsubishi Electric Research Laboratories, Cambridge, MA 02139, USA · Department of Mechanical and Aerospace Engineering, University of California, Los Angeles, Los Angeles, CA 90095, USA · Mitsubishi Electric Corporation, Japan

Simultaneous Localization and Mapping Robot Navigation Semantic Mapping Autonomous Robot Exploration 3D Semantic Mapping

Ask PDF

Abstract

Semantic Mapping

Explore similar work

ActiveLang: Active Open-Vocabulary 3D Mapping with Semantic-Uncertainty-Guided Exploration

OpenSplatGraph: From Dense Semantic Maps to Structured Scene Graphs for Open-Vocabulary Robot Perception

RoboAtlas: Contextual Active SLAM