OVI-MAP:Open-Vocabulary Instance-Semantic Mapping

Zilong Deng; Federico Tombari; Marc Pollefeys; Johanna Wald; Daniel Barath

cs.CVMar 27, 2026

OVI-MAP:Open-Vocabulary Instance-Semantic Mapping

Authors: Zilong Deng, Federico Tombari, Marc Pollefeys, Johanna Wald, Daniel Barath

Organizations: ETH Zurich · University of Zurich · Google · TU Munich · Microsoft

Abstract

Incremental open-vocabulary 3D instance-semantic mapping is essential for autonomous agents operating in complex everyday environments. However, it remains challenging due to the need for robust instance segmentation, real-time processing, and flexible open-set reasoning. Existing methods often rely on the closed-set assumption or dense per-pixel language fusion, which limits scalability and temporal consistency. We introduce OVI-MAP that decouples instance reconstruction from semantic inference. We propose to build a class-agnostic 3D instance map that is incrementally constructed from RGB-D input, while semantic features are extracted only from a small set of automatically selected views using vision-language models. This design enables stable instance tracking and zero-shot semantic labeling throughout online exploration. Our system operates in real time and outperforms state-of-the-art open-vocabulary mapping baselines on standard benchmarks.

Figures & tables

Figure 1: Overview of OVI-MAP. Given a streaming RGB-D sequence with camera poses, OVI-MAP incrementally reconstructs a volumetric 3D scene while maintaining a class-agnostic instance map. Semantic information is then assigned in a zero-shot manner using selectively chosen views, enabling open-set object recognition. Our method supports real-time, open-world scene reconstruction with instance-level semantic understanding.

Method	3D Recon.	Inst. Map	Zero-Shot Sem.	Online
Ours	TSDF	✔	✔	✔
OVO-SLAM [ 29 ]	PCL/3DGS	✔	✔	✔
OpenFusion [ 53 ]	TSDF	✘	✘	✔
ConceptFusion [ 19 ]	PCL	✘	✔	✔
OV-Octree-Graph [ 48 ]	PCL	✔	✔	✘
DCSEG [ 50 ]	3DGS	✔	✘	✘

Table 1: State-of-the-art online and offline semantic/instance mapping methods. PCL stands for Point Cloud, and 3DGS denotes 3D Gaussian Splatting [ 20 ] .

Panel of Figure 2 — Figure 2: Overview of the proposed pipeline. (a) Class-agnostic instance map reconstruction: each RGB-D frame is segmented into entity proposals and refined with geometry-aware depth segmentation. The fused segments are lifted into 3D and incrementally integrated into a global TSDF-based instance map via super-point registration and spatial voting. (b) Incremental semantic feature aggregation: given the global instance map, each instance is re-projected into new frames via depth-guided ray casting. A view selection module identifies informative viewpoints based on object-centric coverage. Selected views are cropped at multiple scales and masked before being passed to a Vision-Language Model (VLM). The resulting features are aggregated per instance, producing stable open-set semantic embeddings.

Figure 3: Comparison of view selection strategies. The left illustration shows the pixel-counting strategy [ 47 ] , which prioritizes frames with larger object masking area, often leading to redundant front-facing views. The right illustration depicts our proposed object-centric view coverage method, which maintains a spherical map of explored viewing directions and selects frames that provide novel perspectives of the object. This yields a more diverse and informative set of viewpoints for semantic feature extraction.

	Method	Online	mIoU	$\mathbf{AP_{75}}$	$\mathbf{AP_{50}}$	$\mathbf{AP_{25}}$
Replica	Mask3D	✘	23.1	14.3	31.2	56.2
	Segment3D	✘	29.3	1 6.5	14.5	24.9
	OVO-SLAM	✔	42.7	11.1	23.6	32.8
	Ours	✔	36.3	22.0	50.8	76.7
ScanNet	Mask3D	✘	47.6	16.9	36.1	47.8
	Segment3D	✘	42.0	1 2.5	1 9.3	16.5

Table 2: Instance segmentation performance on Replica [ 45 ] and ScanNet [ 7 ] . The Online column indicates whether the method performs incremental mapping during exploration. We achieve comparable or better accuracy to offline approaches while consistently outperforming prior online systems, particularly at high IoU thresholds (

AP_{75}

). Mask3D was trained on ScanNet.

Panel of Figure 4 — Figure 4: Open-vocabulary 3D semantic maps aligned to the ground-truth label sets of the respective datasets. We compare our method with online [ 29 ] and offline [ 10 , 14 , 36 , 18 , 47 ] approaches on the Replica (a) and ScanNet (b) datasets. Colors correspond to the semantic classes defined in each dataset. Gray regions indicate unobserved areas. OVI-MAP produces spatially coherent and semantically accurate reconstructions, maintaining sharp instance boundaries and consistent semantics throughout incremental mapping. Minor discrepancies in color (e.g., pillows in (a) and the table in (b)) arise from closed-set label mapping, where the ground truth uses alternative class names such as “cushion” or “dining table”.

	Method	Online	mIoU	mAcc	$\mathbf{AP_{25}}$	$\mathbf{AP_{50}}$	$\mathbf{AP_{all}}$
Replica	Mask3D+OM3D	✘	18.7	29.4	1 6.8	1 4.5	3.2
	Segment3D+OM3D	✘	17.3	30.9	19.9	15.2	9.0
	Semantic Gaussian	✘	16.1	22.1	–	–	–
	OpenScene	✘	19.8	33.9	–	–	–
	OpenNeRF	✘	19.2	30.9	–	–	–
	OpenFusion	✔	14.9	22.4	–	–	–

Table 3: Comparison of open-vocabulary 3D semantic and instance segmentation performance on Replica [ 45 ] and ScanNet [ 7 ] . OM3D denotes OpenMask3D [ 47 ] . The 30 FPS rows correspond to experiments where only every

n

-th frame is processed to match real-time performance. Our method achieves the best open-vocabulary semantic-instance accuracy among online systems, and maintains performance comparable to offline approaches even under real-time constraints.

Figure 5: Instance highlighting from arbitrary text queries. Given natural language prompts, our system retrieves and highlights corresponding 3D instances based on the learned vision-language embeddings. The examples demonstrate zero-shot grounding of both concrete (“ pillow ”, “ toilet ”) and abstract (“ where to sleep ”, “ where is the music ”) concepts in reconstructed scenes. Darker tones indicate higher cosine similarity between an object and the query.

	Method	mIoU	mAcc	$\mathbf{AP_{25}}$	$\mathbf{AP_{50}}$	AQ $\downarrow$
Replica	Random 8 Views	23.8	30.4	31.4	18.6	50.3
	Pixel Counting	26.5	31.8	33.2	19.8	18.7
	View Coverage (Ours)	26.5	32.2	34.5	21.2	1 8.6
	GT Inst. + Pixel Cnt.	38.0	50.3	37.6	37.6	20.7
	GT Inst. + View Cov.	37.6	48.4	36.2	36.2	10.4
ScanNet	Random 8 Views	14.7	22.4	19.2	13.7	23.0

Table 4: View selection strategies for semantic feature extraction: (1) random view selection, (2) larger visible-pixel counting similar to [ 47 ] , and (3) our proposed object-centric view coverage method. AQ denotes the average number of VLM queries per instance. Our approach achieves comparable zero-shot instance-level semantic precision (

AP_{25}

AP_{50}

) to pixel-counting while requiring substantially fewer (47%) VLM queries. The GT Inst. + Pixel Cnt. and GT Inst. + View Cov. configurations provide an upper bound using ground-truth instance masks.

Figure 6: Incremental performance on Replica as the number of observed views increases. We compare our incremental semantic aggregation using the proposed view coverage strategy against pixel-counting and other baselines. Our method achieves comparable semantic accuracy (mIoU, mAcc, $AP_{25}$ ) while requiring significantly fewer VLM queries per instance, demonstrating efficient and scalable open-vocabulary mapping during online exploration.

Method	Instance Segmentation			Semantic Segmentation
Method	mIoU	$\mathbf{AP_{75}}$	$\mathbf{AP_{50}}$	mIoU	mAcc	$\mathbf{AP_{25}}$	$\mathbf{AP_{50}}$
SAM2	36.6	14.2	27.8	20.2	26.4	26.2	18.6
Cropformer (Ours)	36.3	22.0	50.8	26.9	33.2	36.4	22.0
GT 2D Inst. Seg.	53.9	38.8	65.7	26.6	33.8	36.1	27.3
GT Instances Map	100	100	100	38.0	50.3	37.6	37.6

Table 5: Influence of instance segmentation: instance map built with 2D instance masks from (1) SAM2, (2) CropFormer, and (3) ground-truth 2D instance masks. All methods use the same view-selection strategy for semantic fusion. CropFormer yields the highest zero-shot semantic precision, while the GT Instances row represents an upper bound with perfect instance masks.

Method	mIoU	mAcc	$\mathbf{AP_{25}}$	$\mathbf{AP_{50}}$	$\mathbf{AP_{all}}$
Averaging	26.5	31.8	33.2	19.8	8.3
+ Weighted by Pixel Count	26.9	33.2	36.4	22.0	8.6
Cluster - Max. Cosine Sim.	25.1	31.9	33.1	19.7	8.4
Cluster - Min. $L_{1}$ Distance	24.8	31.2	32.8	19.5	8.3

Table 6: Semantic feature fusion by (1) averaging, (2) weighted averaging by normalized pixel counts (Ours), (3) selecting the feature with maximum average cosine similarity to others, and (4) selecting the feature with minimum average

L_{1}

distance to others.

Method	mIoU	mAcc	$\mathbf{AP_{25}}$	$\mathbf{AP_{50}}$	$\mathbf{AP_{all}}$	Param.
clip-vit-large-patch14-336	15.4	24.7	20.6	14.9	6.7	0.4B
siglip-large-patch16-384 (Ours)	26.9	33.2	36.4	22.0	8.6	0.7B
siglip-so400m-patch14-384	25.2	33.8	32.4	19.1	9.7	0.9B
siglip2-large-patch16-384	26.7	33.9	35.9	22.2	10.4	0.9B
siglip2-so400m-patch14-384	26.4	34.3	36.4	23.1	9.4	1B

Table 7: Semantic feature extraction by (1) ViT-L, (2) siglip-large (Ours), (3) siglip-so400m, (4) siglip2-large, and (5) siglip2-so400m. The last column shows the number of parameters for each model.

Method	Component	Run-Time / Frame	Thread	Skipped Frames
Ours	RGB Segmentation	964.8 ms	1	30
	Depth Segmentation	1 88.4 ms	2
	2D-3D Association	1 76.3 ms
	View Selection	140.8 ms		10
	Feature Extraction	131.3 ms	3	10
OVO-SLAM	Segmentation	1441.8 ms	1	50

Table 8: Runtime breakdown. All components run in parallel across dedicated threads. These steps are performed only on keyframes, meaning only every

n

-th frame is processed to match real-time performance at 30 FPS. The number of skipped frames for each component are listed in the last column. This pipeline design keeps the overall system real-time despite potentially expensive individual modules.

Panel of Figure 7 — Figure 7: We compare our method with online [ 29 ] and offline [ 18 , 43 ] approaches on the Replica (a) and ScanNet (b) datasets. Colors are randomly assigned for all instance maps according to the instance labels. Gray regions indicate unobserved areas for online methods (Ours and OVO-SLAM), and unlabeled for offline methods (Segment3D, Mask3D). OVI-MAP produces spatially coherent accurate reconstructions, maintaining sharp instance boundaries throughout incremental mapping.

Panel of Figure 8 — Figure 8: Heat map visualizations of the semantic queries. The color closer to red indicates the instance is more similar to the query semantic label, while the color closer to blue indicates the instance is less similar to the query semantic label. Unobserved areas are shown in black.

Explore similar work

Jun 13, 2025

OV-MAP: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots

We introduce OV-MAP, a novel approach to open-world 3D mapping for mobile robots by integrating open-features into 3D maps to enhance object recognition capabilities. A significant challenge arises when overlapping features from adjacent voxels reduce instance-level precision, as features spill over voxel boundaries, blending neighboring regions together. Our method overcomes this by employing a class-agnostic segmentation model to project 2D masks into 3D space, combined with a supplemented depth image created by merging raw and synthetic depth from point clouds. This approach, along with a 3D mask voting mechanism, enables accurate zero-shot 3D instance segmentation without relying on 3D supervised segmentation models. We assess the effectiveness of our method through comprehensive experiments on public datasets such as ScanNet200 and Replica, demonstrating superior zero-shot performance, robustness, and adaptability across diverse environments. Additionally, we conducted real-world experiments to demonstrate our method's adaptability and robustness when applied to diverse real-world environments.

Juno Kim, Yesol Park, Hye-Jung Yoon +1

Interdisciplinary Program in AI, Seoul National University · Artificial Intelligence Institute, Seoul National University · Department of Computer Science, Seoul National University

Robotic Perception 3D Instance Segmentation

Ask PDF

May 5, 2026

FUS3DMaps: Scalable and Accurate Open-Vocabulary Semantic Mapping by 3D Fusion of Voxel- and Instance-Level Layers

Open-vocabulary semantic mapping enables robots to spatially ground previously unseen concepts without requiring predefined class sets. Current training-free methods commonly rely on multi-view fusion of semantic embeddings into a 3D map, either at the instance-level via segmenting views and encoding image crops of segments, or by projecting image patch embeddings directly into a dense semantic map. The latter approach sidesteps segmentation and 2D-to-3D instance association by operating on full uncropped image frames, but existing methods remain limited in scalability. We present FUS3DMaps, an online dual-layer semantic mapping method that jointly maintains both dense and instance-level open-vocabulary layers within a shared voxel map. This design enables further voxel-level semantic fusion of the layer embeddings, combining the complementary strengths of both semantic mapping approaches. We find that our proposed semantic cross-layer fusion approach improves the quality of both the instance-level and dense layers, while also enabling a scalable and highly accurate instance-level map where the dense layer and cross-layer fusion are restricted to a spatial sliding window. Experiments on established 3D semantic segmentation benchmarks as well as a selection of large-scale scenes show that FUS3DMaps achieves accurate open-vocabulary semantic mapping at multi-story building scales. Additional material and code will be made available: https://githanonymous.github.io/FUS3DMaps/.

Timon Homberger, Finn Lukas Busch, Jesús Gerardo Ortega Peimbert +2

Division of Robotics, Perception, and Learning, KTH Royal Institute of Technology, Sweden

Open-Vocabulary 3D Segmentation 3D Semantic Segmentation Semantic Mapping 3D Scene Understanding

Ask PDF

Sep 1, 2026

VOIM: Training-Free Open-Vocabulary 3D Instance Mapping for RGB-D and Monocular SLAM

We present Voxel-Grounded Online Instance Manager (VOIM), a training-free voxel-grounded instance manager that builds open-vocabulary 3D instance maps from RGB-D or from monocular RGB alone, a regime no prior training-free system addresses. Online systems typically segment object instances and label them at first detection, committing when evidence is weakest. VOIM instead defers label and instance decisions until soft evidence from unmodified, off-the-shelf perception has accumulated per voxel across views. We show that the mapping stage, rather than the particular perception models, carries the result: across four perception configurations on ScanNet++, varying the region descriptor, the detector label prior and the mask source, the map exceeds the strongest online RGB-D system, OVO-SLAM, by between 4.8 and 11.7 mIoU. Perception is not neutral, and substituting that baseline's own descriptor family costs 4.1 of the margin, yet the baseline carries the marginally better 2D descriptor (33.7 vs. 31.5 mIoU over three scenes) and still realizes the weaker map. Under a like-for-like protocol VOIM reaches 44.07 mIoU on ScanNet++ against 32.37, winning all ten scenes and both aggregations (pooled 33.31 vs. 25.97), and the same system runs unchanged to fully monocular RGB, matching that baseline pooled on Replica (27.80 vs. 27.50). The advantage is regime-specific: under Replica's all-classes scoring, matched inputs give a split result, 28.60 vs. 27.50 pooled against 24.59 vs. 30.11 on the per-scene mean. Room scale is label-limited and building scale drift-limited. Labeling does not run in real time, dominated by per-class detection over the full vocabulary. The maps export occupancy grids and resolve free-form queries to object instances.

Sangmin Song, Sarath Kodagoda, Marc G. Carmichael +4

Robotics Institute, Faculty of Engineering and Information Technology, University of Technology Sydney, Ultimo NSW 2007, Australia · Smart Sensing and Robotics Laboratory (SensR Lab), Centre for Advanced Manufacturing Technology, School of Engineering, Design and Built Environment, Kingswood NSW 2747, Australia · Guide Dogs New South Wales/Australian Capital Territory, Sydney NSW, Australia

Simultaneous Localization and Mapping Open-Vocabulary 3D Segmentation 3D Instance Segmentation

Ask PDF