cs.CVMar 27, 2026
SaveOVI-MAP:Open-Vocabulary Instance-Semantic Mapping
Organizations: ETH Zurich · University of Zurich · Google · TU Munich · Microsoft
Abstract
Incremental open-vocabulary 3D instance-semantic mapping is essential for autonomous agents operating in complex everyday environments. However, it remains challenging due to the need for robust instance segmentation, real-time processing, and flexible open-set reasoning. Existing methods often rely on the closed-set assumption or dense per-pixel language fusion, which limits scalability and temporal consistency. We introduce OVI-MAP that decouples instance reconstruction from semantic inference. We propose to build a class-agnostic 3D instance map that is incrementally constructed from RGB-D input, while semantic features are extracted only from a small set of automatically selected views using vision-language models. This design enables stable instance tracking and zero-shot semantic labeling throughout online exploration. Our system operates in real time and outperforms state-of-the-art open-vocabulary mapping baselines on standard benchmarks.
Figures & tables
Figure 1: Overview of OVI-MAP. Given a streaming RGB-D sequence with camera poses, OVI-MAP incrementally reconstructs a volumetric 3D scene while maintaining a class-agnostic instance map. Semantic information is then assigned in a zero-shot manner using selectively chosen views, enabling open-set object recognition. Our method supports real-time, open-world scene reconstruction with instance-level semantic understanding.
| Method | 3D Recon. | Inst. Map | Zero-Shot Sem. | Online |
|---|---|---|---|---|
| Ours | TSDF | ✔ | ✔ | ✔ |
| OVO-SLAM [ 29 ] | PCL/3DGS | ✔ | ✔ | ✔ |
| OpenFusion [ 53 ] | TSDF | ✘ | ✘ | ✔ |
| ConceptFusion [ 19 ] | PCL | ✘ | ✔ | ✔ |
| OV-Octree-Graph [ 48 ] | PCL | ✔ | ✔ | ✘ |
| DCSEG [ 50 ] | 3DGS | ✔ | ✘ | ✘ |
Table 1: State-of-the-art online and offline semantic/instance mapping methods. PCL stands for Point Cloud, and 3DGS denotes 3D Gaussian Splatting [ 20 ] .
Figure 2: Overview of the proposed pipeline. (a) Class-agnostic instance map reconstruction: each RGB-D frame is segmented into entity proposals and refined with geometry-aware depth segmentation. The fused segments are lifted into 3D and incrementally integrated into a global TSDF-based instance map via super-point registration and spatial voting. (b) Incremental semantic feature aggregation: given the global instance map, each instance is re-projected into new frames via depth-guided ray casting. A view selection module identifies informative viewpoints based on object-centric coverage. Selected views are cropped at multiple scales and masked before being passed to a Vision-Language Model (VLM). The resulting features are aggregated per instance, producing stable open-set semantic embeddings.
Figure 3: Comparison of view selection strategies. The left illustration shows the pixel-counting strategy [ 47 ] , which prioritizes frames with larger object masking area, often leading to redundant front-facing views. The right illustration depicts our proposed object-centric view coverage method, which maintains a spherical map of explored viewing directions and selects frames that provide novel perspectives of the object. This yields a more diverse and informative set of viewpoints for semantic feature extraction.
| Method | Online | mIoU | ||||
|---|---|---|---|---|---|---|
| Replica | Mask3D | ✘ | 23.1 | 14.3 | 31.2 | 56.2 |
| Segment3D | ✘ | 29.3 | 1 6.5 | 14.5 | 24.9 | |
| OVO-SLAM | ✔ | 42.7 | 11.1 | 23.6 | 32.8 | |
| Ours | ✔ | 36.3 | 22.0 | 50.8 | 76.7 | |
| ScanNet | Mask3D | ✘ | 47.6 | 16.9 | 36.1 | 47.8 |
| Segment3D | ✘ | 42.0 | 1 2.5 | 1 9.3 | 16.5 |
Table 2: Instance segmentation performance on Replica [ 45 ] and ScanNet [ 7 ] . The Online column indicates whether the method performs incremental mapping during exploration. We achieve comparable or better accuracy to offline approaches while consistently outperforming prior online systems, particularly at high IoU thresholds ( ). Mask3D was trained on ScanNet.
Figure 4: Open-vocabulary 3D semantic maps aligned to the ground-truth label sets of the respective datasets. We compare our method with online [ 29 ] and offline [ 10 , 14 , 36 , 18 , 47 ] approaches on the Replica (a) and ScanNet (b) datasets. Colors correspond to the semantic classes defined in each dataset. Gray regions indicate unobserved areas. OVI-MAP produces spatially coherent and semantically accurate reconstructions, maintaining sharp instance boundaries and consistent semantics throughout incremental mapping. Minor discrepancies in color (e.g., pillows in (a) and the table in (b)) arise from closed-set label mapping, where the ground truth uses alternative class names such as “cushion” or “dining table”.
| Method | Online | mIoU | mAcc | ||||
| Replica | Mask3D+OM3D | ✘ | 18.7 | 29.4 | 1 6.8 | 1 4.5 | 3.2 |
| Segment3D+OM3D | ✘ | 17.3 | 30.9 | 19.9 | 15.2 | 9.0 | |
| Semantic Gaussian | ✘ | 16.1 | 22.1 | – | – | – | |
| OpenScene | ✘ | 19.8 | 33.9 | – | – | – | |
| OpenNeRF | ✘ | 19.2 | 30.9 | – | – | – | |
| OpenFusion | ✔ | 14.9 | 22.4 | – | – | – |
Table 3: Comparison of open-vocabulary 3D semantic and instance segmentation performance on Replica [ 45 ] and ScanNet [ 7 ] . OM3D denotes OpenMask3D [ 47 ] . The 30 FPS rows correspond to experiments where only every -th frame is processed to match real-time performance. Our method achieves the best open-vocabulary semantic-instance accuracy among online systems, and maintains performance comparable to offline approaches even under real-time constraints.
Figure 5: Instance highlighting from arbitrary text queries. Given natural language prompts, our system retrieves and highlights corresponding 3D instances based on the learned vision-language embeddings. The examples demonstrate zero-shot grounding of both concrete (“ pillow ”, “ toilet ”) and abstract (“ where to sleep ”, “ where is the music ”) concepts in reconstructed scenes. Darker tones indicate higher cosine similarity between an object and the query.
| Method | mIoU | mAcc | AQ | |||
| Replica | Random 8 Views | 23.8 | 30.4 | 31.4 | 18.6 | 50.3 |
| Pixel Counting | 26.5 | 31.8 | 33.2 | 19.8 | 18.7 | |
| View Coverage (Ours) | 26.5 | 32.2 | 34.5 | 21.2 | 1 8.6 | |
| GT Inst. + Pixel Cnt. | 38.0 | 50.3 | 37.6 | 37.6 | 20.7 | |
| GT Inst. + View Cov. | 37.6 | 48.4 | 36.2 | 36.2 | 10.4 | |
| ScanNet | Random 8 Views | 14.7 | 22.4 | 19.2 | 13.7 | 23.0 |
Table 4: View selection strategies for semantic feature extraction: (1) random view selection, (2) larger visible-pixel counting similar to [ 47 ] , and (3) our proposed object-centric view coverage method. AQ denotes the average number of VLM queries per instance. Our approach achieves comparable zero-shot instance-level semantic precision ( / ) to pixel-counting while requiring substantially fewer (47%) VLM queries. The GT Inst. + Pixel Cnt. and GT Inst. + View Cov. configurations provide an upper bound using ground-truth instance masks.
Figure 6: Incremental performance on Replica as the number of observed views increases. We compare our incremental semantic aggregation using the proposed view coverage strategy against pixel-counting and other baselines. Our method achieves comparable semantic accuracy (mIoU, mAcc, ) while requiring significantly fewer VLM queries per instance, demonstrating efficient and scalable open-vocabulary mapping during online exploration.
| Method | Instance Segmentation | Semantic Segmentation | |||||
| mIoU | mIoU | mAcc | |||||
| SAM2 | 36.6 | 14.2 | 27.8 | 20.2 | 26.4 | 26.2 | 18.6 |
| Cropformer (Ours) | 36.3 | 22.0 | 50.8 | 26.9 | 33.2 | 36.4 | 22.0 |
| GT 2D Inst. Seg. | 53.9 | 38.8 | 65.7 | 26.6 | 33.8 | 36.1 | 27.3 |
| GT Instances Map | 100 | 100 | 100 | 38.0 | 50.3 | 37.6 | 37.6 |
Table 5: Influence of instance segmentation: instance map built with 2D instance masks from (1) SAM2, (2) CropFormer, and (3) ground-truth 2D instance masks. All methods use the same view-selection strategy for semantic fusion. CropFormer yields the highest zero-shot semantic precision, while the GT Instances row represents an upper bound with perfect instance masks.
| Method | mIoU | mAcc | |||
|---|---|---|---|---|---|
| Averaging | 26.5 | 31.8 | 33.2 | 19.8 | 8.3 |
| + Weighted by Pixel Count | 26.9 | 33.2 | 36.4 | 22.0 | 8.6 |
| Cluster - Max. Cosine Sim. | 25.1 | 31.9 | 33.1 | 19.7 | 8.4 |
| Cluster - Min. Distance | 24.8 | 31.2 | 32.8 | 19.5 | 8.3 |
Table 6: Semantic feature fusion by (1) averaging, (2) weighted averaging by normalized pixel counts (Ours), (3) selecting the feature with maximum average cosine similarity to others, and (4) selecting the feature with minimum average distance to others.
| Method | mIoU | mAcc | Param. | |||
|---|---|---|---|---|---|---|
| clip-vit-large-patch14-336 | 15.4 | 24.7 | 20.6 | 14.9 | 6.7 | 0.4B |
| siglip-large-patch16-384 (Ours) | 26.9 | 33.2 | 36.4 | 22.0 | 8.6 | 0.7B |
| siglip-so400m-patch14-384 | 25.2 | 33.8 | 32.4 | 19.1 | 9.7 | 0.9B |
| siglip2-large-patch16-384 | 26.7 | 33.9 | 35.9 | 22.2 | 10.4 | 0.9B |
| siglip2-so400m-patch14-384 | 26.4 | 34.3 | 36.4 | 23.1 | 9.4 | 1B |
Table 7: Semantic feature extraction by (1) ViT-L, (2) siglip-large (Ours), (3) siglip-so400m, (4) siglip2-large, and (5) siglip2-so400m. The last column shows the number of parameters for each model.
| Method | Component | Run-Time / Frame | Thread | Skipped Frames |
| Ours | RGB Segmentation | 964.8 ms | 1 | 30 |
| Depth Segmentation | 1 88.4 ms | 2 | ||
| 2D-3D Association | 1 76.3 ms | |||
| View Selection | 140.8 ms | 10 | ||
| Feature Extraction | 131.3 ms | 3 | ||
| OVO-SLAM | Segmentation | 1441.8 ms | 1 | 50 |
Table 8: Runtime breakdown. All components run in parallel across dedicated threads. These steps are performed only on keyframes, meaning only every -th frame is processed to match real-time performance at 30 FPS. The number of skipped frames for each component are listed in the last column. This pipeline design keeps the overall system real-time despite potentially expensive individual modules.
Figure 7: We compare our method with online [ 29 ] and offline [ 18 , 43 ] approaches on the Replica (a) and ScanNet (b) datasets. Colors are randomly assigned for all instance maps according to the instance labels. Gray regions indicate unobserved areas for online methods (Ours and OVO-SLAM), and unlabeled for offline methods (Segment3D, Mask3D). OVI-MAP produces spatially coherent accurate reconstructions, maintaining sharp instance boundaries throughout incremental mapping.
Figure 8: Heat map visualizations of the semantic queries. The color closer to red indicates the instance is more similar to the query semantic label, while the color closer to blue indicates the instance is less similar to the query semantic label. Unobserved areas are shown in black.