Semantic 3D mapping is a powerful representation that enables platform agnostic mobile robot navigation in challenging offroad conditions. Current vision foundation models can accurately predict semantics zero-shot on structured scenes but performance is compromised in natural environments. On the other hand, 3D point clouds encode valuable information for discriminating semantics in scenarios where vision foundation models fail. In this work, we propose fitting a sphericity-elevation distribution for each semantic class to act as geometric anchor. Only the points with the most confident vision foundation model predictions are considered when fitting to produce high quality geometric prior distributions. The geometric prior is fused with the vision foundation model prediction to produce the final semantic mapping. The proposed prior improves SAM3 baseline by +6.3 in mIoU and +8.9 in accuracy on the RELLIS-3D dataset, and by +1.7 in mIoU and +2.1 in accuracy on the WildScenes dataset. The source code will be made public upon paper acceptance.
Figures & tables
Fig. 1 : The proposed method improves semantic mapping by fusing the vision foundation model prediction with a geometric prior computed from the pointcloud.
Fig. 2 : Geometric prior computation and utilization illustrated. The most confident image segmentation predictions are used to fit the sphericity-elevation distributions for each class, which are then used to assign prior weights for each point in the fusion phase.
Fig. 3 : Sphericity and elevation visualized. Tree trunks and even ground have low sphericity and tree foliage and vegetation high sphericity.
Fig. 4 : Elevation-sphericity histograms fitted on the top 5 % SAM3 detections on Rellis3D. Number of samples used for fitting each class in parentheses.
Fig. 5 : Elevation-sphericity histograms fitted on the top 5 % SAM3 detections on WildScenes. Mud excluded due to insufficient number of points. Number of samples used for fitting each class in parentheses.
Method
mIoU
Acc
Asphalt
Barrier
Bush
Concrete
Fence
Grass
Log
Mud
Pole
Puddle
Rubble
Tree
Water
OTAS
20.1
69.6
0.0
5.2
9.6
36.6
5.4
68.7
0.1
21.9
0.0
20.8
22.6
70.8
0.0
SAM 3
28.1
73.9
0.0
68.7
7.2
42.7
5.8
73.4
12.1
30.2
0.3
28.5
27.1
68.8
0.0
+ OURS
34.4
82.8
0.0
68.7
49.4
42.2
4.6
82.2
12.6
30.6
0.2
33.7
26.9
67.5
28.7
DINOv2.txt
21.0
65.5
0.0
18.7
41.0
43.2
3.1
65.5
9.7
16.9
0.7
18.3
37.0
19.2
0.0
+ OURS
22.5
68.0
0.0
18.3
46.5
44.4
3.3
67.4
9.1
17.2
0.6
18.6
37.2
30.3
0.0
OpenSeg
21.5
75.1
0.0
41.1
41.7
0.0
22.8
77.3
0.0
14.7
9.3
22.4
25.3
24.9
0.0
TABLE I : Semantic mapping performance on Rellis3d dataset. Highest values bolded
Method
mIoU
Acc
Bush
Dirt
Fence
Grass
Gravel
Log
Mud
Rock
Structure
Tree-Foliage
Tree-Trunk
OTAS
14.5
22.1
3.4
15.3
7.5
40.9
5.5
0.0
0.0
0.1
69.0
0.4
16.9
SAM 3
47.9
80.5
13.4
69.8
48.3
57.0
50.7
35.8
0.0
42.7
82.3
78.5
48.1
+ OURS
49.6
82.6
17.2
77.7
44.0
67.8
50.8
34.9
0.0
40.3
83.9
79.8
49.1
DINOv2.txt
21.3
24.0
1.2
23.3
7.3
17.7
7.5
14.3
1.5
31.1
73.8
17.0
39.1
+ OURS
23.4
33.2
1.4
27.1
6.5
22.6
8.0
14.3
1.3
31.6
74.1
29.3
41.4
OpenSeg
27.0
73.9
14.9
56.6
49.2
17.1
18.4
2.3
9.0
0.5
51.7
77.4
0.2
TABLE II : Semantic mapping performance on WildScenes dataset. Highest values bolded
TABLE III : Examples of semantic pointcloud map predictions from ego view.
TABLE IV : Examples of semantic pointcloud map predictions from birds eye view.
Wildscenes
Rellis3d
Method
mIoU
Acc
mIoU
Acc
SAM 3
47.9
80.5
28.1
73.9
+ SPHE w =0.1
48.9
81.5
31.4
78.3
+ SPHE w =0.25
49.2
81.6
32.4
81.8
+ SPHE w =0.5
47.8
80.2
32.5
82.5
+ SPHE w =1.0
43.5
77.1
30.8
80.3
TABLE V : Geometry prior ablations with different smoothing values w for the geometric prior. ELEV defines the geometric prior based on elevation only, SPHE based on sphericity only and BOTH utilizes their joint distribution as presented in Section III-C . Highest values bolded.
Foundation features from self-supervised vision models and text-to-image diffusion models have proven effective for semantic correspondence estimation. However, because these features are learned primarily from 2D image objectives, they lack explicit 3D awareness and often confuse symmetric object sides, repeated parts, and visually similar structures that are distinct in 3D. We introduce a 3D-aware post-training framework that goes beyond available 2D foundation features by incorporating priors from 3D foundation models. Given an image, our method uses SAM3D to estimate object geometry and pose, and refines the pose through render-and-compare optimization. Subsequently, we render PartField descriptors from the reconstructed geometry into the image plane based on the estimated object pose. The resulting geometry-aware feature maps complement DINO and Stable Diffusion features, while geodesic distances on the reconstructed shapes enable reliable filtering of candidate correspondences. We use the filtered matches as supervision to train a lightweight adapter on top of DINO and Stable Diffusion for semantic correspondence. In contrast to prior post-training approaches that require pose annotations and rely on coarse spherical geometry, our method automatically obtains instance-specific 3D structure and uses it to guide correspondence learning. Experiments show that our approach improves semantic correspondence over the prior methods while reducing manual geometric supervision. Code and model can be found at https:/github.com/GenIntel/3D-SC.
Artur Jesslen, Olaf Dünkel, Adam Kortylewski
University of Freiburg, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany · CISPA Helmholtz Center for Information Security, Germany
Zero-shot 3D Visual Grounding (3DVG) is a critical capability for open-world embodied AI. However, existing methods are fundamentally bottlenecked by the poor quality of open-vocabulary 3D proposals, suffering from inaccurate categories and imprecise geometries, as well as the spatial redundancy of exhaustive multi-view reasoning. To address these challenges, we propose MCM-VG, a novel framework that achieves robust zero-shot 3DVG by explicitly establishing Multiple Consistent 2D-3D Mappings. Instead of passively relying on noisy 3D segments, MCM-VG enforces 2D-3D consistency across three fundamental dimensions to achieve precise target localization and reliable reasoning. First, a Semantic Alignment module corrects category mismatches via LLM-driven query parsing and coarse-to-fine 2D-3D matching. Second, an Instance Rectification module leverages VLM-guided 2D segmentations to reconstruct missing targets, back-projecting these reliable visual priors to establish accurate 3D geometries. Finally, to eliminate spatial redundancy, a Viewpoint Distillation module clusters 3D camera directions to extract optimal frames. By pairing these optimal RGB frames with Bird's Eye View maps into concise visual prompt sets, we formulate the final target disambiguation as a multiple-choice reasoning task for Vision-Language Models. Extensive evaluations on ScanRefer and Nr3D benchmarks demonstrate that MCM-VG sets a new state-of-the-art for zero-shot 3D visual grounding. Remarkably, it achieves 62.0% and 53.6% in [email protected] and [email protected] on ScanRefer, outperforming previous baselines by substantial margins of 6.4% and 4.0%.
Yufei Yin, Jie Zheng, Qianke Meng +7
Hangzhou Dianzi University Hangzhou, Zhejiang, China · Zhejiang University Hangzhou, Zhejiang, China · Alibaba Group Hangzhou, Zhejiang, China
Robots deployed in human-centered environments need maps that support both geometric navigation and open-vocabulary semantic queries beyond a fixed label set. A key challenge is enriching such maps with semantic information without relying on manually annotated segmentation masks or closed-set semantic categories. Existing geometry-driven active mapping maximizes coverage but treats semantics as a passive by-product of the collected trajectory, whereas active semantic mapping typically depends on supervised semantic segmentation. We present AOV-GS, an active 3D Gaussian mapping system that guides exploration using frozen foundation models. During coarse exploration, AOV-GS re-ranks geometrically promising candidate views according to CLIP mask-embedding novelty computed from SAM instance segmentation, biasing exploration toward semantically informative regions. Frozen SAM and CLIP provide instance masks and language-aligned embeddings throughout mapping, while a lightweight scene-specific language field is fitted offline directly to the collected CLIP embeddings rather than manually annotated masks, enabling open-vocabulary text-query segmentation on novel views. In a controlled comparison using the same SAM-CLIP perception stack, AOV-GS improves average language-field mIoU over geometry-only active planning by 1.3 percentage points on Replica (34.6% on eight scenes) and by 1.8 points on Matterport3D (23.1% on five scenes). With CorrCLIP, AOV-GS reaches 35.6% on Replica, 2.0 points above the matched Geometry-based + CorrCLIP baseline. Novel-view PSNR remains comparable to the geometry-only baseline on both datasets while depth L1 improves on Replica. Code is available at https://github.com/OptimusFaber/AOV-GS.
Rodion Shkokov, Tatiana Zemskova, Dmitry Yudin
Moscow Institute of Physics and Technology (MIPT), Dolgoprudny, Russia · AIRI, Moscow, Russia