Open-vocabulary 3D semantic maps have emerged as a key spatial memory representation for long-horizon robot autonomy, enabling robots to reason about objects, their spatial relationships, and past observations across time. However, existing object-centric semantic maps are primarily designed for static environments and rely on an append-only point integration strategy, where newly observed points are accumulated but previously integrated points are rarely reevaluated. Consequently, perception errors accumulate over time, and outdated points persist after objects move or disappear. Although recent dynamic mapping methods update object-level representations in changing scenes, they do not explicitly revise noisy or obsolete points within each object. We propose an online refinable semantic mapping framework that continuously refines object representations through probabilistic point-level confidence updating. Each object point maintains a confidence score that is updated online using multi-view temporal evidence: consistent observations reinforce reliable points, while inconsistent segmentation masks, depth measurements, and visibility cues suppress spurious and outdated points. Combined with object tracking and re-identification to preserve instance identities under motion and relocation, our framework enables object-centric semantic maps to be continuously refined rather than monotonically accumulated. Experiments demonstrate improved mapping accuracy over the previous method by 3.46% on Replica and 2.55% on ScanNet, show robustness in dynamic environments and under perception noise, and provide an effective spatial memory for object navigation and embodied question-answering with success rates of 82.4% and 76.7%.
Figures & tables
Fig. 1 : Dynamic open-vocabulary semantic mapping. Our method maintains a refinable object-centric 3D semantic map through point-level probabilistic updates. It handles dynamic objects that newly appear, move within the robot’s view, or are relocated out of view and later re-identified.
Fig. 2 : Overview of the proposed dynamic semantic mapping framework. Given posed RGB-D observations, open-vocabulary detections are generated and associated with object instances in the 3D semantic map. For each association, we perform a visibility-based point confidence update. Points supported by current detections, such as the green points on the table, receive higher confidence. In contrast, inconsistent outlier points, such as red points mistakenly fused into a bed instance due to previous segmentation errors, are down-weighted when later observations identify them as pillows. Points belonging to removed objects are identified through visibility reasoning, assigned lower confidence, and eventually pruned. Unobserved points in the current frame, shown in gray, remain unchanged until more evidence is collected in later frames. The semantic map is updated online in this frame-by-frame manner, while periodic instance-fragment merging reduces duplicate instances and re-identification preserves object identities when objects are relocated and re-observed at new locations.
Dataset
Method
mIoU
F-mIoU
mAcc
Replica
ConceptGraphs
0.1501
0.3858
0.3559
HOV-SG
0.2050
0.4846
0.3835
DualMap
0.2538
0.5207
0.4024
Ours
0.2629
0.5530
0.4671
ScanNet
ConceptGraphs
0.0882
0.3077
0.3538
HOV-SG
0.1333
0.3381
0.3714
TABLE I: Evaluation results of open-vocabulary 3D semantic mapping methods on Replica and ScanNet datasets.
Dataset
Method
Memory (MB)
TPF (s)
Avg. Points / Scene
CPU Avg.
CPU Peak
GPU Avg.
GPU Peak
Replica
DualMap
5391.5
7813.3
7921.0
12248.0
0.2219
255,234
Ours
4492.6
4966.7
5136.4
7002.0
0.1811
209,617
ScanNet
DualMap
4668.5
6027.9
6596.8
8608.0
0.1544
218,785
Ours
4651.4
5835.7
5766.6
7574.0
0.1324
194,454
TABLE II: Computational efficiency and final map size.
Method
mIoU
F-mIoU
mAcc
Duplication
w/o confidence update
0.2529
0.5384
0.4629
1.2568
w/o fragment merging
0.2502
0.5479
0.4539
1.3426
Ours
0.2629
0.5530
0.4671
1.1690
TABLE III: Ablation of point-level confidence updating and fragment merging on the Replica dataset.
Method
Δ mIoU
Δ F-mIoU
Δ mAcc
DualMap
-11.62%
-3.06%
-10.13%
Ours
-6.29%
-3.04%
-7.77%
TABLE IV: Impact of simulated segmentation errors on semantic mapping performance on the Replica dataset.
Fig. 3 : Example of handling inaccurate masks. A wooden chair with thin and hollow structures is observed from multiple viewpoints. Inaccurate masks may fill the gaps between chair parts, introducing outlier points when fused into 3D. DualMap retains these mask-induced artifacts, while our point-level confidence update suppresses inconsistent points across views and reconstructs cleaner chair geometry.
Fig. 4 : Example of handling incorrect detections. A plastic bag is mistakenly detected as a pillow, creating a false instance in the map. Our confidence update mechanism accumulates positive evidence for consistently detected objects and negative evidence for unsupported instances, eventually removing the false pillow while preserving the true one.
Fig. 5 : Example of object fragment merging. A table is initially reconstructed as two separate instances due to temporally disconnected partial observations at 1s and 10s. As more observations become available at 12s, our method merges the fragments into a complete object instance.
Fig. 6 : Example of handling under-segmentation. A pillow is initially fused into the chair instance due to under-segmentation. Once the pillow is correctly segmented in later frames, our method adds it as a separate instance and gradually removes the corresponding points from the chair instance through confidence updates.
Fig. 7 : Example of in-view object tracking. Our method tracks the moving bowl in images and updates its location in the semantic map while preserving the same instance ID.
Method
Moved Objs.
Found Old
Removed Old
Found New
Succ. ReID
DualMap
26
12 (46.2%)
10 (38.5%)
15 (57.7%)
–
Ours
26 (100%)
26 (100%)
24 (92.3%)
20 (76.9%)
TABLE V: Comparison of out-of-view object relocation.
Fig. 8 : Real-world VLN evaluation under environment changes. After objects are moved, the agent executes 7 object navigation tasks in sequence. Our full system travels 76.41 m, whereas the ablation without map refinement travels 103.22 m because it revisits stale object locations.
Fig. 9 : Examples of object navigation and EQA tasks executed on the Spot robot. Our semantic map serves as an online memory that helps the LLM localize target objects, plan embodied skills, and answer questions.
Object Navigation
EQA
Scenes
Trials
SR % ( ↑ )
SPL ( ↑ )
Trials
SR % ( ↑ )
SPL ( ↑ )
Hallway
11
72.7
0.603
9
66.7
0.532
Office
12
91.7
0.790
12
83.3
0.714
Outdoor
11
81.8
0.685
9
77.8
0.641
All
34
82.4
0.696
30
76.7
0.638
TABLE VI: Performance on real-world object navigation and embodied question answering tasks.
Continuous semantic mapping allows autonomous robots to understand both the spatial structure and the semantic content of complex environments. However, most existing methods process the entire space, treat voxels as independent units, and do not keep the semantic labels consistent over time. This leads to high computational cost and reduced robustness in dynamic scenes. This paper proposes a semantic mapping method that brings spatial and temporal relationships into the semantic inference process. The method adjusts the inference range according to the local semantic uncertainty and fuses labels over time to improve map stability and computational efficiency. Experiments on the SemanticKITTI dataset show that the proposed method improves mapping accuracy by about 12% and reaches an mIoU of 54.92%, which is 13.18 percentage points higher than spatial-only mapping. These results show that spatiotemporal reasoning is effective for continuous semantic mapping in autonomous robotic systems.
My Le Pham, Dinh Trieu Duong, Xiem HoangVan +1
University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan.
Semantic occupancy mapping supports embodied perception during exploration and detailed scene understanding afterward, giving rise to two complementary requirements: online incremental mapping for maintaining an up-to-date global map under causal constraints, and offline global map refinement for improving the completed scene representation with additional computation. We present VEOcc++, a unified voxel-centric framework that connects both stages through a persistent sparse voxel feature map. During exploration, the map expands with newly observed regions and integrates aligned local features through reliability-guided updates, while context-aware decoding produces the current semantic occupancy map. After exploration, cached local features are consolidated as complementary evidence and combined with the completed online representation for multi-scale spatial refinement. On EmbodiedOcc-ScanNet, VEOcc++ achieves 67.12% IoU and 58.40% mIoU, establishing new state-of-the-art performance for online incremental mapping. With the local predictor fixed, our online mapping pipeline outperforms VEOcc fusion by 3.70 IoU and 3.74 mIoU points, demonstrating substantial gains from the proposed global mapping design. Offline global map refinement further improves the completed map to 67.79% IoU and 59.37% mIoU using only the observations acquired during exploration. Zero-shot deployment on a mobile robot further demonstrates transferability to unseen environments, onboard feasibility, and scalability to large-scale indoor mapping. Project page: https://wryzju.github.io/VEOccpp/.
Ruoyu Wang, Jiahan Li, Yong Liu +1
APRIL Lab, Institute of Cyber-Systems and Control, Zhejiang University, China
Dense 3D mapping with semantic understanding is essential for robotic perception in complex environments. Recent 3D Gaussian Splatting-based mapping approaches enable high-fidelity geometry and efficient open-vocabulary perception, but typically represent semantics as unstructured feature fields that limit object-centric reasoning. In contrast, 3D scene graphs explicitly model objects and their relationships for structured reasoning, but are commonly constructed from sparse geometric representations that do not fully exploit dense semantic maps. In this work, we present OpenSplatGraph, a unified framework that constructs persistent 3D scene graphs directly from an online Gaussian-based open-vocabulary semantic map. The proposed framework augments the dense semantic map with a reliability-aware semantic field that maintains lightweight observation statistics for confidence-aware, query-conditioned object extraction. Extracted object instances are associated with persistent graph nodes, allowing object attributes and relationships to be incrementally updated across observations and queries. By tightly coupling dense semantic mapping with persistent object-centric representations, our framework supports both language-guided object grounding and structured relational reasoning while preserving the geometric fidelity of Gaussian-based mapping. Comprehensive evaluations on standard 3D scene understanding benchmarks and real-world robotic experiments demonstrate that OpenSplatGraph achieves competitive performance for online open-vocabulary perception and downstream robotic tasks. Project page: https://csiro-robotics.github.io/OpenSplatGraph.
Binh Long Nguyen, Kien Nguyen, Clinton Fookes +1
School of Electrical Engineering and Robotics, Queensland University of Technology (QUT), Brisbane, QLD 4000, Australia · CSIRO Robotics, CSIRO, Brisbane, QLD 4069, Australia