VoxelFix: Post-Hoc Semantic Correction of Completed 3D Voxel Maps
Authors: Sunesh Praveen Raja Sundarasami, Taehyoung Kim, Johannes Scherer, Tomaž Cotič, Sivasubiramaniam Subbiah, Andreas Greiner, Paul Spannaus, Sebastian Houben
Organizations: Fraunhofer IVI · Hochschule Bonn-Rhein-Sieg · University of Bologna · FAU · THI
Semantic 3D maps are increasingly constructed automatically for aerial robotics by integrating learned semantic predictions into 3D representations. While this avoids costly manual 3D annotation, errors in the perception and mapping pipeline can persist in the resulting map, reducing its reliability for downstream autonomous tasks. Existing 3D semantic map refinement methods either rely on the original observations, treat occupancy as part of the prediction problem, or apply non-learned local regularization to completed maps. Instead, we study post-hoc semantic correction, asking whether semantic accuracy can be recovered directly from the completed map while keeping its geometry and occupancy fixed. We introduce \method, a graph-based model that corrects voxel labels based on local geometry and neighboring semantic information. To obtain training pairs, we corrupt contiguous regions of annotated OccuFly maps according to class confusions observed in upstream maps. We evaluate \method on completed OccuFly maps generated from predictions of four independently trained 2D segmentation models. \method consistently improves mIoU by 4.23--5.00 percentage points, with gains broadly distributed across the evaluated semantic classes and particularly strong improvements for tree, roof, and wall. Results on an independently reconstructed out-of-distribution aerial scene further suggest that the learned correction can transfer beyond the environments seen during training.
Figures & tables
Figure 1: Map-only post-hoc semantic correction. One common semantic mapping pipeline predicts 2D semantics and back-projects them using depth and camera poses before aggregating the observations into a semantic voxel map. Upstream perception and mapping errors can persist in the completed map. Our setting begins after map construction: source images, depth maps, camera poses, and intermediate mapping state are unavailable, while voxel geometry and occupancy remain fixed. VoxelFix uses only the completed voxel geometry and semantic labels to correct inconsistent semantics.
Method
No Obs.
No Pose
Map
Fixed
Learned
SemanticFusion [ 32 ]
✗
✗
✗
✓
✗
P2Net [ 33 ]
✗
✗
✗
✓
✓
OccFiner [ 37 ]
✗
✗
✗
✗
✓
ESSC-RM [ 54 ]
✓
✓
✗ †
✗
✓
CRF (baseline)
✓
✓
✓
✓
✗
KNN (baseline)
✓
✓
✓
✓
✗
Table 1: Comparison of approaches for improving semantic map quality. No Obs. and No Pose indicate that original observations and pose information are not required; Map denotes operation on a completed semantic map, and Fixed indicates that geometry and occupancy are not modified.
Figure 2: VoxelFix. Occupied voxels form a fixed k -NN graph whose node descriptors combine local geometry gi , frozen volumetric embeddings zi , current labels y^i , and class-conditioned geometric agreement features ai,δi . The geometric branch attends to geometric edge attributes eij , while the semantic branch additionally uses sij containing class compatibility from the co-occurrence prior A and label-consistency cues. Their outputs are fused by a voxel-wise gate for semantic relabeling. Dashed paths denote training-only corruption, prototype updates, and auxiliary error-detection supervision.
Figure 3: Qualitative map-level correction on the fixed OccuFly test scene. From left to right: uncorrected Radix input, KNN, CRF, geometry heuristic, MinkUNet, VoxelFix, and ground truth. VoxelFix more consistently corrects spatially coherent semantic errors while preserving surrounding labels compared with local smoothing and geometry-only refinement.
Method
mIoU ↑
Δ mIoU ↑
Acc. ↑
ECR ↑
DR ↓
Radix (input)
25.80
–
60.46
–
–
KNN
26.14
+0.34
60.89
5.12
2.64
CRF
26.19
+0.39
61.19
10.65
5.75
Geometry Heuristic
28.60
+2.80
66.27
24.51
6.41
MinkUNet
28.52 ± 1.17
+2.72 ± 1.17
65.83 ± 2.34
16.16 ± 7.09
1.69 ± 0.82
VoxelFix (Ours)
30.66 ± 0.55
+4.86 ± 0.55
68.34 ± 0.62
25.36 ± 1.67
3.55 ± 0.58
Table 2: Map-level semantic correction on the fixed OccuFly test scene using SegFormer-MiT-B3 upstream predictions. Learned models report mean ± std over eight independently trained cross-validation splits. Δ mIoU is measured relative to the uncorrected Radix input map.
Method
Road
Tree
Vehicle
Grass
Wall
Roof
Obstacle
Gravel
Person
Dirt
Bicycle
mIoU
Acc.
Radix (input)
68.53
42.85
42.41
40.25
18.18
52.97
9.37
6.83
0.00
2.43
0.00
25.80
60.46
KNN
69.50
42.49
43.75
40.75
18.16
53.38
10.28
6.91
0.00
2.36
0.00
26.14
60.89
CRF
71.24
41.39
44.71
40.68
17.14
53.53
11.31
6.71
0.00
1.42
0.00
26.19
61.19
Geometry Heuristic
70.73
66.17
31.89
49.07
17.21
52.56
12.02
12.27
0.00
2.71
0.00
28.60
66.27
MinkUNet
70.48 ± 0.18
61.02 ± 8.73
45.14 ± 0.51
46.16 ± 2.97
19.05 ± 1.07
53.65 ± 0.60
8.93 ± 0.23
6.84 ± 0.07
0.00
2.48 ± 0.10
0.00
28.52 ± 1.17
65.83 ± 2.34
Table 3: Per-class IoU (%) and overall performance on the fixed OccuFly test scene using SegFormer-MiT-B3 upstream predictions. Only classes present in the test scene are shown. Best and second-best results are shown in bold and underlined , respectively.
Upstream model
Input mIoU
Corrected mIoU ↑
Δ mIoU ↑
SegFormer-MiT-B3
25.80
30.66 ± 0.55
+4.86 ± 0.55
UPerNet-SwinV2-T
22.64
27.25 ± 0.74
+4.61 ± 0.74
UPerNet-ConvNeXtV2-T
24.88
29.11 ± 0.47
+4.23 ± 0.47
Mask2Former-Swin-T
23.70
28.69 ± 0.41
+5.00 ± 0.41
Table 4: Effect of the upstream segmentation model on map-level correction on the fixed OccuFly test scene. Corrected results show mean ± std over eight cross-validation splits.
Method
mIoU ↑
Δ mIoU ↑
ECR ↑
DR ↓
Radix (input)
30.95
–
–
–
KNN
31.98
+1.04
11.95
1.65
CRF
31.95
+1.00
25.88
3.72
Geometry Heuristic
31.10
+0.16
7.01
1.87
MinkUNet
31.58
+0.63
6.41
0.72
VoxelFix (Ours)
32.66
+1.71
15.65
1.03
Table 5: OOD map-level correction on an independently reconstructed scene using Mask2Former upstream predictions. The scene is not used during training or model selection.
Figure 4: Qualitative map-level correction on the OOD aerial scene using Mask2Former upstream predictions. From left to right: uncorrected Radix input, KNN, CRF, geometry heuristic, MinkUNet, VoxelFix, and ground truth. VoxelFix corrects several semantic inconsistencies on previously unseen scene geometry while largely preserving correctly labeled regions.
Variant
mIoU ↑
Δ mIoU ↑
ECR ↑
w/o UNet embeddings (6D geometry only)
27.43 ± 1.43
+3.18 ± 0.36
12.31 ± 2.01
single-layer GAT
27.65 ± 1.43
+3.39 ± 0.21
13.79 ± 1.60
w/o geometric branch
27.43 ± 1.47
+3.17 ± 0.49
14.77 ± 3.00
w/o gated fusion
27.88 ± 1.47
+3.63 ± 0.39
15.81 ± 2.68
reversed noise curriculum (0.05 → 0.50)
25.80 ± 1.35
+1.55 ± 0.19
6.73 ± 1.24
low-noise curriculum (0.10 → 0.02)
25.41 ± 1.20
+1.15 ± 0.28
4.95 ± 1.16
Table 6: Ablation of the main VoxelFix components on the fixed OccuFly test scene. Results report mean ± std across the four upstream segmentation models, evaluated on a single held-out split of the 7/1/1 protocol.
Parameter
Value
Graph neighbors k
30
Structure-tensor neighbors kgeo
50
Density radius r
3 m
Shared hidden dimension d
32
GATv2 layers per branch
2
Attention heads per layer
2
Table 7: VoxelFix architectural configuration.
Parameter
Value
Optimizer
AdamW
Initial learning rate
1×10−3
Weight decay
1×10−4
Epochs
500 max., early stopping (patience 20)
LR scheduler
ReduceLROnPlateau ( ×0.5 , patience 5)
Batch / subgraph size
1 scene graph / tile
Table 8: VoxelFix training configuration.
Stage
Configuration / time
Hardware
1 × NVIDIA A100 80 GB
STPLS3D pretraining
∼ 8 h
OccuFly fine-tuning
∼ 1 h
VoxelFix training
∼ 3 h per split
Table 9: Training compute and approximate wall-clock cost.
Table 11: STPLS3D pretraining configuration for the geometry encoder.
Parameter
Value
Optimizer
AdamW
Learning rate
5×10−4 , cosine decay after 5-epoch warmup
Weight decay
1×10−5
Number of epochs
200 max., early stop (patience 40)
Loss
Cross-entropy
Class weighting
Effective-number ( β=0.999 ), clipped to [0.25,4.0]
Table 12: OccuFly fine-tuning configuration for the geometry encoder.
Figure 5: Distribution of semantic substitutions for synthetic corruption. The heatmap visualizes the transition probabilities derived from our heuristically constructed class-transition mapping. Plausible substitutions are restricted to the most prominent voxel-level confusions observed in the training scenes, resulting in a targeted prior for realistic synthetic errors.
Parameter
Value
Connectivity / neighborhood definition
Euclidean ball around a random seed voxel (KD-tree radius query), not graph-connected growth
Minimum patch size
radius 2.5 voxels ( 1.25 m)
Maximum patch size
radius 7.5 voxels ( 3.75 m)
Patch-size distribution
radius =5×u , u∼U(0.5,1.5)
Maximum patches per map
⌊Ntarget/50⌋ , Ntarget=rate×∣V∣
Overlapping patches allowed
Yes; patch centers sampled independently, no overlap exclusion
Table 13: Synthetic corruption parameters.
Figure 6: Examples of confusion-guided synthetic corruptions used for training. Spatially contiguous voxel regions are selected from clean semantic maps and relabeled according to class-transition probabilities derived from voxel-level upstream confusion statistics. Unlike independent label flipping, the resulting corruptions form coherent erroneous regions resembling those observed in completed upstream maps.
Dataset
unlabeled
road
dirt
gravel
rock
grass
vegetation
tree
ground-obstacle
animals
person
bicycle
vehicle
water
boat
building
roof
sky
drone
train-track
cable
cable-tower
wind-turbine-blade
wind-turbine-tower
walkway
bridge
airplanes
CART [ 24 ]
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
ICG [ 20 ]
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
OkutSwiss [ 38 ]
✓
✓
✓
✓
✓
✓
✓
✓
✓
UAVid [ 31 ]
✓
✓
✓
✓
✓
✓
✓
✓
✓
UDD [ 6 ]
✓
✓
✓
✓
✓
✓
✓
VDD [ 3 ]
✓
✓
✓
✓
✓
✓
✓
Table 14: Semantic class coverage across the datasets used to train the upstream segmentation models. The unlabeled class is mapped to the ignore index and is not part of the 26-class taxonomy.
Method
Backbone
mIoU
Val
Unseen
SegFormer [ 51 ]
MiT-B3
66.56
38.91
UPerNet [ 50 ]
Swin-v2-T
66.53
38.11
UPerNet [ 50 ]
ConvNeXt-v2-T
69.23
38.05
Mask2Former [ 7 ]
Swin-T
69.55
36.60
Table 15: Performance of the upstream segmentation models on the validation and unseen datasets.
Table 16: Semantic taxonomy alignment from upstream segmentation to VoxelFix. The 26-class upstream predictions are first aligned with the OccuFly taxonomy and subsequently reduced to the 12 classes used for VoxelFix training and evaluation. Classes without a corresponding target class are ignored.
Parameter
Value
Number of neighbors
5
Distance metric
Euclidean, in metres
Include query voxel
Yes (counted within k=5 )
Distance weighting
None (uniform vote)
Number of iterations
1
Tie-breaking rule
Keep the voxel’s current label
Table 17: KNN baseline parameters.
Parameter
Value
Unary potential
Upstream scalar confidence on the input label, remaining mass uniform over the other 11 classes
Pairwise neighborhood k
30 (self excluded)
Spatial standard deviation σ
1.5 m
Edge weighting
Gaussian, exp(−d2/2σ2) , row-normalized
Pairwise weight (Potts)
15.0
Compatibility term
Potts: class-independent, uniform penalty for unlike labels
KNN majority vote over valid ground classes; k=50 , widened to k=100 when too few valid neighbors are found
Above-ground clustering
DBSCAN, ϵ=0.5 m, minimum 5 samples
Cluster relabeling
Majority vote within each cluster; dense components merged into a single cluster
Table 19: Geometry-based heuristic parameters.
Figure 7: Qualitative correction on held-out OccuFly validation scenes using SegFormer-MiT-B3 upstream predictions. Each scene is corrected by the model of the split in which it was held out. From left to right: uncorrected Radix input, VoxelFix, and ground truth.
12-D one-hot of the input label map only (no geometry, no embeddings)
Voxel size
0.5 m
Optimizer
AdamW
Learning rate
1×10−3 , ReduceLROnPlateau ( ×0.5 , patience 5)
Weight decay
1×10−4
Table 20: MinkUNet baseline configuration.
Method
Road
Tree
Vehicle
Grass
Wall
Roof
Obstacle
Gravel
Person
Dirt
mIoU
Radix (input)
76.57
5.65
23.07
87.41
34.92
54.92
21.68
0.00
5.00
0.28
30.95
KNN
78.75
6.52
26.33
88.50
35.75
56.32
23.25
0.00
3.95
0.47
31.98
CRF
80.35
8.91
30.56
90.31
32.38
57.84
19.13
0.00
0.00
0.00
31.95
Geometry Heuristic
78.94
5.14
25.90
85.67
33.68
56.50
19.94
0.00
5.00
0.27
31.10
MinkUNet
79.18
6.03
26.53
87.35
34.92
55.01
20.72
0.00
5.75
0.28
31.58
Table 21: Per-class IoU (%) on the independently reconstructed OOD scene using single-split Mask2Former upstream predictions. Only classes present in the scene are shown; mIoU is averaged over these 10 classes and is therefore not directly comparable to the in-distribution values in Tab. 3 . Best and second-best results are shown in bold and underlined , respectively.
Figure 8: Qualitative correction on the independently reconstructed OOD scene using Mask2Former upstream predictions. From left to right: uncorrected Radix input, VoxelFix, and ground truth. Wall and roof regions are recovered along the building structures, and spurious water labels on the grass surface are removed. Remaining errors concentrate in structures for which the completed map carries no distinguishing evidence.
Split
Training scenes
Validation scene
Fixed test scene
1
1, 2, 7, 3, 9, 4, 5
6
8
2
6, 2, 7, 3, 9, 4, 5
1
8
3
6, 1, 7, 3, 9, 4, 5
2
8
4
6, 1, 2, 3, 9, 4, 5
7
8
5
6, 1, 2, 7, 9, 4, 5
3
8
6
6, 1, 2, 7, 3, 4, 5
9
8
Table 22: OccuFly train/validation/test composition for the eight 7/1/1 cross-scene splits.
Open-vocabulary 3D semantic maps have emerged as a key spatial memory representation for long-horizon robot autonomy, enabling robots to reason about objects, their spatial relationships, and past observations across time. However, existing object-centric semantic maps are primarily designed for static environments and rely on an append-only point integration strategy, where newly observed points are accumulated but previously integrated points are rarely reevaluated. Consequently, perception errors accumulate over time, and outdated points persist after objects move or disappear. Although recent dynamic mapping methods update object-level representations in changing scenes, they do not explicitly revise noisy or obsolete points within each object. We propose an online refinable semantic mapping framework that continuously refines object representations through probabilistic point-level confidence updating. Each object point maintains a confidence score that is updated online using multi-view temporal evidence: consistent observations reinforce reliable points, while inconsistent segmentation masks, depth measurements, and visibility cues suppress spurious and outdated points. Combined with object tracking and re-identification to preserve instance identities under motion and relocation, our framework enables object-centric semantic maps to be continuously refined rather than monotonically accumulated. Experiments demonstrate improved mapping accuracy over the previous method by 3.46% on Replica and 2.55% on ScanNet, show robustness in dynamic environments and under perception noise, and provide an effective spatial memory for object navigation and embodied question-answering with success rates of 82.4% and 76.7%.
Taimeng Fu, Yi Du, Zitong Zhan +4
Spatial AI & Robotics Lab, Department of Computer Science and Engineering, University at Buffalo, USA.
Semantic occupancy mapping supports embodied perception during exploration and detailed scene understanding afterward, giving rise to two complementary requirements: online incremental mapping for maintaining an up-to-date global map under causal constraints, and offline global map refinement for improving the completed scene representation with additional computation. We present VEOcc++, a unified voxel-centric framework that connects both stages through a persistent sparse voxel feature map. During exploration, the map expands with newly observed regions and integrates aligned local features through reliability-guided updates, while context-aware decoding produces the current semantic occupancy map. After exploration, cached local features are consolidated as complementary evidence and combined with the completed online representation for multi-scale spatial refinement. On EmbodiedOcc-ScanNet, VEOcc++ achieves 67.12% IoU and 58.40% mIoU, establishing new state-of-the-art performance for online incremental mapping. With the local predictor fixed, our online mapping pipeline outperforms VEOcc fusion by 3.70 IoU and 3.74 mIoU points, demonstrating substantial gains from the proposed global mapping design. Offline global map refinement further improves the completed map to 67.79% IoU and 59.37% mIoU using only the observations acquired during exploration. Zero-shot deployment on a mobile robot further demonstrates transferability to unseen environments, onboard feasibility, and scalability to large-scale indoor mapping. Project page: https://wryzju.github.io/VEOccpp/.
Ruoyu Wang, Jiahan Li, Yong Liu +1
APRIL Lab, Institute of Cyber-Systems and Control, Zhejiang University, China
Open-vocabulary semantic mapping enables robots to spatially ground previously unseen concepts without requiring predefined class sets. Current training-free methods commonly rely on multi-view fusion of semantic embeddings into a 3D map, either at the instance-level via segmenting views and encoding image crops of segments, or by projecting image patch embeddings directly into a dense semantic map. The latter approach sidesteps segmentation and 2D-to-3D instance association by operating on full uncropped image frames, but existing methods remain limited in scalability. We present FUS3DMaps, an online dual-layer semantic mapping method that jointly maintains both dense and instance-level open-vocabulary layers within a shared voxel map. This design enables further voxel-level semantic fusion of the layer embeddings, combining the complementary strengths of both semantic mapping approaches. We find that our proposed semantic cross-layer fusion approach improves the quality of both the instance-level and dense layers, while also enabling a scalable and highly accurate instance-level map where the dense layer and cross-layer fusion are restricted to a spatial sliding window. Experiments on established 3D semantic segmentation benchmarks as well as a selection of large-scale scenes show that FUS3DMaps achieves accurate open-vocabulary semantic mapping at multi-story building scales. Additional material and code will be made available: https://githanonymous.github.io/FUS3DMaps/.
Timon Homberger, Finn Lukas Busch, Jesús Gerardo Ortega Peimbert +2
Division of Robotics, Perception, and Learning, KTH Royal Institute of Technology, Sweden