VoxSAOR: Semantic-Aware Occupancy Robustification via Aleatoric Uncertainty Modeling
Organizations: The authors are with the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha, China
Abstract
Vision-based 3D occupancy prediction provides a dense semantic representation of scenes for autonomous driving and embodied perception, yet remains vulnerable to occlusion, depth ambiguity, and projection errors that induce spatially varying reliability in voxel features. Existing methods typically map voxel representations to semantic predictions without explicitly modeling such observation-induced feature variations. To address this limitation, we propose VoxSAOR, a semantic-aware occupancy robustification framework that models aleatoric uncertainty through feature-space proxies. It first quantifies Local Semantic Consistency (LSC) to derive a voxel-wise structural stability prior and uses cross-view semantic consensus to filter unreliable semantic correspondences. Within each semantic category, spatially matched stable and ambiguous voxels form stable-to-ambiguous feature transitions, which are modeled by a class-conditional diagonal Gaussian distribution, capturing both feature displacement and dispersion. Guided by these proxy distributions, Aleatoric Regime Ensemble Learning (AREL) trains independently parameterized prediction heads using feature perturbations at progressively increasing scales, with perturbation strength modulated by structural reliability. At inference, proxy modeling and sampling are disabled, and all heads process the same observed features; their output probabilities are then averaged to produce a deterministic consensus prediction without requiring sampling. Experiments across multiple datasets and occupancy backbones demonstrate the effectiveness of VoxSAOR in improving semantic occupancy prediction, long-tail recognition, and predictive calibration. The source code will be made publicly available at https://github.com/MengfeiD/VoxSAOR.
Figures & tables
| Method | Arch. | T | IoU | mIoU | road (15.30%) | sidewalk (11.13%) | parking (1.12%) | other-grnd. (0.56%) | building (14.1%) | car (3.92%) | truck (0.16%) | bicycle (0.03%) | motorcycle (0.03%) | other-veh. (0.20%) | vegetation (39.3%) | trunk (0.51%) | terrain (9.17%) | person (0.07%) | bicyclist (0.07%) | motorcyclist (0.05%) | fence (3.90%) | pole (0.29%) | traf.-sign (0.08%) |
| MonoScene [ 2 ] | Eff-B7 | 34.16 | 11.08 | 54.70 | 27.10 | 24.80 | 5.70 | 14.40 | 18.80 | 3.30 | 0.50 | 0.70 | 4.40 | 14.90 | 2.40 | 19.50 | 1.00 | 1.40 | 0.40 | 11.10 | 3.30 | 2.10 | |
| TPVFormer [ 24 ] | Eff-B7 | 34.25 | 11.26 | 55.10 | 27.20 | 27.40 | 6.50 | 14.80 | 19.20 | 3.70 | 1.00 | 0.50 | 2.30 | 13.90 | 2.60 | 20.40 | 1.10 | 2.40 | 0.30 | 11.00 | 2.90 | 1.50 | |
| SurroundOcc [ 6 ] | Eff-B7 | 34.72 | 11.86 | 56.90 | 28.30 | 30.20 | 6.80 | 15.20 | 20.60 | 1.40 | 1.60 | 1.20 | 4.40 | 14.90 | 3.40 | 19.30 | 1.40 | 2.00 | 0.10 | 11.30 | 3.90 | 2.40 | |
| OccFormer [ 4 ] | Eff-B7 | 34.53 | 12.32 | 55.90 | 30.30 | 31.50 | 6.50 | 15.70 | 21.60 | 1.20 | 1.50 | 1.70 | 3.20 | 16.80 | 3.90 | 21.30 | 2.20 | 1.10 | 0.20 | 11.90 | 3.80 | 3.70 | |
| IAMSSC [ 66 ] | R-50 | 43.74 | 12.37 | 54.00 | 25.50 | 24.70 | 6.90 | 19.20 | 21.30 | 3.80 | 1.10 | 0.60 | 3.90 | 22.70 | 5.80 | 19.40 | 1.50 | 2.90 | 0.50 | 11.90 | 5.30 | 4.10 | |
| VoxFormer [ 3 ] | R-50 | 42.95 | 12.20 | 53.90 | 25.30 | 21.10 | 5.60 | 19.80 | 20.80 | 3.50 | 1.00 | 0.70 | 3.70 | 22.40 | 7.50 | 21.30 | 1.40 | 2.60 | 0.20 | 11.10 | 5.10 | 4.90 |
| Method | T | IoU | mIoU | road (15.30%) | sidewalk (11.13%) | parking (1.12%) | other-grnd. (0.56%) | building (14.1%) | car (3.92%) | truck (0.16%) | bicycle (0.03%) | motorcycle (0.03%) | other-veh. (0.20%) | vegetation (39.3%) | trunk (0.51%) | terrain (9.17%) | person (0.07%) | bicyclist (0.07%) | motorcyclist (0.05%) | fence (3.90%) | pole (0.29%) | traf.-sign (0.08%) |
| MonoScene [ 2 ] | 36.86 | 11.08 | 56.52 | 26.72 | 14.27 | 0.46 | 14.09 | 23.26 | 6.98 | 0.61 | 0.45 | 1.48 | 17.89 | 2.81 | 29.64 | 1.86 | 1.20 | 0.00 | 5.84 | 4.14 | 2.25 | |
| TPVFormer [ 24 ] | 35.61 | 11.36 | 56.50 | 25.87 | 20.60 | 0.85 | 13.88 | 23.81 | 8.08 | 0.36 | 0.05 | 4.35 | 16.92 | 2.26 | 30.38 | 0.51 | 0.89 | 0.00 | 5.94 | 3.14 | 1.52 | |
| OccFormer [ 4 ] | 36.50 | 13.46 | 58.85 | 26.88 | 19.61 | 0.31 | 14.40 | 25.09 | 25.53 | 0.81 | 1.19 | 8.52 | 19.63 | 3.93 | 32.62 | 2.78 | 2.82 | 0.00 | 5.61 | 4.26 | 2.86 | |
| IAMSSC [ 66 ] | 44.29 | 12.45 | 54.55 | 25.85 | 16.02 | 0.70 | 17.38 | 26.26 | 8.74 | 0.60 | 0.15 | 5.06 | 24.63 | 4.95 | 30.13 | 1.32 | 3.46 | 0.01 | 6.86 | 6.35 | 3.56 | |
| VoxFormer [ 3 ] | 44.02 | 12.35 | 54.76 | 26.35 | 15.50 | 0.70 | 17.65 | 25.79 | 5.63 | 0.59 | 0.51 | 3.77 | 24.39 | 5.08 | 29.96 | 1.78 | 3.32 | 0.00 | 7.64 | 7.11 | 4.18 | |
| VoxFormer [ 3 ] | ✓ | 44.15 | 13.35 | 53.57 | 26.52 | 19.69 | 0.42 | 19.54 | 26.54 | 7.26 | 1.28 | 0.56 | 7.81 | 26.10 | 6.10 | 33.06 | 1.93 | 1.97 | 0.00 | 7.31 | 9.15 | 4.94 |
| Method | IoU | mIoU | car (2.85%) | bicycle (0.01%) | motorcycle (0.01%) | truck (0.16%) | other-veh. (5.75%) | person (0.02%) | road (14.98%) | parking (2.31%) | sidewalk (6.43%) | other-grnd. (2.05%) | building (15.67%) | fence (0.96%) | vegetation (41.99%) | terrain (7.10%) | pole (0.22%) | traf.-sign (0.06%) | other-struct. (4.33%) | other-obj. (0.28%) |
| LiDAR-based methods | ||||||||||||||||||||
| SSCNet [ 77 ] | 53.58 | 16.95 | 31.95 | 0.00 | 0.17 | 10.29 | 0.00 | 0.07 | 65.70 | 17.33 | 41.24 | 3.22 | 44.41 | 6.77 | 43.72 | 28.87 | 0.78 | 0.75 | 8.69 | 0.67 |
| LMSCNet [ 78 ] | 47.35 | 13.65 | 20.91 | 0.00 | 0.00 | 0.26 | 0.58 | 0.00 | 62.95 | 13.51 | 33.51 | 0.20 | 43.67 | 0.33 | 40.01 | 26.80 | 0.00 | 0.00 | 3.63 | 0.00 |
| Camera-based methods | ||||||||||||||||||||
| MonoScene [ 2 ] | 37.87 | 12.31 | 19.34 | 0.43 | 0.58 | 8.02 | 2.03 | 0.86 | 48.35 | 11.38 | 28.13 | 3.32 | 32.89 | 3.53 | 26.15 | 16.75 | 6.92 | 5.67 | 4.20 | 3.09 |
| TPVFormer [ 24 ] | 40.22 | 13.64 | 21.56 | 1.09 | 1.37 | 8.06 | 2.57 | 2.38 | 52.99 | 11.99 | 31.07 | 3.78 | 34.83 | 4.80 | 30.08 | 17.52 | 7.46 | 5.86 | 5.48 | 2.70 |
| HOMO (within-city) | ||||||||||||
| Method | IoU | mIoU | road (34.04%) | sidewalk (26.24%) | building (10.00%) | vegetation (24.02%) | car (2.92%) | truck (0.99%) | bus (0.25%) | two_wheeler (0.07%) | person (0.97%) | pole (0.50%) |
| VoxFormer - S [ 3 ] | 31.57 | 11.09 | 22.27 | 17.44 | 2.86 | 5.65 | 0.87 | 0.14 | 0.00 | 0.00 | 61.46 | 0.18 |
| SGN - T [ 25 ] | 46.05 | 28.64 | 54.55 | 44.61 | 15.51 | 35.16 | 23.91 | 7.68 | 4.52 | 15.77 | 65.24 | 19.46 |
| SGN - S [ 25 ] | 45.98 | 28.56 | 54.81 | 47.44 | 15.64 | 33.22 | 24.23 | 8.53 | 2.62 | 13.51 | 65.88 | 19.72 |
| OccFormer [ 4 ] | 45.98 | 24.88 | 54.11 | 46.41 | 15.14 | 32.84 | 17.57 | 4.30 | 0.00 | 1.65 | 65.08 | 11.72 |
| OneOcc [ 16 ] | 55.73 | 37.29 | 63.82 | 62.92 | 22.86 | 41.03 | 33.74 | 17.79 | 6.47 | 26.58 | 71.13 | 26.55 |
| Method | IoU | mIoU | Tail mIoU | ECE | NLL |
| SGN [ 25 ] | 43.60 | 14.55 | 4.07 | 0.2309 | 1.5635 |
| VoxSAOR (+SGN) | 44.08 | 14.79 | 4.19 | 0.2301 | 1.4749 |
| CGFormer [ 28 ] | 45.99 | 16.87 | 6.22 | 0.2212 | 1.2581 |
| VoxSAOR (+CGFormer) | 45.64 | 17.58 | 6.84 | 0.1972 | 0.9619 |
| VoxDet [ 1 ] | 47.36 | 18.73 | 8.47 | 0.1626 | 0.8397 |
| VoxSAOR (+VoxDet) | 47.70 | 19.19 | 9.04 | 0.1621 | 0.8298 |
| SAUM | AREL | mIoU | IoU |
| 18.59 | 47.51 | ||
| ✓ | 18.88 | 47.73 | |
| ✓ | 18.53 | 47.51 | |
| ✓ | ✓ | 19.19 | 47.70 |
| CSC | LSC | Semantic | Spatial | mIoU | IoU |
| ✓ | ✓ | ✓ | 18.45 | 47.71 | |
| ✓ | ✓ | 18.46 | 47.63 | ||
| ✓ | ✓ | ✓ | 18.88 | 47.51 | |
| ✓ | ✓ | ✓ | 18.62 | 47.58 | |
| ✓ | ✓ | ✓ | 18.27 | 47.62 | |
| ✓ | ✓ | ✓ | ✓ | 19.19 | 47.70 |
| mIoU | IoU | ECE | NLL | #Params ( M ) | |
| 1 | 18.88 | 47.73 | 0.1636 | 0.8413 | 55.34 |
| 2 | 19.19 | 47.70 | 0.1621 | 0.8298 | 55.56 |
| 3 | 18.84 | 47.71 | 0.1559 | 0.8102 | 55.78 |
| 4 | 18.62 | 47.59 | 0.1623 | 0.8267 | 56.01 |
| mIoU | IoU | |
| 0.005 | 18.60 | 47.73 |
| 0.01 | 19.19 | 47.70 |
| 0.02 | 18.95 | 47.61 |
| 0.03 | 18.79 | 48.03 |
| 0.04 | 18.65 | 47.35 |
| mIoU | IoU | ||
| 18.43 | 47.77 | ||
| ✓ | 18.38 | 47.36 | |
| ✓ | ✓ | 19.19 | 47.70 |
| mIoU | IoU | mIoU | IoU | ||
| 2 | 18.93 | 47.76 | 24 | 18.62 | 47.42 |
| 4 | 19.19 | 47.70 | 28 | 18.80 | 47.82 |
| 6 | 18.83 | 47.58 | 32 | 19.19 | 47.70 |
| 8 | 18.73 | 47.71 | 36 | 18.51 | 47.86 |
| 10 | 18.58 | 47.86 | 40 | 18.35 | 47.45 |
3D Occupancy Prediction
3D occupancy prediction estimates the geometry and semantics of occupied and free space around an agent from camera, LiDAR, or radar data.
See all 125 papers