LiNO: Lightweight Network for Occupancy Estimation
Authors: Ruan Bispo, Tim Brophy, Reenu Mohandas, Anthony Scanlan, Ciaran Eising
Organizations: The authors are with the Data Driven Computer Engineering Research Centre, Department of Electronic and Computer Engineering, University of Limerick, Limerick V94 T9PX, Ireland, and also with Lero, the Research Ireland Centre for Software, University of Limerick, Limerick V94 NYD3, Ireland.
Occupancy estimation for automated vehicles is a crucial task that builds a 3D map using vehicle sensors. This map allows the vehicle to identify drivable areas and various obstacle classes. The task involves complex subtasks such as depth estimation, object classification, and segmentation to align semantic information with the 3D grid. Moreover, producing these estimations from multiple sensors demands significant computational power and memory, as high-dimensional data fusion requires complex feature extraction and processing. Consequently, most existing methods rely on large and computationally intensive architectures that are not suitable for real-time or real-world applications. We present a real-time, lightweight architecture that leverages camera and radar sensors through a residual multi-stage head. Unlike prior methods that rely on heavy camera-only blocks, our architecture employs camera-radar fusion and lightweight components. Additionally, we introduce a task-specific data augmentation strategy that enhances data diversity and improves generalisation. Our method achieves state-of-the-art (SOTA) performance, attaining 45.7 mean Intersection over Union (mIoU) on Occ3D and 41.9 Ray-level Intersection over Union (RayIoU) (41.9) on OpenOcc datasets, running at 17.2 frames per second (FPS) and using only 5.3 GB of memory, compared with 4.1 FPS and 7.7 GB for STCOcc.
Figures & tables
Fig. 1: Our proposed architecture using camera and radar data. Firstly, the camera and radar features are extracted using the respective feature extractors. Secondly, we lift and fuse these features into a BEV using a BEV Transformation and fusion module. Finally, the proposed occupancy head (highlighted in blue) processes the BEV features to produce the 3D occupancy estimation.
Figure 2
Fig. 4: Proposed data augmentation based on morphological closing to fill empty regions.
Method
Backb.
Input
M
mIoU ↑
others
barrier
bike
bus
car
constrv
mcycle
ped
tcone
trailer
truck
drivsurf
othflat
sidewalk
terrain
manmade
veg
BEVFormer [ 7 ]
R101
1600×900
C
26.9
5.9
37.8
17.9
40.4
42.4
7.4
23.9
21.8
21.0
22.4
30.7
55.4
28.4
36.0
28.1
20.0
17.7
CTF-Occ [ 45 ]
R101
1600×900
C
28.5
8.1
39.3
20.6
38.3
42.2
16.9
24.5
22.7
21.1
23.0
31.1
53.3
33.8
38.0
33.2
20.8
18.0
TPVFormer [ 19 ]
R101
1600×900
C
27.8
7.2
38.9
13.7
40.8
45.9
17.2
20.0
18.9
14.3
26.7
34.2
55.7
35.5
37.6
30.7
19.4
16.8
OSP* [ 50 ]
R101
1600×900
C
41.2
10.9
49.0
27.7
50.2
55.9
22.9
31.0
30.9
30.3
35.6
31.2
82.1
42.6
51.9
55.1
44.8
38.2
SparseOcc [ 17 ]
R50
704×256
C
30.9
10.6
39.2
20.2
32.9
43.3
19.4
23.8
23.4
29.3
21.4
29.3
67.7
36.3
44.6
40.9
22.0
21.9
FlashOcc(M5)* [ 26 ]
R50
704×256
C
38.8
11.0
47.2
22.2
41.4
50.6
23.2
23.0
23.8
26.2
32.1
37.4
82.2
42.5
53.3
56.6
46.7
39.7
TABLE I: Comparison of mIoU performance on the Occ3D [ 45 ] dataset. * indicates models trained with camera masks. M denotes the input modality: C for camera, R for radar, and CR for both. Higher values of IoU indicate better performance.
Method
Backb.
Input
Venue
RayIoU 1m ↑
RayIoU 2m ↑
RayIoU 4m ↑
RayIoU ↑
VRAM ↓
FPS ↑
BEVFormer (4f) [ 7 ]
R101
1600×900
PAMI 2024
26.1
32.9
38.0
32.4
24.1 GB
3.0
RenderOcc (6f) [ 47 ]
Swin-B
1408×512
ICRA 2024
13.4
19.6
25.5
19.5
17.5 GB
-
BEVDet-Occ (8f) [ 22 ]
R50
704×384
arXiv 2022
26.6
33.1
38.2
32.6
10.2 GB
2.6
FB-Occ (16f) [ 6 ]
R50
704×256
ICCV 2023
26.7
34.1
39.7
33.5
9.6 GB
10.3
TEOcc-RC (9f) [ 8 ]
R50
704×256
arXiv 2024
32.0
36.2
41.2
35.8
-
1.1
SparseOcc (16f) [ 17 ]
R50
704×256
ECCV 2024
30.2
36.8
41.2
36.1
23.7 GB
12.5
TABLE II: Comparison of RayIoU performance on the Occ3D [ 45 ] dataset. We report RayIoU at multiple distances (1m, 2m, 4m, and overall), along with VRAM usage and runtime in FPS . All methods were evaluated using an A100 GPU. For TEOcc-RC VRAM, the train code was not available to measure, and there are no reported metrics in the original paper. Unavailable for replication or unreported values are marked with a dash (-).
Method
Backb.
Input
RayIoU ↑
OccNeRF-C [ 52 ]
R101
1600 × 900
21.6
RenderOcc [ 47 ]
R101
1600 × 900
36.7
OccNet [ 3 ]
R101
1600 × 900
39.7
BEVFormer † [ 7 ]
R50
1600 × 900
28.1
FB-Occ † [ 6 ]
R50
704 × 256
32.3
SparseOcc † [ 17 ]
R50
704 × 256
33.4
TABLE III: Comparison of RayIoU performance on OpenOcc [ 3 ] , our second benchmark. OpenOcc uses a different occupancy annotation protocol from Occ3D and contains denser voxel occupancy. LiNO achieves SOTA performance, demonstrating generalisation beyond a single dataset convention. † denotes results reported in [ 5 ] .
1 frame
2 frames
3 frames
RayIoU ↑
29.5
30.3
30.8
FPS ↑
19.32
19.16
19.01
TABLE IV: Temporal frame-count variation.
Z -emb.
Res-fus.
RayIoU ↑
not
not
27.1
8
not
26.8
16
not
27.5
16
not
27.5
16
yes
29.0
TABLE V: Z -embedding and Residual fusion impacts to LiNO model.
Closing ratio
RayIoU ↑
0.0
29.0
0.2
30.8
0.5
29.9
0.8
30.6
1.0
30.4
TABLE VI: Morphological closing ratio.
Dual-Head setup
RayIoU ↑
HU(Haux(Ffpn))
30.12
HU(Haux(Ffpn))+Haux(Ffpn)
29.56
HU(Ffpn)+Haux(Ffpn)
31.32
TABLE VII: Head configuration.
Structure shape
RayIoU ↑
(3×3×3)+(3×3×1)
27.78
(5×5×5)+(5×5×1)
27.82
(7×7×7)+(7×7×1)
27.75
(3×3×3)+(5×5×1)
28.40
(5×5×5)+(3×3×1)
27.84
TABLE VIII: Masks for augmentation.
Name
Type
Params
backbone_img_module
RVTLSSFPN
31.6M
- img_backbone
ResNet
23.5M
- img_neck
SECONDFPN
2.0M
- depth_net
DepthNet
5.4M
- view_aggregation_net
ViewAggregation
807K
fuser
MFAFuser
1.2M
TABLE IX: Parameter breakdown for LiNO-R50.
Module
Latency (ms)
img_backbone
7.74
img_dep
1.42
img_transform
9.31
img_pool
0.48
total img
19.82
pts_voxelize
2.04
TABLE X: Latency mapping of the main modules considering LiNO-R50.
Fig. 5: Per-class RayIoU results for the Occ3D [ 45 ] (Fig. 5a ) and OpenOcc [ 3 ] (Fig. 5b ) datasets. Each class is evaluated under three distance tolerances: RayIoU@1, RayIoU@2, and RayIoU@4 (1 m, 2 m, and 4 m). Black numbers above each marker denote the mean RayIoU per class across the three thresholds. As the distance tolerance increases, the metric becomes more permissive, leading to higher RayIoU values as larger spatial errors are still counted as correct matches.
Fig. 6: Comparison on the Occ3D [ 45 ] . LiNO accurately reconstructs occupancy maps even under GT errors (see camera views). Despite slightly lower RayIoU, LiNO produces visually denser and more consistent voxel predictions than the baseline, which may result from GT sparsity and the sensitivity of the RayIoU to mislabels (red boxes).
The point-based occupancy prediction paradigm has achieved an attractive trade-off between accuracy and efficiency by modeling 3D space sparsely. However, its predictions inherently mismatch the dense voxel-based occupancy required by self-driving systems, necessitating hand-crafted heuristics during training and inference that limit final performance. To overcome these limitations, we propose OPUS-V2, a novel framework built upon the pioneering OPUS (occupancy prediction using a sparse set) point-based approach. OPUS-V2 incorporates a lightweight point-voxel transformation (PVT) module behind the decoder to adaptively map sparse predictions into the dense voxel space, eliminating the need for suboptimal operations and improving model accuracy. Furthermore, our architecture decouples feature and occupancy generation processes, allowing OPUS-V2 to adapt to arbitrary occupancy resolutions. OPUS-V2 achieves a state-of-the-art rayIoU of 44.0 on the Occ3D dataset. On the more challenging OpenOccupancy dataset, it attains a competitive 16.4 mIoU while running in real time at 20.6 FPS.
This paper introduces VLMFusionOcc3D, a robust multimodal framework for dense 3D semantic occupancy prediction in autonomous driving. Current voxel-based occupancy models often struggle with semantic ambiguity in sparse geometric grids and performance degradation under adverse weather conditions. To address these challenges, we leverage the rich linguistic priors of Vision-Language Models (VLMs) to anchor ambiguous voxel features to stable semantic concepts. Our framework initiates with a dual-branch feature extraction pipeline that projects multi-view images and LiDAR point clouds into a unified voxel space. We propose Instance-driven VLM Attention (InstVLM), which utilizes gated cross-attention and LoRA-adapted CLIP embeddings to inject high-level semantic and geographic priors directly into the 3D voxels. Furthermore, we introduce Weather-Aware Adaptive Fusion (WeathFusion), a dynamic gating mechanism that utilizes vehicle metadata and weather-conditioned prompts to re-weight sensor contributions based on real-time environmental reliability. To ensure structural consistency, a Depth-Aware Geometric Alignment (DAGA) loss is employed to align dense camera-derived geometry with sparse, spatially accurate LiDAR returns. Extensive experiments on the nuScenes and SemanticKITTI datasets demonstrate that our plug-and-play modules consistently enhance the performance of state-of-the-art voxel-based baselines. Notably, our approach achieves significant improvements in challenging weather scenarios, offering a scalable and robust solution for complex urban navigation.
Occupancy prediction has garnered increasing attention in recent years for its comprehensive fine-grained environmental representation and strong generalization to open-set objects. Nevertheless, mainstream occupancy prediction methods employ cumbersome voxel features as the scene representation, incurring substantial overheads in both memory and computation. When comparing the occupancy distribution in each spatial dimension, we find that the information entropy of the height dimension is much lower than the other two dimensions that constitute the Bird's Eye View (BEV) plane, which indicates that the height distribution of occupancy is easier to learn and predict. Accordingly, we propose Lightweight Spatial Embedding that can represent complete height information in a more compact way than voxel features, thus significantly enhancing its deployability. First, Single-Channel Occupancy is sampled from the multi-view depth distributions, which is then processed by Spatial-to-Channel mechanism to extract Lightweight Spatial Embeddings of different views by 2D convolution. These embeddings will interact with each other through the Lightweight Cross-View Interaction module to obtain the Unified Embedding, which can directly supplement BEV features with height information. Furthermore, we extract Edge-aware Spatial Embedding and apply Geometric Supervision on Spatial Embeddings, aiming to enhance their capability to represent spatial information. We also propose BEV-CutMix, a feature-level data augmentation strategy, to increase the diversity of the driving scenes. We integrate these innovative components into a pure 2D convolutional model, namely LightOcc. Sufficient experimental results show that LightOcc achieves state-of-the-art performance on multiple benchmarks while demonstrating significant efficiency advantages.
Jinqing Zhang, Yanan Zhang, Wenrui Cai +2
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University. · School of Computer Science and Information Engineering, Hefei University of Technology. · Hangzhou Innovation Institute, Beihang University.