LiNO: Lightweight Network for Occupancy Estimation
Authors: Ruan Bispo, Tim Brophy, Reenu Mohandas, Anthony Scanlan, Ciaran Eising
Organizations: The authors are with the Data Driven Computer Engineering Research Centre, Department of Electronic and Computer Engineering, University of Limerick, Limerick V94 T9PX, Ireland, and also with Lero, the Research Ireland Centre for Software, University of Limerick, Limerick V94 NYD3, Ireland.
Occupancy estimation for automated vehicles is a crucial task that builds a 3D map using vehicle sensors. This map allows the vehicle to identify drivable areas and various obstacle classes. The task involves complex subtasks such as depth estimation, object classification, and segmentation to align semantic information with the 3D grid. Moreover, producing these estimations from multiple sensors demands significant computational power and memory, as high-dimensional data fusion requires complex feature extraction and processing. Consequently, most existing methods rely on large and computationally intensive architectures that are not suitable for real-time or real-world applications. We present a real-time, lightweight architecture that leverages camera and radar sensors through a residual multi-stage head. Unlike prior methods that rely on heavy camera-only blocks, our architecture employs camera-radar fusion and lightweight components. Additionally, we introduce a task-specific data augmentation strategy that enhances data diversity and improves generalisation. Our method achieves state-of-the-art (SOTA) performance, attaining 45.7 mean Intersection over Union (mIoU) on Occ3D and 41.9 Ray-level Intersection over Union (RayIoU) (41.9) on OpenOcc datasets, running at 17.2 frames per second (FPS) and using only 5.3 GB of memory, compared with 4.1 FPS and 7.7 GB for STCOcc.
Figures & tables
Fig. 1: Our proposed architecture using camera and radar data. Firstly, the camera and radar features are extracted using the respective feature extractors. Secondly, we lift and fuse these features into a BEV using a BEV Transformation and fusion module. Finally, the proposed occupancy head (highlighted in blue) processes the BEV features to produce the 3D occupancy estimation.
Figure 2
Fig. 4: Proposed data augmentation based on morphological closing to fill empty regions.
Method
Backb.
Input
M
mIoU ↑
others
barrier
bike
bus
car
constrv
mcycle
ped
tcone
trailer
truck
drivsurf
othflat
sidewalk
terrain
manmade
veg
BEVFormer [ 7 ]
R101
1600×900
C
26.9
5.9
37.8
17.9
40.4
42.4
7.4
23.9
21.8
21.0
22.4
30.7
55.4
28.4
36.0
28.1
20.0
17.7
CTF-Occ [ 45 ]
R101
1600×900
C
28.5
8.1
39.3
20.6
38.3
42.2
16.9
24.5
22.7
21.1
23.0
31.1
53.3
33.8
38.0
33.2
20.8
18.0
TPVFormer [ 19 ]
R101
1600×900
C
27.8
7.2
38.9
13.7
40.8
45.9
17.2
20.0
18.9
14.3
26.7
34.2
55.7
35.5
37.6
30.7
19.4
16.8
OSP* [ 50 ]
R101
1600×900
C
41.2
10.9
49.0
27.7
50.2
55.9
22.9
31.0
30.9
30.3
35.6
31.2
82.1
42.6
51.9
55.1
44.8
38.2
SparseOcc [ 17 ]
R50
704×256
C
30.9
10.6
39.2
20.2
32.9
43.3
19.4
23.8
23.4
29.3
21.4
29.3
67.7
36.3
44.6
40.9
22.0
21.9
FlashOcc(M5)* [ 26 ]
R50
704×256
C
38.8
11.0
47.2
22.2
41.4
50.6
23.2
23.0
23.8
26.2
32.1
37.4
82.2
42.5
53.3
56.6
46.7
39.7
TABLE I: Comparison of mIoU performance on the Occ3D [ 45 ] dataset. * indicates models trained with camera masks. M denotes the input modality: C for camera, R for radar, and CR for both. Higher values of IoU indicate better performance.
Method
Backb.
Input
Venue
RayIoU 1m ↑
RayIoU 2m ↑
RayIoU 4m ↑
RayIoU ↑
VRAM ↓
FPS ↑
BEVFormer (4f) [ 7 ]
R101
1600×900
PAMI 2024
26.1
32.9
38.0
32.4
24.1 GB
3.0
RenderOcc (6f) [ 47 ]
Swin-B
1408×512
ICRA 2024
13.4
19.6
25.5
19.5
17.5 GB
-
BEVDet-Occ (8f) [ 22 ]
R50
704×384
arXiv 2022
26.6
33.1
38.2
32.6
10.2 GB
2.6
FB-Occ (16f) [ 6 ]
R50
704×256
ICCV 2023
26.7
34.1
39.7
33.5
9.6 GB
10.3
TEOcc-RC (9f) [ 8 ]
R50
704×256
arXiv 2024
32.0
36.2
41.2
35.8
-
1.1
SparseOcc (16f) [ 17 ]
R50
704×256
ECCV 2024
30.2
36.8
41.2
36.1
23.7 GB
12.5
TABLE II: Comparison of RayIoU performance on the Occ3D [ 45 ] dataset. We report RayIoU at multiple distances (1m, 2m, 4m, and overall), along with VRAM usage and runtime in FPS . All methods were evaluated using an A100 GPU. For TEOcc-RC VRAM, the train code was not available to measure, and there are no reported metrics in the original paper. Unavailable for replication or unreported values are marked with a dash (-).
Method
Backb.
Input
RayIoU ↑
OccNeRF-C [ 52 ]
R101
1600 × 900
21.6
RenderOcc [ 47 ]
R101
1600 × 900
36.7
OccNet [ 3 ]
R101
1600 × 900
39.7
BEVFormer † [ 7 ]
R50
1600 × 900
28.1
FB-Occ † [ 6 ]
R50
704 × 256
32.3
SparseOcc † [ 17 ]
R50
704 × 256
33.4
TABLE III: Comparison of RayIoU performance on OpenOcc [ 3 ] , our second benchmark. OpenOcc uses a different occupancy annotation protocol from Occ3D and contains denser voxel occupancy. LiNO achieves SOTA performance, demonstrating generalisation beyond a single dataset convention. † denotes results reported in [ 5 ] .
1 frame
2 frames
3 frames
RayIoU ↑
29.5
30.3
30.8
FPS ↑
19.32
19.16
19.01
TABLE IV: Temporal frame-count variation.
Z -emb.
Res-fus.
RayIoU ↑
not
not
27.1
8
not
26.8
16
not
27.5
16
not
27.5
16
yes
29.0
TABLE V: Z -embedding and Residual fusion impacts to LiNO model.
Closing ratio
RayIoU ↑
0.0
29.0
0.2
30.8
0.5
29.9
0.8
30.6
1.0
30.4
TABLE VI: Morphological closing ratio.
Dual-Head setup
RayIoU ↑
HU(Haux(Ffpn))
30.12
HU(Haux(Ffpn))+Haux(Ffpn)
29.56
HU(Ffpn)+Haux(Ffpn)
31.32
TABLE VII: Head configuration.
Structure shape
RayIoU ↑
(3×3×3)+(3×3×1)
27.78
(5×5×5)+(5×5×1)
27.82
(7×7×7)+(7×7×1)
27.75
(3×3×3)+(5×5×1)
28.40
(5×5×5)+(3×3×1)
27.84
TABLE VIII: Masks for augmentation.
Name
Type
Params
backbone_img_module
RVTLSSFPN
31.6M
- img_backbone
ResNet
23.5M
- img_neck
SECONDFPN
2.0M
- depth_net
DepthNet
5.4M
- view_aggregation_net
ViewAggregation
807K
fuser
MFAFuser
1.2M
TABLE IX: Parameter breakdown for LiNO-R50.
Module
Latency (ms)
img_backbone
7.74
img_dep
1.42
img_transform
9.31
img_pool
0.48
total img
19.82
pts_voxelize
2.04
TABLE X: Latency mapping of the main modules considering LiNO-R50.
Fig. 5: Per-class RayIoU results for the Occ3D [ 45 ] (Fig. 5a ) and OpenOcc [ 3 ] (Fig. 5b ) datasets. Each class is evaluated under three distance tolerances: RayIoU@1, RayIoU@2, and RayIoU@4 (1 m, 2 m, and 4 m). Black numbers above each marker denote the mean RayIoU per class across the three thresholds. As the distance tolerance increases, the metric becomes more permissive, leading to higher RayIoU values as larger spatial errors are still counted as correct matches.
Fig. 6: Comparison on the Occ3D [ 45 ] . LiNO accurately reconstructs occupancy maps even under GT errors (see camera views). Despite slightly lower RayIoU, LiNO produces visually denser and more consistent voxel predictions than the baseline, which may result from GT sparsity and the sensitivity of the RayIoU to mislabels (red boxes).
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University. · School of Computer Science and Information Engineering, Hefei University of Technology. · Hangzhou Innovation Institute, Beihang University.