LiPS: Lightweight Panoptic Segmentation for Resource-Constrained Robotics
Authors: Calvin Galagain, Martyna Poreba, François Goulette, Cyrill Stachniss
Organizations: Université Paris-Saclay, CEA LIST, F-91120 Palaiseau, France · U2IS, ENSTA, Institut Polytechnique de Paris, 91120 Palaiseau, France · University of Bonn, Center for Robotics, Bonn, Germany
Panoptic segmentation is a key enabler for robotic perception, as it unifies semantic understanding with object-level reasoning. However, the increasing complexity of state-of-the-art models makes them unsuitable for deployment on resource-constrained platforms such as mobile robots. We propose a novel approach called LiPS that addresses the challenge of efficient-to-compute panoptic segmentation with a lightweight design that retains query-based decoding while introducing a streamlined feature extraction and fusion pathway. It aims at providing a strong panoptic segmentation performance while substantially lowering the computational demands. Evaluations on standard benchmarks demonstrate that LiPS attains accuracy comparable to much heavier baselines, while providing up to 4.5 higher throughput, measured in frames per second, and requiring nearly 6.8 times fewer computations. This efficiency makes LiPS a highly relevant bridge between modern panoptic models and real-world robotic applications.
Figures & tables
Figure 1: LiPS architecture. Encoder produces a four-level feature hierarchy. A routing step selects a subset of levels (1-4), which are downsampled with strided convolutions, enriched with sine positional encodings, and fused through a shallow deformable-attention pixel decoder. A lightweight top-down FPN exposes three mask-feature scales for the masked transformer decoder, which predicts class labels and panoptic masks from a fixed set of queries. By default, two routed levels (1 and 2) are used for embedded deployment. Dashed arrows denote skipped levels; solid arrows indicate active information flow.
Model Variant
PQ(%)
mIoU(%)
AP(%)
GFLOPs
Params (M)
FPS
Mask2Former-R50
39.6
46.0
26.5
147.2
44.0
7.1
LiPS (1 level)
34.8
43.9
19.8
24.6
20.1
18.3
LiPS (2 levels)
35.8
44.0
21.4
26.4
20.0
17.5
LiPS (3 levels)
36.1
44.4
22.0
28.4
19.9
15.7
LiPS (full)
36.4
45.3
22.1
36.8
19.8
10.7
Table 1: LiPS performance on ADE20K across different routed encoder levels.
Model Variant
PQ(%)
mIoU(%)
AP(%)
GFLOPs
Params (M)
FPS
Mask2Former-R50
62.1
77.5
37.3
527.4
44.0
2.4
LiPS (1 level)
54.1
72.7
24.9
77.4
19.9
10.8
LiPS (2 levels)
54.5
73.3
25.9
84.2
19.9
10.7
LiPS (3 levels)
55.3
73.0
25.6
91.7
19.8
8.5
LiPS (full)
55.3
71.6
26.9
121.5
19.7
4.5
Table 2: LiPS performance on Cityscapes across different routed encoder levels.
Enc. Layers
PQ(%)
mIoU(%)
AP(%)
Params (M)
GFLOPs
6
39.6
46.0
26.5
44.0
147
5
39.6
45.9
26.6
43.26
143
4
38.4
46.3
25.3
42.52
134
3
38.7
44.5
24.7
41.79
125
2
38.3
46.1
24.5
41.06
116
1
36.9
46.3
23.2
40.33
107
Table 3: Ablation study on the number of Transformer encoder layers in Mask2Former-R50.
Channel Dims
PQ(%)
mIoU(%)
AP(%)
Params (M)
GFLOPs
256
39.6
46.0
26.5
44.0
147
128
38.0
46.0
25.6
40.7
97.7
64
35.9
44.3
23.4
39.2
76.6
Table 4: Ablation study on the width of the pixel decoder and mask branch in Mask2Former-R50.
Model
Lat. (ms)
FPS
RAM (MB)
GPU Mem. (MB)
ADE20K
Mask2Former-R50
982
1.02
19288
2466
LiPS (1 level)
630
1.59
19155
1386
LiPS (2 levels)
651
1.54
19201
1419
LiPS (3 levels)
729
1.37
19214
1470
LiPS (full)
814
1.23
19231
1634
Table 5: Jetson AGX Orin measurements in 15 W mode. RAM and GPU memory denote peak inference usage.
Figure 2: Qualitative comparison on Cityscapes Columns (left → right): (a) input, (b) ground truth, (c) LiPS (2 levels), (d) LiPS (full), (e) Mask2Former-R50. A fixed ROI is cropped and magnified identically across methods to enable like-for-like inspection of boundary adherence (e.g., curb/sidewalk), thin-structure fidelity (poles, traffic signs), and stuff/thing separation.