LiDSeg: Linear DehazeFormer Guidance for Efficient Semantic Segmentation in Adverse Weather Conditions
Organizations: KIOS Research and Innovation Center of Excellence, University of Cyprus, Nicosia, Cyprus · Department of Electrical and Computer Engineering, University of Cyprus, Nicosia, Cyprus
Abstract
Robust semantic segmentation in adverse weather is essential for safety-critical autonomous systems. Existing restore-then-segment pipelines introduce latency and prioritize pixel-level restoration over semantic discriminability, while domain adaptation methods often face representation bottlenecks and computational complexity. We introduce LiDSeg, a framework that uses reconstruction-informed guidance to improve segmentation under adverse weather. During training, a lightweight guidance encoder is jointly optimized with the segmentation branch using segmentation and auxiliary reconstruction objectives. At inference, the reconstruction decoder is discarded, while the guidance encoder is retained to extract weather-robust features. A lightweight gated fusion module integrates these features into the segmentation branch, preserving clear-weather accuracy while improving adverse-weather robustness. On Foggy Cityscapes, LiDSeg limits the clear-to-dense-fog mIoU degradation to 2.5-2.8 percentage points, reducing the baseline performance drop by more than fivefold. On the official ACDC test server, LiDSeg-PIDNet achieves 75.07% pooled all-condition mIoU, comparable to supervised HRNet-W48 at 75.0%, while remaining substantially smaller. The gains generalize zero-shot to Foggy Driving and Foggy Zurich without training on real-fog images. LiDSeg adds only 1.9-3.1M parameters across several models, with total model sizes of 5.2-39.5M parameters and no inference-time reconstruction overhead. It delivers real-time inference on the NVIDIA Jetson AGX Orin. Code: https://github.com/yobsanb/LiDSeg
Figures & tables
| Method | Backbone | Params | GFLOPs | Clear (mIoU) | Light (mIoU) | Medium (mIoU) | Dense (mIoU) | |
| Real-time semantic segmentation models | ||||||||
| BiSeNetV2 [ Yu et al.(2021)Yu, Gao, Wang, Yu, Shen, and Sang ] | – | 3.36M | 99.7 | 69.1 | 60.9 | 58.8 | 54.2 | 14.9 |
| DDRNet-23s [ Hong et al.(2021)Hong, Pan, Sun, and Jia ] | – | 5.70M | 36.7 | 77.8 | 66.1 | 62.2 | 56.1 | 21.7 |
| STDC1-Seg [ Fan et al.(2021)Fan, Lai, Huang, Wei, Chai, Luo, and Wei ] | – | 8.43M | 166.3 | 71.8 | 67.5 | 62.5 | 56.1 | 15.7 |
| PIDNet-L [ Xu et al.(2023)Xu, Xiong, and Bhatt ] | – | 37.31M | 276.2 | 74.0 | 68.4 | 63.0 | 58.9 | 15.1 |
| Unguided models with incremental (matched budget, same curriculum) | ||||||||
| Method | Protocol | Params | GFLOPs | Fog | Night | Rain | Snow | Pooled All | Mean |
|---|---|---|---|---|---|---|---|---|---|
| Supervised ACDC labels | |||||||||
| RefineNet [ Lin et al.(2017)Lin, Milan, Shen, and Reid ] | Sup. | 118M | – | 65.7 | 55.5 | 68.7 | 65.9 | 65.3 | 64.0 |
| DeepLabV2 [ Sakaridis et al.(2021a)Sakaridis, Dai, and Van Gool ] | Sup. | 44M | – | 54.5 | 45.3 | 59.3 | 57.1 | 55.3 | 54.1 |
| DeepLabV3+ [ Chen et al.(2018)Chen, Zhu, Papandreou, Schroff, and Adam ] | Sup. | 62.7M | 2032 | 69.1 | 60.9 | 74.1 | 69.6 | 70.0 | 68.4 |
| HRNet-W48 [ Wang et al.(2020)Wang, Sun, Cheng, Jiang, Deng, Zhao, Liu, Ding, Guo, Xu, et al. ] | Sup. | 65.9M | 696 | 74.7 | 65.3 | 77.7 | 76.3 | 75.0 | 73.5 |
| Mask2Former [ Cheng et al.(2022)Cheng, Misra, Schwing, Kirillov, and Garg ] | Sup. | 107M | 466 | – | – | – | – | 77.3 | – |
| Method | Road | Side. | Buil. | Wall | Fenc. | Pole | Ligh. | Sign | Vege. | Terr. | Sky | Pers. | Ride. | Car | Truc. | Bus | Trai. | Moto. | Bicy. | mIoU |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ADVENT [ Vu et al.(2019)Vu, Jain, Bucher, Cord, and Pérez ] | 72.9 | 14.3 | 40.5 | 16.6 | 21.2 | 9.3 | 17.4 | 21.2 | 63.8 | 23.8 | 18.3 | 32.6 | 19.5 | 69.5 | 36.2 | 34.5 | 46.2 | 26.9 | 36.1 | 32.7 |
| MGCDA † [ Sakaridis et al.(2020)Sakaridis, Dai, and Van Gool ] | 73.4 | 28.7 | 69.9 | 19.3 | 26.3 | 36.8 | 53.0 | 53.3 | 75.4 | 32.0 | 84.6 | 51.0 | 26.1 | 77.6 | 43.2 | 45.9 | 53.9 | 32.7 | 41.5 | 48.7 |
| DANNet † [ Wu et al.(2021)Wu, Han, Lin, Uzunbas, Goldstein, Lim, and Metaxas ] | 84.3 | 54.2 | 77.6 | 38.0 | 30.0 | 18.9 | 41.6 | 35.2 | 71.3 | 39.4 | 86.6 | 48.7 | 29.2 | 76.2 | 41.6 | 43.0 | 58.6 | 32.6 | 43.9 | 50.0 |
| DAFormer [ Hoyer et al.(2022a)Hoyer, Dai, and Van Gool ] | 58.4 | 51.3 | 84.0 | 42.7 | 35.1 | 50.7 | 30.0 | 57.0 | 74.8 | 52.8 | 51.3 | 58.3 | 32.6 | 82.7 | 58.3 | 54.9 | 82.4 | 44.1 | 50.7 | 55.4 |
| SSAC † [ Li et al.(2020)Li, Chen, Yuan, Liu, Zhang, Xing, Wang, and Liu ] | 76.5 | 59.8 | 82.7 | 52.6 | 34.5 | 51.2 | 58.1 | 61.2 | 63.9 | 58.1 | 67.2 | 61.5 | 40.0 | 81.8 | 59.2 | 61.7 | 83.4 | 47.5 | 55.6 | 59.6 |
| HRDA [ Hoyer et al.(2022b)Hoyer, Dai, and Van Gool ] | 88.3 | 57.9 | 88.1 | 55.2 | 36.7 | 56.3 | 62.9 | 65.3 | 74.2 | 57.7 | 85.9 | 68.8 | 45.7 | 88.5 | 76.4 | 82.4 | 87.7 | 52.7 | 60.4 | 68.0 |
| Cond. | Setup | Model | road | sidew. | build. | wall | fence | pole | light | sign | veget. | terrain | sky | person | rider | car | truck | bus | train | motorc. | bicycle | mIoU |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Fog | Uber | BiSeNetV2 | 97.1 | 85.9 | 88.8 | 69.9 | 44.8 | 68.0 | 78.0 | 75.0 | 90.6 | 61.1 | 98.3 | 68.2 | 77.5 | 85.5 | 64.3 | 96.2 | 91.7 | 53.9 | 62.1 | 76.7 |
| DDRNet | 97.2 | 85.5 | 88.5 | 70.3 | 44.9 | 68.1 | 77.3 | 74.2 | 90.5 | 60.7 | 98.3 | 66.9 | 75.1 | 84.5 | 62.2 | 95.7 | 90.5 | 52.1 | 65.1 | 76.2 | ||
| STDC | 97.0 | 85.4 | 88.5 | 70.5 | 45.0 | 68.3 | 76.6 | 74.3 | 90.4 | 61.1 | 98.3 | 67.7 | 76.4 | 85.2 | 66.5 | 95.7 | 90.8 | 57.8 | 65.6 | 76.9 | ||
| PIDNet | 97.2 | 86.1 | 88.7 | 69.9 | 44.3 | 68.6 | 78.3 | 75.1 | 90.6 | 61.8 | 98.3 | 69.6 | 74.9 | 85.2 | 63.2 | 95.8 | 91.6 | 52.3 | 65.6 | 76.7 | ||
| Expert | BiSeNetV2 | 97.2 | 86.6 | 87.5 | 68.0 | 41.6 | 64.6 | 72.4 | 67.9 | 90.2 | 61.4 | 98.3 | 61.0 | 67.2 | 85.0 | 54.4 | 86.2 | 87.6 | 31.2 | 57.3 | 71.9 | |
| DDRNet | 96.5 | 84.7 | 86.2 | 66.3 | 40.0 | 61.3 | 70.0 | 65.6 | 89.8 | 61.5 | 98.2 | 61.8 | 68.3 | 83.8 | 53.2 | 88.3 | 83.6 | 30.6 | 58.6 | 71.0 |
| Model | Backbone | Method | Unlabeled FZ | FZ | FD |
|---|---|---|---|---|---|
| RefineNet [ Lin et al.(2017)Lin, Milan, Shen, and Reid ] | ResNet-101 | SFT | 34.6 | 44.3 | |
| RefineNet-lw [ Nekrasov et al.(2018)Nekrasov, Dharmasiri, Spek, Drummond, Shen, and Reid ] | ResNet-101 | SFT | 28.5 | 43.6 | |
| AdaptSegNet [ Tsai et al.(2018) ] | ResNet-101 | UDA | ✓ | 25.0 | 29.7 |
| AdvEnt [ Vu et al.(2019)Vu, Jain, Bucher, Cord, and Pérez ] | ResNet-101 | UDA | ✓ | 39.7 | 46.9 |
| FDA [ Yang and Soatto(2020) ] | ResNet-101 | STL | ✓ | 22.2 | 21.8 |
| DANN [ Ganin et al.(2016) ] | ResNet-101 | UDA | ✓ | 43.1 | 46.0 |
| Model | Variant | Params (M) | GFLOPs | FPS | Mem (MB) | Clear | Dense | |
|---|---|---|---|---|---|---|---|---|
| BiSeNetV2 | Baseline | 3.36 | 24.92 | 94.2 | 51 | 69.1 | 54.2 | 14.9 |
| DDRNet | Baseline | 5.70 | 9.17 | 350.4 | 42 | 77.8 | 56.1 | 21.7 |
| STDC | Baseline | 8.43 | 41.57 | 267.0 | 47 | 71.8 | 56.1 | 15.7 |
| PIDNet-L | Baseline | 37.31 | 69.04 | 116.9 | 111 | 74.0 | 58.9 | 15.1 |
| TopFormer-B | Baseline | 7.06 | 9.69 | 271.7 | 76 | 75.8 | 59.0 | 16.8 |
| SeaFormer-B | Baseline | 8.58 | 3.53 | 166.1 | 58 | 77.7 | 64.6 | 13.0 |
| Setting | Rec. | Fusion | Params (M) | FLOPs (G) | Mean | ACDC |
|---|---|---|---|---|---|---|
| (a) Seg. only | No | – | 3.36 | 99.7 | 58.0 | 56.0 |
| (b) Recon.+Concat | Yes | Concat | 5.70 | 110.8 | 66.1 | 65.0 |
| (c) LGF, no recon. | No | LGF | 5.22 | 108.8 | 66.7 | 67.0 |
| (d) LiDSeg full | Yes | LGF | 5.22 | 108.8 | 75.4 | 73.6 |
| Guidance | Attn. | Struct. | Fusion | Rec. | Params (M) | FLOPs (G) | Mean |
| Dehazing + Segment | Softmax | Seq. | – | Yes | 6.7 | 631.4 | 67.8 |
| Heavy guidance | Softmax | Asy. | Concat | Yes | 5.9 | 446.2 | 67.1 |
| Heavy guidance | Softmax | Asy. | LGF | Yes | 5.9 | 446.0 | 75.4 |
| LiDSeg | Linear | Seq. | – | Yes | 6.3 | 143.0 | 66.3 |
| LiDSeg | Linear | Asy. | Concat | Yes | 5.7 | 110.8 | 66.1 |
| LiDSeg | Linear | Asy. | Spatial | Yes | 6.3 | 126.5 | 75.6 |
| Backbone | Setting | Params (M) | GFLOPs | FPS | Fog | Night | Rain | Snow | Mean |
|---|---|---|---|---|---|---|---|---|---|
| TopFormer-B [ Zhang et al.(2022)Zhang, Huang, Luo, Chen, Wang, Liu, Yu, and Shen ] | Baseline | 7.06 | 9.69 | 271.7 | 67.5 | 45.0 | 61.5 | 63.5 | 59.4 |
| Expert | 9.01 | 12.02 | 126.4 | 75.8 | 52.4 | 67.9 | 69.6 | 66.4 | |
| Uber | 76.4 | 55.3 | 69.5 | 71.4 | 68.2 | ||||
| SeaFormer-B [ Wan et al.(2023)Wan, Huang, Lu, Yu, and Zhang ] | Baseline | 8.58 | 3.53 | 166.1 | 67.0 | 44.5 | 61.0 | 63.0 | 58.9 |
| Expert | 10.98 | 6.05 | 130.3 | 75.4 | 52.0 | 67.5 | 69.2 | 66.0 | |
| Uber | 76.0 | 54.7 | 69.1 | 70.8 | 67.7 |
| Semantic segmentation model | 0.2 | 0.3 | 0.4 (default) | 0.5 |
|---|---|---|---|---|
| BiSeNetV2 | 74.1 | 74.3 | 75.4 | 75.2 |
| DDRNet-23s | 73.0 | 74.2 | 75.3 | 74.1 |
| STDC1-Seg | 75.1 | 75.2 | 75.3 | 75.0 |
| PIDNet-L | 74.0 | 74.9 | 75.3 | 74.6 |
| Setting | BiSeNetV2 | DDRNet-23s | STDC1-Seg | PIDNet-L |
|---|---|---|---|---|
| (a) Multi-scale fusion | ||||
| none (baseline) | 58.0 | 61.5 | 62.0 | 63.4 |
| 69.5 | 70.6 | 70.8 | 71.3 | |
| 70.4 | 71.3 | 71.5 | 71.9 | |
| 67.8 | 69.3 | 69.5 | 70.1 | |
| 73.6 | 73.9 | 73.9 | 74.1 | |
| Method | Road | Side. | Buil. | Wall | Fenc. | Pole | Ligh. | Sign | Vege. | Terr. | Sky | Pers. | Ride. | Car | Truc. | Bus | Trai. | Moto. | Bicy. | mIoU |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LiDSeg (BiSeNetV2) | 98 | 83 | 90 | 58 | 55 | 62 | 66 | 74 | 92 | 66 | 95 | 76 | 60 | 93 | 66 | 76 | 70 | 56 | 74 | 74.2 |
| LiDSeg (DDRNet-23s) | 99 | 82 | 90 | 57 | 54 | 62 | 66 | 73 | 92 | 67 | 95 | 75 | 60 | 93 | 69 | 75 | 71 | 55 | 73 | 74.1 |
| LiDSeg (STDC1-Seg) | 98 | 84 | 89 | 58 | 55 | 62 | 64 | 73 | 92 | 67 | 95 | 75 | 61 | 94 | 68 | 74 | 71 | 57 | 75 | 74.3 |
| LiDSeg (PIDNet-L) | 99 | 84 | 89 | 57 | 53 | 62 | 66 | 74 | 92 | 66 | 96 | 78 | 58 | 94 | 67 | 75 | 70 | 57 | 73 | 74.2 |
| LiDSeg (TopFormer-B) | 97 | 82 | 89 | 57 | 54 | 62 | 66 | 75 | 92 | 65 | 95 | 74 | 59 | 92 | 66 | 76 | 69 | 54 | 74 | 73.6 |
| LiDSeg (SeaFormer-B) | 97 | 82 | 89 | 56 | 53 | 61 | 66 | 74 | 91 | 66 | 95 | 73 | 59 | 92 | 65 | 76 | 70 | 54 | 74 | 73.3 |
| PSNR | SSIM | mIoU | |
|---|---|---|---|
| 0.005 | 34.2 | 0.946 | 76.3 |
| 0.01 | 33.3 | 0.929 | 75.7 |
| 0.02 | 31.0 | 0.902 | 74.2 |
| Pipeline | Params | GFLOPs | FPS | mIoU |
|---|---|---|---|---|
| DehazeFormer BiSeNetV2 | 0 5.9M | 61.4 | 0 54 | 71.5 |
| Restormer BiSeNetV2 | 29.4M | 213.0 | 0 18 | 72.3 |
| LiDSeg (BiSeNetV2) | 5.22M | 27.2 | 83.4 | 74.2 |
| Semantic segmentation model | FZ (unguided) | FZ (LiDSeg) | FD (unguided) | FD (LiDSeg) |
|---|---|---|---|---|
| BiSeNetV2 | 38.2 | 53.5 | 41.0 | 50.4 |
| DDRNet-23s | 40.1 | 52.8 | 42.5 | 51.0 |
| PIDNet-L | 39.5 | 52.6 | 41.8 | 50.8 |
| STDC1-Seg | 39.0 | 53.4 | 42.0 | 52.4 |
| Model | Variant | Params (M) | GFLOPs | FPS | Mem |
|---|---|---|---|---|---|
| BiSeNetV2 | Baseline | 3.36 | 6.23 | 161.5 | 18 |
| DDRNet | Baseline | 5.70 | 2.29 | 707.3 | 18 |
| STDC | Baseline | 8.43 | 10.39 | 724.2 | 23 |
| SCTNet-B | Baseline | 17.44 | 7.04 | 509.3 | 44 |
| PIDNet-L | Baseline | 37.31 | 17.26 | 308.0 | 80 |
| TopFormer-B | Baseline | 7.06 | 2.42 | 715.2 | 24 |
| Model | Expert | Fog | Night | Rain | Snow |
|---|---|---|---|---|---|
| BiSeNetV2 (5.22M, 108.8 GFLOPs) | |||||
| BiSeNetV2 | Fog | 71.9 | 41.7 | 54.4 | 51.5 |
| BiSeNetV2 | Night | 50.4 | 53.1 | 43.4 | 42.1 |
| BiSeNetV2 | Rain | 53.3 | 42.8 | 68.4 | 54.2 |
| BiSeNetV2 | Snow | 51.8 | 40.7 | 57.1 | 67.1 |
| DDRNet-23s (7.65M, 46.0 GFLOPs) | |||||
Semantic Segmentation
Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.
See all 447 papers