Multi-Scale Spectral Attention Module-based Hyperspectral Segmentation in Autonomous Driving Scenarios
Organizations: School of Engineering, University of Galway, Ireland · Ryan Institute, University of Galway, Ireland · Valeo Vision Systems, Tuam, Ireland
Abstract
Recent advances in autonomous driving (AD) have highlighted the potential of hyperspectral imaging (HSI) for enhanced environmental perception, particularly in challenging weather and lighting conditions. However, efficiently processing high-dimensional spectral data remains a significant challenge. This paper presents an empirical investigation of a Multi-Scale Attention Mechanism (MSAM) for enhanced spectral feature extraction through three parallel 1D convolutions with varying kernel sizes (1-11) and adaptive feature aggregation. By integrating MSAM into UNet's skip connections, we evaluate performance improvements in semantic segmentation across multiple HSI datasets for urban driving scenarios. Comprehensive ablation studies demonstrate that MSAM consistently outperforms baseline UNet-SC, achieving average improvements of 2.32% in mIoU and 2.88% in mF1, while maintaining competitive GPU performance against established attention mechanisms. Our findings reveal that optimal kernel combinations are dataset-specific, with configurations such as (1;5;11) and (3;7;11) demonstrating particularly strong performance. This empirical investigation advances understanding of HSI processing capabilities for AD applications and establishes a foundation for adaptive multi-scale spectral feature extraction in automotive deployment.
Figures & tables
| HSI Dataset * | Year | Wavelength (nm) | Channels | Image Resolution | No of Images | Classes | Urban | Rural | Off-terrain |
|---|---|---|---|---|---|---|---|---|---|
| Hyko v2 [ 17 ] 1 | 2017 | VIS: 470-630 | 15 | 254x510 | 163 | 10 | ✓ | ✓ | × |
| NIR: 630-975 | 25 | 214x407 | 78 | 10 | × | × | × | ||
| HSI-Drive [ 3 ] | 2023 | 600-975 | 25 | 209x416 | 752 | 9 | ✓ | × | × |
| H-City [ 4 ] 2 | 2022 | 450-950 | 128 | 1422x1889 | 1,330 | 19 | ✓ | × | × |
| HSI Road [ 18 ] 3 | 2020 | 680-960 | 25 | 384x192 | 3,799 | 2 | × | ✓ | ✓ |
| HSI-Drive | HyKo-VIS | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Classes | Instances | % | Pixels a | % | Classes | Instances | % | Pixels a | % |
| Road | 748 | 0.15 | 26.58 | 0.61 | Road | 235 | 0.18 | 11.7 | 0.48 |
| Road Marks | 736 | 0.15 | 1.32 | 0.03 | Lane Markers | 134 | 0.10 | 0.32 | 0.01 |
| Vegetation | 659 | 0.13 | 9.30 | 0.21 | Vegetation | 157 | 0.12 | 2.53 | 0.10 |
| P. Metal b | 688 | 0.14 | 0.94 | 0.02 | Vehicles f | 97 | 0.08 | 0.58 | 0.02 |
| Sky | 530 | 0.11 | 2.51 | 0.06 | Sky | 158 | 0.12 | 2.73 | 0.11 |
| Author | Year | Dataset Used | Method and Techniques |
|---|---|---|---|
| Basterretxea et al. [ 23 ] | 2021 | HSI-Drive | Per-pixel ANN followed by two-staged spatial regularization |
| Gutiérrez-Zaballa et al. [ 8 ] | 2022 | HSI-Drive | Patches-based UNet, and Per-pixel ANN |
| Gutiérrez-Zaballa et al. [ 3 ] | 2023 | HSI-Drive | Patches-based UNet [ 16 ] |
| Ding et al. [ 24 ] | 2023 | H-City | HSI and pseudo-RGB based spectral feature fusion and multi-scale decoder |
| Theisen et al. [ 9 ] | 2024 | HyKo-VIS, HSI-Drive, and H-City | RGB-HSI benchmarking framework using DeepLabv3+ [ 25 ] and UNet |
| Gutiérrez-Zaballa et al. | 2024 | HSI-Drive | Parameter perturbations [ 26 ] and activation functions [ 27 ] impact in DNN |
| Datasets | HyKo-VIS | HSI-Drive | H-City** |
|---|---|---|---|
| Input Size | 254x510x15 | 209x416x25 | 355x472x128 |
| Dataset Size | 163 | 752 | 1,330 |
| Batch and Acc* | 16, 2 | 16, 2 | 8, 4 |
| Original Classes | 10 | 9 (less water) | 18 (less train) |
| Relabeled Classes | 7 | 7 | 6 |
| LR, AF and Epoch | 7e-4, LeakyReLU [ 40 ] , 300 | ||
| Class Labels | HyKo-VIS | HSI-Drive | H-City** | |||
|---|---|---|---|---|---|---|
| Count | % | Count | % | Count | % | |
| Road | 11.73 | 47.74 | 26.58 | 61.10 | 154.49 | 33.87 |
| Vegetation | 6.06 | 24.67 | 9.3 | 21.38 | 2.00 | 0.44 |
| Sky | 2.73 | 11.11 | 2.51 | 5.77 | 0.901 | 0.20 |
| Metal | 0.861 | 3.50 | 1.29 | 2.97 | 153.16 | 33.58 |
| Infrastructure | 2.81 | 11.44 | 2.29 | 5.26 | 143.86 | 31.54 |
| Dataset | Configuration | UNet 16 : (16, 32, 64, 128) | UNet 32 : (32, 64, 128, 256) | UNet 64 : (64, 128, 256, 512) | |||
|---|---|---|---|---|---|---|---|
| mIoU | mF1 | mIoU | mF1 | mIoU | mF1 | ||
| HyKo-VIS | UNet-SC | 54.33 | 69.79 | 61.78 | 70.34 | 62.53 | 76.81 |
| UNet-MSAM (1;3;11) | 61.58 | 76.85 | 65.42 | 79.47 | 64.79 | 79.02 | |
| (1;3;5) | 59.11 | 74.13 | 65.25 | 79.43 | 66.68 | 80.21 | |
| (1;5;11) | 60.28 | 75.46 | 65.30 | 78.88 | 68.17 | 80.67 | |
| (1;5;9) | 61.08 | 75.94 | 66.61 | 80.16 | 67.17 | 80.35 | |
| Data | Backbone UNet 32 | Original Dataset Classes | ||||
|---|---|---|---|---|---|---|
| param. a | CPU b | GPU b | mIoU | mF1 | ||
| HyKo-VIS | SE [ 43 ] | 7.778 | 70.43 | 4.07 | 67.27 | 79.68 |
| CBAM [ 30 ] | 7.779 | 79.55 | 45.54 | 67.39 | 79.66 | |
| CA [ 31 ] | 7.779 | 73.95 | 7.04 | 66.71 | 79.50 | |
| MSAM (3;7;11) | 7.768 | 231.91 | 7.30 | 67.03 | 80.09 | |
| MSAM (1;5;11) | 7.768 | 253.28 | 7.45 | 67.31 | 80.11 | |
Semantic Segmentation
Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.
See all 441 papers