Learnable Quantum Efficiency Filters for Urban Hyperspectral Segmentation

Imad Ali Shah; Jiarong Li; Ethan Delaney; Enda Ward; Martin Glavin; Edward Jones; Brian Deegan

cs.CVMar 27, 2026

Learnable Quantum Efficiency Filters for Urban Hyperspectral Segmentation

Authors: Imad Ali Shah, Jiarong Li, Ethan Delaney, Enda Ward, Martin Glavin, Edward Jones, Brian Deegan

Organizations: School of Engineering, University of Galway, Ireland · Ryan Institute, University of Galway, Ireland · Valeo Vision Systems, Tuam, Ireland

Abstract

Hyperspectral sensing provides rich spectral information for scene understanding in urban driving, but its high dimensionality poses challenges for interpretation and efficient learning. We introduce Learnable Quantum Efficiency (LQE), a physics-inspired, interpretable dimensionality reduction (DR) method that parameterizes smooth high-order spectral response functions that emulate plausible sensor quantum efficiency curves. Unlike conventional methods or unconstrained learnable layers, LQE enforces physically motivated constraints, including a single dominant peak, smooth responses, and bounded bandwidth. This formulation yields a compact spectral representation that preserves discriminative information while remaining fully differentiable and end-to-end trainable within semantic segmentation models (SSMs). We conduct systematic evaluations across three publicly available multi-class hyperspectral urban driving datasets, comparing LQE against six conventional and seven learnable baseline DR methods across six SSMs. Averaged across all SSMs and configurations, LQE achieves the highest average mIoU, improving over conventional methods by 2.45%, 0.45%, and 1.04%, and over learnable methods by 1.18%, 1.56%, and 0.81% on HyKo, HSI-Drive, and Hyperspectral City, respectively. LQE maintains strong parameter efficiency (12--36 parameters compared to 51--22K for competing learnable approaches) and competitive inference latency. Ablation studies show that low-order configurations are optimal, while the learned spectral filters converge to dataset-intrinsic wavelength patterns. These results demonstrate that physics-informed spectral learning can improve both performance and interpretability, providing a principled bridge between hyperspectral perception and data-driven multispectral sensor design for automotive vision systems.

Figures & tables

Figure 1: Comparison of visual images, RGB Planes, and hypercube.

Figure 2: Overview of conventional imaging pipelines and the proposed learnable quantum efficiency (LQE) framework. (Top) Standard RGB systems employ fixed Bayer CFA, producing low-dimensional data that are computationally efficient but susceptible to metamerism and loss of material-specific information. (Middle) HSI captures dense spectra but incurs substantial computational and memory overhead due to high dimensionality, requiring DR. (Bottom) The proposed LQE approach introduces continuously differentiable, high-order QE filters that are jointly optimized with the downstream perception model and optically plausible. This framework learns a compact, task-adaptive spectral projection of the hypercube, mitigating the computational burden of HSI while preserving discriminative spectral content.

Figure 3: Example QE curves of the ZWO ASI662MC (left) and ASI462MC (right) sensors’ CFA [ 12 ] , exhibiting a dominant spectral peak, smooth spectral response, and bounded bandwidth.

Datasets	Spatial	Spectral	Classes	Range	No of
	Dimension	Channels		(nm)	Images
HyKo-VIS [ 15 ]	254x512	15	10	470-630	163
HSI-Drive [ 7 ]	209x416	25	9	600-975	752
H-City * [ 6 ]	1422x1889	128	19	450-950	1,330

Table 1: All available multiclass HSI datasets for urban scenes

Figure 4: Comparison of learnable DR layers. All methods transform input hypercubes from $C$ spectral channels to $F$ reduced channels ( $F\ll C$ ) while preserving spatial dimensions $H\times W$ . (a) 1×1Conv: Standard Conv with batch normalization and ReLU. (b) AE. (c) seAttn: Gating mechanism with SE channel reweighting. (d) DSC: Depthwise separable convolution. (e) CN: ConvNeXt-style block. Others include original CBAM and ECA style layers. (f) Proposed LQE method parameterizes spectral response filters as high-order Gaussian basis functions $Q_{f}$ . Numbers in parentheses indicate: kernel sizes. BN: Batch Normalization, and LN: Layer Normalization.

Detail	HyKo-VIS	HSI-Drive	H-City
Batch ( $B$ ), Accumulate	16, 1	16, 1	8, 2
Optimizer, Seed	AdamW (0.0001 initial learning rate), and 42
Loss Function	Combined class-weighted cross entropy and dice loss
Epochs, Scheduler	300, with early stop of 30 patience on mIoU metric
Hardware	Intel Xeon Gold, 256GB RAM, Dual Nvidia A6000
Training Time	Approx. 1 hour for HyKo-VIS to 8 hours of H-City

Table 2: Experimental settings and hyperparameters used

Category	HyKo-VIS			HSI-Drive			H-City
Category	mIoU	mF1	Kappa	mIoU	mF1	Kappa	mIoU	mF1	Kappa
Orig. a	$64.48\pm 3.56$	$76.29\pm 3.27$	$90.32\pm 1.74$	$60.51\pm 0.87$	$73.47\pm 1.13$	$83.08\pm 0.88$	$53.30\pm 2.28$	$65.23\pm 2.24$	$89.99\pm 1.18$
Conv. b	$56.32\pm 4.12$	$69.44\pm 3.91$	$80.18\pm 3.97$	$59.51\pm 1.13$	$71.77\pm 0.89$	$88.01\pm 0.52$	$49.42\pm 0.80$	$61.34\pm 0.73$	$87.97\pm 0.74$
Learn. c	$57.59\pm 1.13$	$70.81\pm 0.89$	$80.99\pm 0.95$	$58.40\pm 3.52$	$70.69\pm 3.74$	$86.27\pm 4.72$	$49.65\pm 2.81$	$61.38\pm 3.07$	$87.35\pm 3.57$
LQE	$58.77\pm 0.66$	$71.84\pm 0.55$	$81.81\pm 0.47$	$59.96\pm 1.05$	$72.32\pm 0.91$	$87.34\pm 1.62$	$50.46\pm 1.27$	$62.33\pm 1.22$	$88.51\pm 0.38$
a Original Dataset Channels (Table 1 ). b Conventional. c Learnable.

Table 4: Overall average performance across categorical DR methods, over

F=3,5,7

and SSMs: Individual method-wise details are shown in Table 3 .

Figure 5: LQE spectral response filters on HyKo-VIS dataset with $F=3$ and $P=3$ . Filters are trained on different backbones and initialized with random perturbations as in Epoch-1. Both architectures converge to similar spectral response patterns by the final epoch, as evidenced by the substantial shift in Filter 1 (blue curves) from the blue-green region ( $\sim$ 510 nm) toward the blue region ( $\sim$ 490 nm). Solid curves represent the complete LQE filter responses, while dashed vertical lines indicate starting peak centroids (blue/green). Red dashed lines highlight spectral regions where filter coverage transitions, showing how two adjacent filters adapt to maintain complete spectral coverage as one filter shifts, specifically at $\sim$ 530nm.

Configuration	( $F=3$ , $P=1$ )		( $F=3$ , $P=3$ )
Configuration	mIoU	mF1	mIoU	mF1
with $\mathcal{L}_{\text{sep}}$	59.54	71.73	58.99	72.43
with $\mathcal{L}_{\text{bw}}$	60.61	72.44	60.13	73.44
with $\mathcal{L}_{\text{dom}}$	61.21	73.29	61.25	73.25
Full LQE ( $\mathcal{L}_{\text{reg}}$ )	62.79	75.97	61.94	74.14

Table 7: Regularization components: HyKo-VIS over UNet++.

Dataset	Method	Param (Million)	CPU (ms) *	GPU (ms) *
HyKo-VIS	1x1Conv	51	$2.92\pm 0.89$	$\mathbf{0.16\pm 0.05}$
	AE	846	$2.59\pm 0.27$	$0.40\pm 0.02$
	ECA	52	$1.44\pm 0.10$	$0.29\pm 0.08$
	CBAM	155	$3.88\pm 0.22$	$5.70\pm 2.09$
	seAttn	63	$\mathbf{1.08\pm 0.06}$	$0.28\pm 0.04$
	DSC	204	$2.71\pm 0.34$	$0.25\pm 0.38$

Table 8: Computational Performance (

F=3

), not accounting for downstream SSMs performance.

Explore similar work

May 27, 2026

Transfer learning RGB models to hyperspectral images with trainable tensor decompositions

Transfer learning makes it possible to use large vision networks on a variety of domains, by specializing their models' general filters to new tasks. However, these networks assume the input images to have 3 input channels, making them incompatible with multi- or hyperspectral images. Current approaches that mitigate this incompatibility sacrifice information in either the image, or the model. This work proposes a novel approach that preserves the image and spatial information present in the model by using partially trainable tensor decompositions. We create such decompositions of pretrained convolutional filters, separating the filters into spatial and spectral components. The spectral components are then replaced with trainable components of higher channel dimensionality. This creates hyperspectral filters that can specialize to new datasets, while retaining the spatial patterns of the original filter. Experiments on a variety of hyperspectral datasets show that our approach is more accurate and robust than other hyperspectral transfer learning methods.

Mariette Schönfeld, Laurens Devos, Wannes Meert +1

KU Leuven, Dept. of Computer Science, B-3000 Leuven, Belgium · Leuven.AI - KU Leuven Institute for AI, B-3000 Leuven, Belgium

Tensor Decomposition Hyperspectral Imaging Transfer Learning Convolutional Neural Networks

Ask PDF

May 18, 2026

LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift

Modeling hyperspectral imagery (HSI) across different sensors presents a fundamental challenge due to variations in wavelength coverage, band sampling, and channel dimensionality. As a result, models trained under a fixed spectral configuration often fail to generalize to other sensors. Existing Vision Transformer (ViT) approaches either rely on implicit spectral modeling with fixed channel assumptions or adopt explicit spatial-spectral attention with prohibitive computational cost, leading to a fundamental trade-off between efficiency and expressiveness. In this work, we introduce Low-rank Efficient Spatial-Spectral ViT (LESSViT), a sensor-flexible architecture for cross-spectral generalization. LESSViT is built on LESS Attention, a structured low-rank factorization that models joint spatial-spectral interactions through separable spatial and spectral components, reducing the complexity of full spatial-spectral attention from

O(N^2 C^2)

O(rNC)

, where

N

is the number of spatial tokens,

C

is the number of spectral channels, and

r

is the rank of the low-rank approximation. We further incorporate channel-agnostic patch embedding and wavelength-aware positional encoding to support flexible spectral inputs. To enable efficient and robust pretraining, we introduce a hyperspectral masked autoencoder (HyperMAE) with decoupled spatial-spectral masking and hierarchical channel sampling. We evaluate LESSViT under a cross-spectral generalization setting that simulates cross-sensor variability. Experiments on the SpectralEarth benchmark demonstrate that LESSViT improves robustness under spectral shifts while remaining competitive in-distribution, and explicit and efficient spatial-spectral modeling is essential for scalable and generalizable hyperspectral representation learning.

Haozhe Si, Yuxuan Wan, Yuqing Wang +2

Department of Electrical and Computer Engineering · 2Siebel School of Computing and Data Science

Representation Learning Masked Autoencoders Hyperspectral Imaging Efficient ViTs

Ask PDF

Sep 25, 2026

Band-Selection Stability and Semantic Segmentation Performance: A Study on Hyperspectral City

Resource constraints make high-dimensional hyperspectral imaging challenging in autonomous perception, motivating the use of band selection methods. However, the sensitivity of band-selection methods to sampled data and their relationship to semantic segmentation models (SSMs) remain underexplored. This study evaluates six band selection methods on ten independently sampled, class-balanced region-of-interest (ROI) sets, yielding 60 top-25 band subsets from the Hyperspectral City V2 (128 bands: 450-950nm) dataset. Top-

K

bands (

K\in\{3,5, ... 13\}

) from the first three ROI sets are evaluated with three SSMs against the corresponding 128-band baseline. Experiments show that intra-method stability is method-dependent: Sim-LP shows the highest stability (pairwise Jaccard similarity) and, together with JMIM+CSNR, yields the best segmentation results. Top-

K

based SSMs remain competitive with baselines, with gains of up to 2.01 mIoU and 1.72 mF1 points, and 18-22x faster CPU inference for

K=9

. However, performance does not improve monotonically with

K

, and stability shows no consistent association with SSM performance. These findings suggest that intra-method stability is informative but an unreliable indicator of downstream segmentation performance, highlighting the need to evaluate band-selection methods across repeated samples, subset sizes, and SSMs.

Jiarong Li, Imad Ali Shah, Enda Ward +3

School of Engineering and Ryan Institute, University of Galway, Ireland · Valeo Vision Systems, Tuam, Ireland

Hyperspectral Imaging Remote Sensing Image Segmentation Semantic Segmentation

Ask PDF