Learnable Quantum Efficiency Filters for Urban Hyperspectral Segmentation

Imad Ali Shah; Jiarong Li; Ethan Delaney; Enda Ward; Martin Glavin; Edward Jones; Brian Deegan

cs.CVMar 27, 2026

Learnable Quantum Efficiency Filters for Urban Hyperspectral Segmentation

Authors: Imad Ali Shah, Jiarong Li, Ethan Delaney, Enda Ward, Martin Glavin, Edward Jones, Brian Deegan

Organizations: School of Engineering, University of Galway, Ireland · Ryan Institute, University of Galway, Ireland · Valeo Vision Systems, Tuam, Ireland

Abstract

Hyperspectral sensing provides rich spectral information for scene understanding in urban driving, but its high dimensionality poses challenges for interpretation and efficient learning. We introduce Learnable Quantum Efficiency (LQE), a physics-inspired, interpretable dimensionality reduction (DR) method that parameterizes smooth high-order spectral response functions that emulate plausible sensor quantum efficiency curves. Unlike conventional methods or unconstrained learnable layers, LQE enforces physically motivated constraints, including a single dominant peak, smooth responses, and bounded bandwidth. This formulation yields a compact spectral representation that preserves discriminative information while remaining fully differentiable and end-to-end trainable within semantic segmentation models (SSMs). We conduct systematic evaluations across three publicly available multi-class hyperspectral urban driving datasets, comparing LQE against six conventional and seven learnable baseline DR methods across six SSMs. Averaged across all SSMs and configurations, LQE achieves the highest average mIoU, improving over conventional methods by 2.45%, 0.45%, and 1.04%, and over learnable methods by 1.18%, 1.56%, and 0.81% on HyKo, HSI-Drive, and Hyperspectral City, respectively. LQE maintains strong parameter efficiency (12--36 parameters compared to 51--22K for competing learnable approaches) and competitive inference latency. Ablation studies show that low-order configurations are optimal, while the learned spectral filters converge to dataset-intrinsic wavelength patterns. These results demonstrate that physics-informed spectral learning can improve both performance and interpretability, providing a principled bridge between hyperspectral perception and data-driven multispectral sensor design for automotive vision systems.

Figures & tables

Figure 1: Comparison of visual images, RGB Planes, and hypercube.

Figure 2: Overview of conventional imaging pipelines and the proposed learnable quantum efficiency (LQE) framework. (Top) Standard RGB systems employ fixed Bayer CFA, producing low-dimensional data that are computationally efficient but susceptible to metamerism and loss of material-specific information. (Middle) HSI captures dense spectra but incurs substantial computational and memory overhead due to high dimensionality, requiring DR. (Bottom) The proposed LQE approach introduces continuously differentiable, high-order QE filters that are jointly optimized with the downstream perception model and optically plausible. This framework learns a compact, task-adaptive spectral projection of the hypercube, mitigating the computational burden of HSI while preserving discriminative spectral content.

Figure 3: Example QE curves of the ZWO ASI662MC (left) and ASI462MC (right) sensors’ CFA [ 12 ] , exhibiting a dominant spectral peak, smooth spectral response, and bounded bandwidth.

Datasets	Spatial	Spectral	Classes	Range	No of
	Dimension	Channels		(nm)	Images
HyKo-VIS [ 15 ]	254x512	15	10	470-630	163
HSI-Drive [ 7 ]	209x416	25	9	600-975	752
H-City * [ 6 ]	1422x1889	128	19	450-950	1,330

Table 1: All available multiclass HSI datasets for urban scenes

Figure 4: Comparison of learnable DR layers. All methods transform input hypercubes from $C$ spectral channels to $F$ reduced channels ( $F\ll C$ ) while preserving spatial dimensions $H\times W$ . (a) 1×1Conv: Standard Conv with batch normalization and ReLU. (b) AE. (c) seAttn: Gating mechanism with SE channel reweighting. (d) DSC: Depthwise separable convolution. (e) CN: ConvNeXt-style block. Others include original CBAM and ECA style layers. (f) Proposed LQE method parameterizes spectral response filters as high-order Gaussian basis functions $Q_{f}$ . Numbers in parentheses indicate: kernel sizes. BN: Batch Normalization, and LN: Layer Normalization.

Detail	HyKo-VIS	HSI-Drive	H-City
Batch ( $B$ ), Accumulate	16, 1	16, 1	8, 2
Optimizer, Seed	AdamW (0.0001 initial learning rate), and 42
Loss Function	Combined class-weighted cross entropy and dice loss
Epochs, Scheduler	300, with early stop of 30 patience on mIoU metric
Hardware	Intel Xeon Gold, 256GB RAM, Dual Nvidia A6000
Training Time	Approx. 1 hour for HyKo-VIS to 8 hours of H-City

Table 2: Experimental settings and hyperparameters used

Category	HyKo-VIS			HSI-Drive			H-City
Category	mIoU	mF1	Kappa	mIoU	mF1	Kappa	mIoU	mF1	Kappa
Orig. a	$64.48\pm 3.56$	$76.29\pm 3.27$	$90.32\pm 1.74$	$60.51\pm 0.87$	$73.47\pm 1.13$	$83.08\pm 0.88$	$53.30\pm 2.28$	$65.23\pm 2.24$	$89.99\pm 1.18$
Conv. b	$56.32\pm 4.12$	$69.44\pm 3.91$	$80.18\pm 3.97$	$59.51\pm 1.13$	$71.77\pm 0.89$	$88.01\pm 0.52$	$49.42\pm 0.80$	$61.34\pm 0.73$	$87.97\pm 0.74$
Learn. c	$57.59\pm 1.13$	$70.81\pm 0.89$	$80.99\pm 0.95$	$58.40\pm 3.52$	$70.69\pm 3.74$	$86.27\pm 4.72$	$49.65\pm 2.81$	$61.38\pm 3.07$	$87.35\pm 3.57$
LQE	$58.77\pm 0.66$	$71.84\pm 0.55$	$81.81\pm 0.47$	$59.96\pm 1.05$	$72.32\pm 0.91$	$87.34\pm 1.62$	$50.46\pm 1.27$	$62.33\pm 1.22$	$88.51\pm 0.38$
a Original Dataset Channels (Table 1 ). b Conventional. c Learnable.

Table 4: Overall average performance across categorical DR methods, over

F=3,5,7

and SSMs: Individual method-wise details are shown in Table 3 .

Figure 5: LQE spectral response filters on HyKo-VIS dataset with $F=3$ and $P=3$ . Filters are trained on different backbones and initialized with random perturbations as in Epoch-1. Both architectures converge to similar spectral response patterns by the final epoch, as evidenced by the substantial shift in Filter 1 (blue curves) from the blue-green region ( $\sim$ 510 nm) toward the blue region ( $\sim$ 490 nm). Solid curves represent the complete LQE filter responses, while dashed vertical lines indicate starting peak centroids (blue/green). Red dashed lines highlight spectral regions where filter coverage transitions, showing how two adjacent filters adapt to maintain complete spectral coverage as one filter shifts, specifically at $\sim$ 530nm.

Configuration	( $F=3$ , $P=1$ )		( $F=3$ , $P=3$ )
Configuration	mIoU	mF1	mIoU	mF1
with $\mathcal{L}_{\text{sep}}$	59.54	71.73	58.99	72.43
with $\mathcal{L}_{\text{bw}}$	60.61	72.44	60.13	73.44
with $\mathcal{L}_{\text{dom}}$	61.21	73.29	61.25	73.25
Full LQE ( $\mathcal{L}_{\text{reg}}$ )	62.79	75.97	61.94	74.14

Table 7: Regularization components: HyKo-VIS over UNet++.

Dataset	Method	Param (Million)	CPU (ms) *	GPU (ms) *
HyKo-VIS	1x1Conv	51	$2.92\pm 0.89$	$\mathbf{0.16\pm 0.05}$
	AE	846	$2.59\pm 0.27$	$0.40\pm 0.02$
	ECA	52	$1.44\pm 0.10$	$0.29\pm 0.08$
	CBAM	155	$3.88\pm 0.22$	$5.70\pm 2.09$
	seAttn	63	$\mathbf{1.08\pm 0.06}$	$0.28\pm 0.04$
	DSC	204	$2.71\pm 0.34$	$0.25\pm 0.38$

Table 8: Computational Performance (

F=3

), not accounting for downstream SSMs performance.

Explore similar work

Transfer learning RGB models to hyperspectral images with trainable tensor decompositions
May 27, 2026Mariette Schönfeld, Laurens Devos, Wannes Meert +1Tensor Decomposition Hyperspectral Imaging
KU Leuven, Dept. of Computer Science, B-3000 Leuven, Belgium · Leuven.AI - KU Leuven Institute for AI, B-3000 Leuven, Belgium
LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift
May 18, 2026Haozhe Si, Yuxuan Wan, Yuqing Wang +2Representation Learning Masked Autoencoders
Department of Electrical and Computer Engineering · 2Siebel School of Computing and Data Science
Band-Selection Stability and Semantic Segmentation Performance: A Study on Hyperspectral City
Sep 25, 2026Jiarong Li, Imad Ali Shah, Enda Ward +3Hyperspectral Imaging Remote Sensing Image Segmentation
School of Engineering and Ryan Institute, University of Galway, Ireland · Valeo Vision Systems, Tuam, Ireland

May 27, 2026J/K move · Enter open · S save

Transfer learning RGB models to hyperspectral images with trainable tensor decompositions

Mariette Schönfeld, Laurens Devos, Wannes Meert, Hendrik Blockeel

KU Leuven, Dept. of Computer Science, B-3000 Leuven, Belgium · Leuven.AI - KU Leuven Institute for AI, B-3000 Leuven, Belgium

Tensor Decomposition Hyperspectral Imaging Transfer Learning Convolutional Neural Networks

Open paper AskPDF arXiv

Transfer learning makes it possible to use large vision networks on a variety of domains, by specializing their models' general filters to new tasks. However, these networks assume the input images to have 3 input channels, making them incompatible with multi- or hyperspectral images. Current approaches that mitigate this incompatibility sacrifice information in either the image, or the model. This work proposes a novel approach that preserves the image and spatial information present in the model by using partially trainable tensor decompositions. We create such decompositions of pretrained convolutional filters, separating the filters into spatial and spectral components. The spectral components are then replaced with trainable components of higher channel dimensionality. This creates hyperspectral filters that can specialize to new datasets, while retaining the spatial patterns of the original filter. Experiments on a variety of hyperspectral datasets show that our approach is more accurate and robust than other hyperspectral transfer learning methods.

Abstract

Explore similar work

Transfer learning RGB models to hyperspectral images with trainable tensor decompositions

LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift

Band-Selection Stability and Semantic Segmentation Performance: A Study on Hyperspectral City