Organizations: School of Management, Zhengzhou University, Zhengzhou, 450001, China · Center for Intelligent Decision-making and Machine Learning, Xi’an Jiaotong University, Xi’an, 710049, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, China
Hyperspectral and multispectral image fusion (HMIF) aims to reconstruct a high-resolution hyperspectral image (HR-HSI) by combining the fine spatial details of a high-resolution multispectral image (HR-MSI) with the rich spectral information of a low-resolution hyperspectral image (LR-HSI). Recent advances in implicit neural representations (INRs) have enabled flexible coordinate-based modeling for HMIF; however, existing INR-based approaches may not fully capture fine-grained spatial structures and rich spectral dependencies. Moreover, the LR-HSI and HR-MSI are primarily incorporated through degradation-consistency constraints, leaving their complementary information underexploited. To address these limitations, we propose Two-Stage Reconstruction with Implicit Tensor Neural Representation (TSR-ITNR), a unified self-supervised framework integrating representation refinement and observation-guided calibration. In Stage 1, TSR-ITNR learns an implicit Tucker representation and refines its low-rank spatial coefficient tensor and spectral basis to better capture fine spatial structures and interband correlations. A fixed pretrained denoiser further provides a deep prior for the preliminary reconstruction. In Stage 2, parameter-free calibration derives complementary and noninterfering corrections from both observations to recover information insufficiently captured in Stage 1. Theoretical analysis establishes the geometry-preserving property of spectral refinement and the orthogonal complementarity of calibration. Extensive experiments on multiple benchmark datasets demonstrate strong quantitative, visual, and spectral reconstruction performance without ground-truth HR-HSI supervision. Beyond conventional reconstruction metrics, we further assess the effectiveness of TSR-ITNR using downstream semantic segmentation accuracy.
Figures & tables
Fig. 1: Overview of the proposed Two-Stage Reconstruction with implicit tensor neural representation (TSR-ITNR) framework and its key refinement modules. (a) Overall framework of TSR-ITNR, comprising two tightly coupled and complementary stages: implicit Tucker representation and refinement (ITRR) and complementary observation-guided calibration (COGC). (b) Architecture of the multi-scale spatial coefficient refiner (MSCR). (c) Architecture of the attention-guided geometry-preserving spectral refiner (AGPSR).
Fig. 2: Detailed architectures of the adaptive feature transformation (AFT) block employed in MSCR and its embedded channel attention mechanism (CAM): (a) AFT block and (b) CAM block.
Dataset
Index
Low Rank
Deep Learning
Hybrid
CTDF
GTNN
CAFE+
EDIP
CS2DIPs
DELTA
CNN-FUS
LRTFR
CLoRF
SSLRDN
TSR-ITNR
Pavia Centre
PSNR ↑
44.85
45.24
42.65
44.57
45.88
41.22
44.90
40.63
44.45
45.41
46.36
SSIM ↑
0.983
0.984
0.975
0.982
0.985
0.971
0.984
0.969
0.983
0.985
0.985
SAM ↓
0.072
0.064
0.078
0.064
0.060
0.093
0.065
0.104
0.065
0.061
0.058
ERGAS ↓
38.25
34.63
42.94
36.07
32.81
51.08
35.86
52.08
36.18
33.94
32.55
TABLE I: Quantitative metrics of the compared approaches on the Pavia dataset
Dataset
Index
Low Rank
Deep Learning
Hybrid
CTDF
GTNN
CAFE+
EDIP
CS2DIPs
DELTA
CNN-FUS
LRTFR
CLoRF
SSLRDN
TSR-ITNR
BGU
PSNR ↑
50.30
50.15
51.17
54.41
51.51
48.62
46.76
50.96
53.53
54.77
56.61
SSIM ↑
0.996
0.997
0.996
0.998
0.998
0.993
0.987
0.997
0.998
0.999
0.999
SAM ↓
0.019
0.020
0.020
0.011
0.018
0.030
0.041
0.020
0.014
0.012
0.010
ERGAS ↓
21.29
22.10
13.97
9.65
16.16
24.02
26.20
14.17
10.62
8.90
7.66
Flower
PSNR ↑
48.09
48.46
46.40
48.88
49.02
44.89
45.31
44.69
49.38
48.81
50.23
TABLE III: Quantitative metrics of the compared approaches on the ICVL dataset
Fig. 3: Visual comparisons of selected representative methods on Pavia Centre ( ×8 ), CAVE Lemons ( ×16 ), and ICVL Hill ( ×16 ). The three large images present pseudocolor outputs for representative samples, with green boxes indicating regions selected for local magnification. For each dataset block, the top row shows enlarged pseudocolor details, and the bottom row shows the corresponding residual maps.
Fig. 4: Spectral curves of different methods in three scenes, from top to bottom: Pavia Centre, CAVE Lemons, and ICVL Hill.
Fig. 5: Visual comparison of different ablation settings on ICVL Hill ( ×16 ).
Fig. 6: Visual comparison of semantic segmentation maps obtained from the Bicubic baseline, different reconstructed HSIs, and the reference HR-HSI on the Pavia Centre test region. GT denotes the ground-truth labels.
This paper addresses the fusion of a pair of spatially unregistered hyperspectral image (HSI) and multispectral image (MSI) covering roughly overlapping regions. HSIs offer high spectral but low spatial resolution, while MSIs provide the opposite. The goal is to integrate their complementary information to enhance both HSI spatial resolution and MSI spectral resolution. While hyperspectral-multispectral fusion (HMF) has been widely studied, the unregistered setting remains challenging. Many existing methods focus solely on MSI super-resolution, leaving HSI unchanged. Supervised deep learning approaches were proposed for HSI super-resolution, but rely on accurate training data, which is often unavailable. Moreover, theoretical analyses largely address the co-registered case, leaving unregistered HMF poorly understood. In this work, an unsupervised framework is proposed to simultaneously super-resolve both MSI and HSI. The method integrates coupled spectral unmixing for MSI super-resolution with latent-space adversarial learning for HSI super-resolution. Theoretical guarantees on the recoverability of the super-resolution MSI and HSI are established under reasonable generative models -- providing, to our best knowledge, the first such insights for unregistered HMF. The approach is validated on semi-real and real HSI-MSI pairs across diverse conditions.
Jiahui Song, Sagar Shrestha, Xiao Fu
The authors are with the School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, OR 97331, United States.
Hyperspectral image fusion aims to reconstruct high-spatial-resolution hyperspectral images (HR-HSI) by integrating complementary information from multi-source inputs. Despite recent progress, existing methods still face two critical challenges: (1) inadequate reconstruction of anisotropic spatial structures, resulting in blurred details and compromised spatial quality; and (2) spectral distortion during fusion, which hinders fine-grained spectral representation. To address these issues, we propose ASSR-Net: an Anisotropic Structure-Aware and Spectrally Recalibrated Network for Hyperspectral Image Fusion. ASSR-Net adopts a two-stage fusion strategy comprising anisotropic structure-aware spatial enhancement (ASSE) and hierarchical prior-guided spectral calibration (HPSC). In the first stage, a directional perception fusion module adaptively captures structural features along multiple orientations, effectively reconstructing anisotropic spatial patterns. In the second stage, a spectral recalibration module leverages the original low-resolution HSI as a spectral prior to explicitly correct spectral deviations in the fused results, thereby enhancing spectral fidelity. Extensive experiments on various benchmark datasets demonstrate that ASSR-Net consistently outperforms state-of-the-art methods, achieving superior spatial detail preservation and spectral consistency.
Qiya Song, Hongzhi Zhou, Lishan Tan +2
School of Information Science and Engineering, Hunan Normal University, Changsha, Hunan 410081, China. · the School of Robotics, Hunan University, Changsha, Hunan 410082, China.
Unregistered hyperspectral image (HSI) super-resolution (SR) typically aims to enhance a low-resolution HSI using an unregistered high-resolution reference image. In this paper, we propose an unmixing-based fusion framework that decouples spatial-spectral information to simultaneously mitigate the impact of unregistered fusion and enhance the learnability of SR models. Specifically, we first utilize singular value decomposition for initial spectral unmixing, preserving the original endmembers while dedicating the subsequent network to enhancing the initial abundance map. To leverage the spatial texture of the unregistered reference, we introduce a coarse-to-fine deformable aggregation module, which first estimates a pixel-level flow and a similarity map using a coarse pyramid predictor. It further performs fine sub-pixel refinement to achieve deformable aggregation of the reference features. The aggregative features are then refined via a series of spatial-channel abundance cross-attention blocks. Furthermore, a spatial-channel modulated fusion module is presented to merge encoder-decoder features using dynamic gating weights, yielding a high-quality, high-resolution HSI. Experimental results on simulated and real datasets confirm that our proposed method achieves state-of-the-art super-resolution performance. The code will be available at https://github.com/yingkai-zhang/UAFL.
Yingkai Zhang, Tao Zhang, Jing Nie +1
Beijing Institute of Technology · Hangzhou Dianzi University