Unregistered Spectral Image Fusion: Unmixing, Adversarial Learning, and Recoverability

Jiahui Song; Sagar Shrestha; Xiao Fu

eess.IVMar 23, 2026

Unregistered Spectral Image Fusion: Unmixing, Adversarial Learning, and Recoverability

Authors: Jiahui Song, Sagar Shrestha, Xiao Fu

Organizations: The authors are with the School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, OR 97331, United States.

Abstract

This paper addresses the fusion of a pair of spatially unregistered hyperspectral image (HSI) and multispectral image (MSI) covering roughly overlapping regions. HSIs offer high spectral but low spatial resolution, while MSIs provide the opposite. The goal is to integrate their complementary information to enhance both HSI spatial resolution and MSI spectral resolution. While hyperspectral-multispectral fusion (HMF) has been widely studied, the unregistered setting remains challenging. Many existing methods focus solely on MSI super-resolution, leaving HSI unchanged. Supervised deep learning approaches were proposed for HSI super-resolution, but rely on accurate training data, which is often unavailable. Moreover, theoretical analyses largely address the co-registered case, leaving unregistered HMF poorly understood. In this work, an unsupervised framework is proposed to simultaneously super-resolve both MSI and HSI. The method integrates coupled spectral unmixing for MSI super-resolution with latent-space adversarial learning for HSI super-resolution. Theoretical guarantees on the recoverability of the super-resolution MSI and HSI are established under reasonable generative models -- providing, to our best knowledge, the first such insights for unregistered HMF. The approach is validated on semi-real and real HSI-MSI pairs across diverse conditions.

Figures & tables

Fig. 1 : Top: Coregistered HMF. Bottom: Unregistered HMF. M-SRI and H-SRI are SRIs covering the spatial regions of the MSI and the HSI, respectively. In all the images, the third mode is the spectral domain.

Fig. 2 : Data acquisition of co-registered HMF setting; see [ 5 , 4 , 7 ] .

Fig. 3 : Data acquisition model of unregistered HMF setting.

Fig. 4 : LMM representations of the MSI (top) and the HSI (bottom) in a case where $R=4$ .

Fig. 5 : Patch boundaries and patch rotation angle. $\bm{w}$ is a continuous global coordinate (e.g., altitude and longitude from the geographic coordinate system).

Fig. 6 : Illustration of the multi-modal patch generative model in Assumption 4 .

Panel of Figure 7 — Fig. 7 : RGB rendering of recovered $\widehat{\underline{\bm{Y}}}^{({\rm M})}_{\rm SRI}$ (Pavia University); nearest-neighbor downsampling used for degradation.

Panel of Figure 8 — Fig. 8 : RGB rendering of recovered $\widehat{\underline{\bm{Y}}}^{({\rm H})}_{\rm SRI}$ (Pavia University); nearest-neighbor downsampling used for degradation.

Panel of Figure 9 — Fig. 9 : RGB rendering of recovered $\widehat{\underline{\bm{Y}}}^{({\rm M})}_{\rm SRI}$ (Terrain); uniform downsampling used for degradation.

Panel of Figure 10 — Fig. 10 : RGB rendering of recovered $\widehat{\underline{\bm{Y}}}^{({\rm M})}_{\rm SRI}$ (Terrain); uniform downsampling used for degradation.

Panel of Figure 11 — Fig. 11 : Comparison of IAT and the proposed FRESCO .

Panel of Figure 12 — Fig. 12 : Performance under various $T(=L)$ on Pavia University.

Panel of Figure 13 — Fig. 13 : Performance under various $T(=L)$ on Terrain.

Panel of Figure 14 — Fig. 14 : Performance comparison of MSR (top row) and HSR (bottom row) tasks under various spatial unregistration levels.

Panel of Figure 15 — Fig. 15 : RGB renderings of from the recovered SRIs with estimated $\widehat{\bm{P}}^{({\rm M})}$ and the ground-truth $\bm{P}^{({\rm M})}$ (Pavia University); Gaussian kernel downsampling with $\sigma=1.7$ .

Stage	Method	Setting	Device	Runtime (min)
MSR	UHIF	Per-pair optimization	CPU	0.32
	u2MDN	Per-pair optimization	GPU	28.49
	IARF	Per-pair optimization	CPU	1.65
	NBFusion	Per-pair optimization	CPU	6.83
	FRESCO (MSR)	Per-pair optimization	CPU	2.56
HSR	Lanczos	Direct interpolation	CPU	0.007

TABLE VII : Runtime comparison on the Pavia University dataset.

Panel of Figure 17 — Fig. 16 : RGB renderings of the recovered SRIs on real data. First row: The recovered SRIs for MSI region; Second row: The recovered SRIs for HSI region.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
Conv	2D convolutional layer
BN	Batch Normalization
LReLU	LeakyReLU activation (slope = $0.2$ )
MaxPool( $u$ )	Max pooling with kernel size of $u$ and stride of $u$ .
Upsample( $u$ )	Bilinear upsampling by factor of $u$ .
K- $k$	convolution kernel of size $k$

Appendix

TABLE IX : Notation for Network Components

Layer Number	Layer Details
1	Upsample( $s$ )
2	Conv( $1\rightarrow 64$ , K- $1$ , S- $1$ , ZP- $0$ , $\#$ , LReLU)
3	Conv( $64\rightarrow 64$ , K- $3$ , S- $1$ , ZP- $1$ , BN, LReLU)
4	Conv( $64\rightarrow 64$ , K- $3$ , S- $1$ , ZP- $1$ , BN, LReLU)
5	Enc( $64\rightarrow 128$ )
6	Enc( $128\rightarrow 256$ )

Appendix

TABLE X : Spatial super-resolution neural network

\bm{f}

; if no normalization or activation function is applied, the corresponding position is denoted by

\#

Layer Number	Layer Details
1	Conv( $1\rightarrow 64$ , K- $1$ , S- $1$ , ZP- $0$ , $\#$ , LReLU)
2	ResBlock( $64\rightarrow 64$ )
3	ResBlock( $64\rightarrow 64$ )
4	Conv( $64\rightarrow 64$ , K- $3$ , S- $s$ , RP- $1$ , $\#$ , LReLU)
5	Conv( $64\rightarrow 1$ , K- $3$ , S- $1$ , RP- $1$ , $\#$ , LReLU)

Appendix

TABLE XI : Inverse mapping neural network

\bm{g}

Layer Number	Layer Details
1	Conv( $1\rightarrow 64$ , K- $3$ , S- $1$ , ZP- $1$ , $\#$ , LReLU).
2	Conv( $64\rightarrow 64$ , K- $3$ , S- $2$ , ZP- $1$ , BN, LReLU)
3	Conv( $64\rightarrow 128$ , K- $3$ , S- $1$ , ZP- $1$ , BN, LReLU)
4	Conv( $128\rightarrow 128$ , K- $3$ , S- $2$ , ZP- $1$ , BN, LReLU)
5	Conv( $128\rightarrow 256$ , K- $3$ , S- $1$ , ZP- $1$ , BN, LReLU)
6	Conv( $256\rightarrow 256$ , K- $3$ , S- $2$ , ZP- $1$ , BN, LReLU)

Appendix

TABLE XII : Discriminator

\bm{d}

Explore similar work

Enhancing Unregistered Hyperspectral Image Super-Resolution via Unmixing-based Abundance Fusion Learning
Mar 9, 2026Yingkai Zhang, Tao Zhang, Jing Nie +1Hyperspectral Image Super-Resolution Hyperspectral Imaging
Beijing Institute of Technology · Hangzhou Dianzi University
Learning Spatial-Spectral Refinement and Calibrating Complementary Observations for Hyperspectral Image Super-Resolution
Sep 4, 2026Liqian Yang, Xingchi Chen, Xinfeng Gui +2Hyperspectral Image Super-Resolution Image Reconstruction
School of Management, Zhengzhou University, Zhengzhou, 450001, China · Center for Intelligent Decision-making and Machine Learning, Xi’an Jiaotong University, Xi’an, 710049, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, China
Lightweight Interpretable RGB-Guided Hyperspectral Super-Resolution under Real Cross-resolution Misalignment
Sep 1, 2026Mohamad Jouni, Aurélien Godet, Mauro Dalla MuraHyperspectral Image Super-Resolution Hyperspectral Imaging
Univ. Grenoble Alpes, CNRS Grenoble INP, GIPSA-Lab 38000 Grenoble, France · Institut Universitaire de France (IUF) Paris, France

Mar 9, 2026J/K move · Enter open · S save

Enhancing Unregistered Hyperspectral Image Super-Resolution via Unmixing-based Abundance Fusion Learning

Yingkai Zhang, Tao Zhang, Jing Nie, Ying Fu

Beijing Institute of Technology · Hangzhou Dianzi University

Hyperspectral Image Super-Resolution Hyperspectral Imaging Multi-Source Image Fusion

Open paper AskPDF arXiv

Unregistered hyperspectral image (HSI) super-resolution (SR) typically aims to enhance a low-resolution HSI using an unregistered high-resolution reference image. In this paper, we propose an unmixing-based fusion framework that decouples spatial-spectral information to simultaneously mitigate the impact of unregistered fusion and enhance the learnability of SR models. Specifically, we first utilize singular value decomposition for initial spectral unmixing, preserving the original endmembers while dedicating the subsequent network to enhancing the initial abundance map. To leverage the spatial texture of the unregistered reference, we introduce a coarse-to-fine deformable aggregation module, which first estimates a pixel-level flow and a similarity map using a coarse pyramid predictor. It further performs fine sub-pixel refinement to achieve deformable aggregation of the reference features. The aggregative features are then refined via a series of spatial-channel abundance cross-attention blocks. Furthermore, a spatial-channel modulated fusion module is presented to merge encoder-decoder features using dynamic gating weights, yielding a high-quality, high-resolution HSI. Experimental results on simulated and real datasets confirm that our proposed method achieves state-of-the-art super-resolution performance. The code will be available at https://github.com/yingkai-zhang/UAFL.

Abstract

Explore similar work

Enhancing Unregistered Hyperspectral Image Super-Resolution via Unmixing-based Abundance Fusion Learning

Learning Spatial-Spectral Refinement and Calibrating Complementary Observations for Hyperspectral Image Super-Resolution

Lightweight Interpretable RGB-Guided Hyperspectral Super-Resolution under Real Cross-resolution Misalignment