Unregistered Spectral Image Fusion: Unmixing, Adversarial Learning, and Recoverability

Jiahui Song; Sagar Shrestha; Xiao Fu

eess.IVMar 23, 2026

Unregistered Spectral Image Fusion: Unmixing, Adversarial Learning, and Recoverability

Authors: Jiahui Song, Sagar Shrestha, Xiao Fu

Organizations: The authors are with the School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, OR 97331, United States.

Abstract

This paper addresses the fusion of a pair of spatially unregistered hyperspectral image (HSI) and multispectral image (MSI) covering roughly overlapping regions. HSIs offer high spectral but low spatial resolution, while MSIs provide the opposite. The goal is to integrate their complementary information to enhance both HSI spatial resolution and MSI spectral resolution. While hyperspectral-multispectral fusion (HMF) has been widely studied, the unregistered setting remains challenging. Many existing methods focus solely on MSI super-resolution, leaving HSI unchanged. Supervised deep learning approaches were proposed for HSI super-resolution, but rely on accurate training data, which is often unavailable. Moreover, theoretical analyses largely address the co-registered case, leaving unregistered HMF poorly understood. In this work, an unsupervised framework is proposed to simultaneously super-resolve both MSI and HSI. The method integrates coupled spectral unmixing for MSI super-resolution with latent-space adversarial learning for HSI super-resolution. Theoretical guarantees on the recoverability of the super-resolution MSI and HSI are established under reasonable generative models -- providing, to our best knowledge, the first such insights for unregistered HMF. The approach is validated on semi-real and real HSI-MSI pairs across diverse conditions.

Figures & tables

Fig. 1 : Top: Coregistered HMF. Bottom: Unregistered HMF. M-SRI and H-SRI are SRIs covering the spatial regions of the MSI and the HSI, respectively. In all the images, the third mode is the spectral domain.

Fig. 2 : Data acquisition of co-registered HMF setting; see [ 5 , 4 , 7 ] .

Fig. 3 : Data acquisition model of unregistered HMF setting.

Fig. 4 : LMM representations of the MSI (top) and the HSI (bottom) in a case where $R=4$ .

Fig. 5 : Patch boundaries and patch rotation angle. $\bm{w}$ is a continuous global coordinate (e.g., altitude and longitude from the geographic coordinate system).

Fig. 6 : Illustration of the multi-modal patch generative model in Assumption 4 .

Panel of Figure 7 — Fig. 7 : RGB rendering of recovered $\widehat{\underline{\bm{Y}}}^{({\rm M})}_{\rm SRI}$ (Pavia University); nearest-neighbor downsampling used for degradation.

Panel of Figure 8 — Fig. 8 : RGB rendering of recovered $\widehat{\underline{\bm{Y}}}^{({\rm H})}_{\rm SRI}$ (Pavia University); nearest-neighbor downsampling used for degradation.

Panel of Figure 9 — Fig. 9 : RGB rendering of recovered $\widehat{\underline{\bm{Y}}}^{({\rm M})}_{\rm SRI}$ (Terrain); uniform downsampling used for degradation.

Panel of Figure 10 — Fig. 10 : RGB rendering of recovered $\widehat{\underline{\bm{Y}}}^{({\rm M})}_{\rm SRI}$ (Terrain); uniform downsampling used for degradation.

Panel of Figure 11 — Fig. 11 : Comparison of IAT and the proposed FRESCO .

Panel of Figure 12 — Fig. 12 : Performance under various $T(=L)$ on Pavia University.

Panel of Figure 13 — Fig. 13 : Performance under various $T(=L)$ on Terrain.

Panel of Figure 14 — Fig. 14 : Performance comparison of MSR (top row) and HSR (bottom row) tasks under various spatial unregistration levels.

Panel of Figure 15 — Fig. 15 : RGB renderings of from the recovered SRIs with estimated $\widehat{\bm{P}}^{({\rm M})}$ and the ground-truth $\bm{P}^{({\rm M})}$ (Pavia University); Gaussian kernel downsampling with $\sigma=1.7$ .

Stage	Method	Setting	Device	Runtime (min)
MSR	UHIF	Per-pair optimization	CPU	0.32
	u2MDN	Per-pair optimization	GPU	28.49
	IARF	Per-pair optimization	CPU	1.65
	NBFusion	Per-pair optimization	CPU	6.83
	FRESCO (MSR)	Per-pair optimization	CPU	2.56
HSR	Lanczos	Direct interpolation	CPU	0.007

TABLE VII : Runtime comparison on the Pavia University dataset.

Panel of Figure 17 — Fig. 16 : RGB renderings of the recovered SRIs on real data. First row: The recovered SRIs for MSI region; Second row: The recovered SRIs for HSI region.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
Conv	2D convolutional layer
BN	Batch Normalization
LReLU	LeakyReLU activation (slope = $0.2$ )
MaxPool( $u$ )	Max pooling with kernel size of $u$ and stride of $u$ .
Upsample( $u$ )	Bilinear upsampling by factor of $u$ .
K- $k$	convolution kernel of size $k$

Appendix

TABLE IX : Notation for Network Components

Layer Number	Layer Details
1	Upsample( $s$ )
2	Conv( $1\rightarrow 64$ , K- $1$ , S- $1$ , ZP- $0$ , $\#$ , LReLU)
3	Conv( $64\rightarrow 64$ , K- $3$ , S- $1$ , ZP- $1$ , BN, LReLU)
4	Conv( $64\rightarrow 64$ , K- $3$ , S- $1$ , ZP- $1$ , BN, LReLU)
5	Enc( $64\rightarrow 128$ )
6	Enc( $128\rightarrow 256$ )

Appendix

TABLE X : Spatial super-resolution neural network

\bm{f}

; if no normalization or activation function is applied, the corresponding position is denoted by

\#

Layer Number	Layer Details
1	Conv( $1\rightarrow 64$ , K- $1$ , S- $1$ , ZP- $0$ , $\#$ , LReLU)
2	ResBlock( $64\rightarrow 64$ )
3	ResBlock( $64\rightarrow 64$ )
4	Conv( $64\rightarrow 64$ , K- $3$ , S- $s$ , RP- $1$ , $\#$ , LReLU)
5	Conv( $64\rightarrow 1$ , K- $3$ , S- $1$ , RP- $1$ , $\#$ , LReLU)

Appendix

TABLE XI : Inverse mapping neural network

\bm{g}

Layer Number	Layer Details
1	Conv( $1\rightarrow 64$ , K- $3$ , S- $1$ , ZP- $1$ , $\#$ , LReLU).
2	Conv( $64\rightarrow 64$ , K- $3$ , S- $2$ , ZP- $1$ , BN, LReLU)
3	Conv( $64\rightarrow 128$ , K- $3$ , S- $1$ , ZP- $1$ , BN, LReLU)
4	Conv( $128\rightarrow 128$ , K- $3$ , S- $2$ , ZP- $1$ , BN, LReLU)
5	Conv( $128\rightarrow 256$ , K- $3$ , S- $1$ , ZP- $1$ , BN, LReLU)
6	Conv( $256\rightarrow 256$ , K- $3$ , S- $2$ , ZP- $1$ , BN, LReLU)

Appendix

TABLE XII : Discriminator

\bm{d}

Explore similar work

Mar 9, 2026

Enhancing Unregistered Hyperspectral Image Super-Resolution via Unmixing-based Abundance Fusion Learning

Unregistered hyperspectral image (HSI) super-resolution (SR) typically aims to enhance a low-resolution HSI using an unregistered high-resolution reference image. In this paper, we propose an unmixing-based fusion framework that decouples spatial-spectral information to simultaneously mitigate the impact of unregistered fusion and enhance the learnability of SR models. Specifically, we first utilize singular value decomposition for initial spectral unmixing, preserving the original endmembers while dedicating the subsequent network to enhancing the initial abundance map. To leverage the spatial texture of the unregistered reference, we introduce a coarse-to-fine deformable aggregation module, which first estimates a pixel-level flow and a similarity map using a coarse pyramid predictor. It further performs fine sub-pixel refinement to achieve deformable aggregation of the reference features. The aggregative features are then refined via a series of spatial-channel abundance cross-attention blocks. Furthermore, a spatial-channel modulated fusion module is presented to merge encoder-decoder features using dynamic gating weights, yielding a high-quality, high-resolution HSI. Experimental results on simulated and real datasets confirm that our proposed method achieves state-of-the-art super-resolution performance. The code will be available at https://github.com/yingkai-zhang/UAFL.

Yingkai Zhang, Tao Zhang, Jing Nie +1

Beijing Institute of Technology · Hangzhou Dianzi University

Hyperspectral Image Super-Resolution Hyperspectral Imaging Multi-Source Image Fusion

Ask PDF

Sep 4, 2026

Learning Spatial-Spectral Refinement and Calibrating Complementary Observations for Hyperspectral Image Super-Resolution

Hyperspectral and multispectral image fusion (HMIF) aims to reconstruct a high-resolution hyperspectral image (HR-HSI) by combining the fine spatial details of a high-resolution multispectral image (HR-MSI) with the rich spectral information of a low-resolution hyperspectral image (LR-HSI). Recent advances in implicit neural representations (INRs) have enabled flexible coordinate-based modeling for HMIF; however, existing INR-based approaches may not fully capture fine-grained spatial structures and rich spectral dependencies. Moreover, the LR-HSI and HR-MSI are primarily incorporated through degradation-consistency constraints, leaving their complementary information underexploited. To address these limitations, we propose Two-Stage Reconstruction with Implicit Tensor Neural Representation (TSR-ITNR), a unified self-supervised framework integrating representation refinement and observation-guided calibration. In Stage 1, TSR-ITNR learns an implicit Tucker representation and refines its low-rank spatial coefficient tensor and spectral basis to better capture fine spatial structures and interband correlations. A fixed pretrained denoiser further provides a deep prior for the preliminary reconstruction. In Stage 2, parameter-free calibration derives complementary and noninterfering corrections from both observations to recover information insufficiently captured in Stage 1. Theoretical analysis establishes the geometry-preserving property of spectral refinement and the orthogonal complementarity of calibration. Extensive experiments on multiple benchmark datasets demonstrate strong quantitative, visual, and spectral reconstruction performance without ground-truth HR-HSI supervision. Beyond conventional reconstruction metrics, we further assess the effectiveness of TSR-ITNR using downstream semantic segmentation accuracy.

Liqian Yang, Xingchi Chen, Xinfeng Gui +2

School of Management, Zhengzhou University, Zhengzhou, 450001, China · Center for Intelligent Decision-making and Machine Learning, Xi’an Jiaotong University, Xi’an, 710049, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, China

Hyperspectral Image Super-Resolution Image Reconstruction Multi-Source Image Fusion Implicit Neural Representations

Ask PDF

Sep 1, 2026

Lightweight Interpretable RGB-Guided Hyperspectral Super-Resolution under Real Cross-resolution Misalignment

Compact snapshot hyperspectral cameras provide rich instantaneous spectral measurements for ground-level machine vision, but at lower spatial resolution than standard RGB cameras. RGB-guided hyperspectral super-resolution (HSR) addresses this limitation by transferring spatial detail from a high-resolution RGB guide to a low-resolution hyperspectral image (HSI). These dual-camera systems are typically in a horizontal rig geometry, requiring cross-camera image alignment due to different fields of view. However, residual misregistration can inject spurious high-frequency details. Existing learned unaligned-fusion methods are usually trained for a fixed spectral support and spatial scale factors and can be computationally demanding, limiting their flexibility across sensors. We propose a lightweight and interpretable RGB-guided HSR framework combining cross-modal flow alignment with model-based Gram-Schmidt orthogonalization fusion. The method first warps the RGB guide onto the HSI grid, then estimates an energy-based confidence weight map by measuring local alignment reliability. This map is then used both in a weighted least-squares spectral regression and in a gated fusion between the super-resolved estimate and an HSI-preserving estimate. Unlike existing learned methods, the proposed framework has a low computational footprint and supports VIS-NIR spectral supports and scale factors without retraining. Experiments on the Real benchmark show that the proposed method improves reconstruction accuracy over learned fusion baselines while remaining substantially faster. On a 34-frame sequence acquired with our real RGB-HSI dual-camera setup, a reduced-resolution quantitative evaluation validates the method under genuine cross-sensor radiometric, noise, and geometric differences, while native-resolution qualitative results demonstrate deployment on the full 51-band VIS-NIR acquisition.

Mohamad Jouni, Aurélien Godet, Mauro Dalla Mura

Univ. Grenoble Alpes, CNRS Grenoble INP, GIPSA-Lab 38000 Grenoble, France · Institut Universitaire de France (IUF) Paris, France

Hyperspectral Image Super-Resolution Hyperspectral Imaging

Ask PDF