ASSR-Net: Anisotropic Structure-Aware and Spectrally Recalibrated Network for Hyperspectral Image Fusion

Qiya Song; Hongzhi Zhou; Lishan Tan; Renwei Dian; Shutao Li

cs.CVApr 7, 2026

ASSR-Net: Anisotropic Structure-Aware and Spectrally Recalibrated Network for Hyperspectral Image Fusion

Authors: Qiya Song, Hongzhi Zhou, Lishan Tan, Renwei Dian, Shutao Li

Organizations: School of Information Science and Engineering, Hunan Normal University, Changsha, Hunan 410081, China. · the School of Robotics, Hunan University, Changsha, Hunan 410082, China.

Abstract

Hyperspectral image fusion aims to reconstruct high-spatial-resolution hyperspectral images (HR-HSI) by integrating complementary information from multi-source inputs. Despite recent progress, existing methods still face two critical challenges: (1) inadequate reconstruction of anisotropic spatial structures, resulting in blurred details and compromised spatial quality; and (2) spectral distortion during fusion, which hinders fine-grained spectral representation. To address these issues, we propose ASSR-Net: an Anisotropic Structure-Aware and Spectrally Recalibrated Network for Hyperspectral Image Fusion. ASSR-Net adopts a two-stage fusion strategy comprising anisotropic structure-aware spatial enhancement (ASSE) and hierarchical prior-guided spectral calibration (HPSC). In the first stage, a directional perception fusion module adaptively captures structural features along multiple orientations, effectively reconstructing anisotropic spatial patterns. In the second stage, a spectral recalibration module leverages the original low-resolution HSI as a spectral prior to explicitly correct spectral deviations in the fused results, thereby enhancing spectral fidelity. Extensive experiments on various benchmark datasets demonstrate that ASSR-Net consistently outperforms state-of-the-art methods, achieving superior spatial detail preservation and spectral consistency.

Figures & tables

Fig. 1: Anisotropy response map of the input image. Brighter regions indicate stronger anisotropy, demonstrating that the image contains substantial directional features.

Fig. 2: Architecture of the proposed ASSR-Net. Stage I enhances anisotropic spatial structures from the low-resolution hyperspectral image (LR-HSI) $\mathbf{X}$ and high-resolution multispectral image (HR-MSI) $\mathbf{Y}$ using a cascade of Variable Direction-Aware Encoder (VDAE), producing an intermediate estimate $\mathbf{Z}_{\text{init}}$ . Stage II then refines spectral fidelity via a Global Spectral Recalibration Transformer (GSRT) that incorporates a spectral prior extracted from $\mathbf{LR-HSI}$ , yielding the Reconstructed HR-HSI $\hat{\mathbf{Z}}$ . Two L1 losses, $\lambda_{1}$ (for $\hat{\mathbf{Z}}$ ) and $\lambda_{2}$ (for $\mathbf{Z}_{\text{init}}$ ), jointly supervise training.

Fig. 3: Detailed architecture of the Directional Attention Enhancement (DAE) module and its core component, the Anisotropic Structure Transform (AST). DAE enhances directional features via a global-local decomposition followed by a learnable gating fusion. AST performs multi-scale subband decomposition, differentiable Radon approximation along adaptively predicted directions, and frequency-adaptive enhancement in the wavelet domain to capture anisotropic structures.

Fig. 4: Illustration of the two complementary attention mechanisms in the Spectral Prior-Guided Attention (SPGA) module. Spectral-Guided Attention (SGA): the spectral prior $\mathbf{sp}$ generates a channel modulation vector $\mathbf{s}$ and a guidance matrix $\mathbf{G}$ , biasing self-attention weights toward spectrally consistent regions. Spatial Differential Attention (SDA): a spatial mask $\mathbf{M}_{\text{spa}}$ computed from local gradients highlights edges and textures, followed by standard self-attention. The outputs of both branches are adaptively fused via a learnable gating mechanism.

Method	CAVE Dataset					Harvard Dataset
Method	PSNR $\uparrow$	SAM $\downarrow$	UIQI $\uparrow$	SSIM $\uparrow$	ERGAS $\downarrow$	PSNR $\uparrow$	SAM $\downarrow$	UIQI $\uparrow$	SSIM $\uparrow$	ERGAS $\downarrow$
DHIF-Net	$47.2893$	$2.4101$	$0.9733$	$0.9947$	$0.6034$	$47.7261$	$2.7313$	$0.8938$	0.9855	$0.5667$
DSPNet	$47.8060$	$2.5041$	$0.9729$	$0.9944$	$0.5756$	$47.6887$	$2.7325$	$0.8941$	$0.9852$	$0.5677$
LRTN	$47.8958$	$2.3644$	$0.9704$	$0.9945$	$0.5612$	$47.6663$	$2.7417$	$0.8941$	$0.9852$	$0.5687$
MIMO-SST	$48.2001$	$2.5618$	$0.9697$	$0.9942$	$0.5575$	$47.6364$	$2.7731$	$0.8939$	$0.9851$	$0.5697$
SINet	$48.6726$	$2.3013$	$0.9743$	$0.9950$	$0.5261$	$47.8412$	$2.7110$	$0.8961$	0.9855	$0.5627$

TABLE I: Quantitative comparison on the CAVE and Harvard datasets. Best results are in bold and the second best are underlined .

Method	DHIF-Net	DSPNet	LRTN	MIMO-SST	SINet	OTIAS	SRLF	Ours
QNR $\uparrow$	0.9711	0.9870	0.9862	0.9871	0.9869	0.9871	0.9872	0.9873

TABLE II: No-reference quality assessment (QNR) on the Gaofen5 dataset. The best result is in bold and the second best is underlined .

Fig. 5: Visual comparison on the CAVE dataset. From left to right: DHIF-Net, DSPNet, LRTN, MIMO-SST, SINet, OTIAS, SRLF, ASSR-Net (ours), and GT.

Fig. 6: Visual comparison on the Harvard dataset. From left to right: DHIF-Net, DSPNet, LRTN, MIMO-SST, SINet, OTIAS, SRLF, ASSR-Net (ours), and GT.

Fig. 7: Visual results on the Gaofen5 dataset. From left to right: DHIF-Net, DSPNet, LRTN, MIMO-SST, SINet, OTIAS, SRLF, ASSR-Net (ours), and GT.

$K$	TB	Params (M) $\downarrow$	GFLOPs $\downarrow$	PSNR $\uparrow$	SAM $\downarrow$	Inference Time (ms) $\downarrow$
$8$	$1$	$10.553$	$18.423$	$49.1465$	$2.1771$	$11.55$
$8$	$2$	$13.308$	$24.169$	$49.1377$	$2.1435$	$13.09$
$8$	$3$	$12.357$	$31.815$	$49.0961$	$2.1064$	$14.39$
$16$	$1$	$10.577$	$18.471$	$49.2673$	$2.1483$	$12.09$
$16$	$2$	$13.332$	$24.218$	$49.1294$	$2.0941$	$13.69$
$16$	$3$	$14.668$	$31.964$	$49.1649$	$2.0712$	$14.95$

TABLE III: Hyperparameter sensitivity analysis of the number of projection directions

K

in ASSE and the number of Transformer Blocks (TB) in GSRT on the CAVE dataset, including computational complexity and inference time (measured on a

64\times 64

patch, averaged over 10 runs).

DACI	DAE	Fusion	GSRT	PSNR $\uparrow$	SAM $\downarrow$	UIQI $\uparrow$	SSIM $\uparrow$	ERGAS $\downarrow$
$\times$	$\times$	$\times$	$\times$	$47.8014$	$2.4821$	$0.9704$	$0.9932$	$0.5675$
✓	$\times$	$\times$	$\times$	$48.1632$	$2.3743$	$0.9712$	$0.9939$	$0.5586$
✓	✓	$\times$	$\times$	$48.4457$	$2.3176$	$0.9721$	$0.9941$	$0.5412$
✓	✓	✓	$\times$	$48.8363$	$2.2312$	$0.9742$	$0.9952$	$0.5144$
$\times$	✓	✓	✓	$49.2151$	$2.1151$	$0.9746$	$0.9955$	$0.4976$
✓	✓	✓	✓	49.5820	2.05	0.9769	0.9961	0.4725

TABLE IV: Ablation study on CAVE dataset. ✓ denotes module included,

\times

denotes module excluded.

$\alpha$	$\beta$	PSNR $\uparrow$	SAM $\downarrow$
$1.0$	$0.0$	$49.1982$	$2.139$
$0.9$	$0.1$	$49.0507$	$2.07$
0.8	0.2	49.5820	2.05
$0.7$	$0.3$	$49.3841$	$2.0761$
$0.6$	$0.4$	$48.9343$	$2.1219$
$0.5$	$0.5$	$49.0914$	$2.0842$

TABLE V: Impact of loss-weight balance (

\alpha\lambda_{1}+\beta\lambda_{2}

) on final performance (CAVE dataset).

Category	F1 scores(%)
	LR-HSI	Predicted HR-HSI
Healthy grass	87.5	92.7
Stressed grass	88.7	94.1
Synthetic grass	92.7	99.9
Trees	83.6	96.5
Soil	97.8	98.8

TABLE VI: Classification results (F1 scores, %) of LR-HSI and the HR-HSI predicted by ASSR-Net on the Houston dataset.

Fig. 8: Visual comparison of Stage I and Stage II outputs. Top row: Spectral error maps (SAM) after Stage I; middle row: Spectral error maps after Stage II; bottom row: Pseudo-color RGB images of the final fusion result. Warmer colors in error maps indicate larger spectral deviation. Stage II significantly reduces errors, especially in complex regions.

Panel of Figure 15 — Fig. 9: Classification results before and after fusion. (a) Classification result of LR-HSI. (b) Classification result of the predicted HR-HSI. (c) Reference.

Panel of Figure 16 — Fig. 10: Spectral profiles of three points (marked in the RGB images) for three different scenes.Stage 2 consistently reduces spectral deviations compared to Stage 1 and outperforms other methods.

Explore similar work

Learning Spatial-Spectral Refinement and Calibrating Complementary Observations for Hyperspectral Image Super-Resolution
Sep 4, 2026Liqian Yang, Xingchi Chen, Xinfeng Gui +2Hyperspectral Image Super-Resolution Image Reconstruction
School of Management, Zhengzhou University, Zhengzhou, 450001, China · Center for Intelligent Decision-making and Machine Learning, Xi’an Jiaotong University, Xi’an, 710049, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, China
CoFusion: Multispectral and Hyperspectral Image Fusion via Spectral Coordinate Attention
Apr 12, 2026Baisong LiMulti-Source Image Fusion Hyperspectral Image Super-Resolution
School of Integrated Circuits, Tsinghua University, Beijing 100084, China
Dual-Domain Manifold Modeling for Hyperspectral Image Fusion
Jul 28, 2026Chengxin Xie, Qiya Song, Yangbangyan Jiang +2Hyperspectral Imaging Manifold Learning
College of Information Science and Engineering, Hunan Normal University, Changsha, China · Institute of Information Engineering, Chinese Academy of Sciences,Beijing, 100000, Beijing, China · School of Artificial Intelligence and Robotics, Hunan University, Changsha, China

Sep 4, 2026J/K move · Enter open · S save

Learning Spatial-Spectral Refinement and Calibrating Complementary Observations for Hyperspectral Image Super-Resolution

Liqian Yang, Xingchi Chen, Xinfeng Gui, Xiangyong Cao+1

School of Management, Zhengzhou University, Zhengzhou, 450001, China · Center for Intelligent Decision-making and Machine Learning, Xi’an Jiaotong University, Xi’an, 710049, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, China

Hyperspectral Image Super-Resolution Image Reconstruction Multi-Source Image Fusion Implicit Neural Representations

Open paper AskPDF arXiv

Hyperspectral and multispectral image fusion (HMIF) aims to reconstruct a high-resolution hyperspectral image (HR-HSI) by combining the fine spatial details of a high-resolution multispectral image (HR-MSI) with the rich spectral information of a low-resolution hyperspectral image (LR-HSI). Recent advances in implicit neural representations (INRs) have enabled flexible coordinate-based modeling for HMIF; however, existing INR-based approaches may not fully capture fine-grained spatial structures and rich spectral dependencies. Moreover, the LR-HSI and HR-MSI are primarily incorporated through degradation-consistency constraints, leaving their complementary information underexploited. To address these limitations, we propose Two-Stage Reconstruction with Implicit Tensor Neural Representation (TSR-ITNR), a unified self-supervised framework integrating representation refinement and observation-guided calibration. In Stage 1, TSR-ITNR learns an implicit Tucker representation and refines its low-rank spatial coefficient tensor and spectral basis to better capture fine spatial structures and interband correlations. A fixed pretrained denoiser further provides a deep prior for the preliminary reconstruction. In Stage 2, parameter-free calibration derives complementary and noninterfering corrections from both observations to recover information insufficiently captured in Stage 1. Theoretical analysis establishes the geometry-preserving property of spectral refinement and the orthogonal complementarity of calibration. Extensive experiments on multiple benchmark datasets demonstrate strong quantitative, visual, and spectral reconstruction performance without ground-truth HR-HSI supervision. Beyond conventional reconstruction metrics, we further assess the effectiveness of TSR-ITNR using downstream semantic segmentation accuracy.

Abstract

Explore similar work

Learning Spatial-Spectral Refinement and Calibrating Complementary Observations for Hyperspectral Image Super-Resolution

CoFusion: Multispectral and Hyperspectral Image Fusion via Spectral Coordinate Attention

Dual-Domain Manifold Modeling for Hyperspectral Image Fusion