Fast Model-guided Instance-wise Adaptation Framework for Real-world Pansharpening with Fidelity Constraints

Zhiqi Yang; Jin-Liang Xiao; Shan Yin; Liang-Jian Deng; Gemine Vivone

cs.CVApr 10, 2026

Fast Model-guided Instance-wise Adaptation Framework for Real-world Pansharpening with Fidelity Constraints

Authors: Zhiqi Yang, Jin-Liang Xiao, Shan Yin, Liang-Jian Deng, Gemine Vivone

Organizations: University of Electronic Science and Technology of China, Chengdu, China · National Research Council, Institute of Methodologies for Environmental Analysis (CNR-IMAA), Tito, Italy

Abstract

Pansharpening aims to generate high-resolution multispectral (HRMS) images by fusing low-resolution multispectral (LRMS) and high-resolution panchromatic (PAN) images while preserving both spectral and spatial information. Although deep learning (DL)-based pansharpening methods achieve impressive performance, they require high training cost and large datasets, and often degrade when the test distribution differs from training, limiting generalization. Recent zero-shot methods, trained on a single PAN/LRMS pair, offer strong generalization but suffer from limited fusion quality, high computational overhead, and slow convergence. To address these issues, we propose FMG-Pan, a fast and generalizable model-guided instance-wise adaptation framework for real-world pansharpening, achieving both cross-sensor generality and rapid training-inference. The framework leverages a pretrained model to guide a lightweight adaptive network through joint optimization with spectral and physical fidelity constraints. We further design a novel physical fidelity term to enhance spatial detail preservation. Extensive experiments on real-world datasets under both intra- and cross-sensor settings demonstrate state-of-the-art performance. On the WorldView-3 dataset, FMG-Pan completes training and inference for a 512x512x8 image within 3 seconds on an RTX 3090 GPU, significantly faster than existing zero-shot methods, making it suitable for practical deployment.

Figures & tables

Figure 1. Comparison among traditional deep learning methods, zero-shot methods, and the proposed FMG-Pan framework in terms of performance, efficiency, and generalization. FMG-Pan achieves a better trade-off by leveraging pretrained models for fast, image-specific adaptation without modifying model architecture or requiring external training data.

Figure 2. Overall framework of the proposed FMG-Pan framework. The pretrained model and the PF module are first executed independently. Subsequently, the parameters of the adaptive model are optimized using three core loss functions: pseudo-reference loss, spectral loss, and physical loss. In the reasoning stage, the final HRMS image is generated by applying the trained adaptive model to the input PAN and LRMS images.

Figure 3. Flowchart of the PF module. At the downsampled resolution, the coefficient matrix $\mathbf{H}$ is obtained by solving Eq. ( 14 ). The reconstructed LRMS $\widehat{\mathcal{Y}}$ is then multiplied with $\mathbf{H}$ along mode-3 to generate the reference detail image for guiding the fusion process.

Figure 4. Comparison of two adaptive models. (1) Standard conv backbone with residual blocks. (2) Lightweight variant using depthwise separable convolutions and fewer channels for reduced complexity. Model (1) is used by default. DWConv $(\cdot,\cdot)$ is the depthwise separable convolution and Conv $(\cdot,\cdot,\cdot)$ is the standard convolution.

Method	WV3 (Real Data): Avg $\pm$ std			WV2 (Real Data): Avg $\pm$ std
Method	HQNR $\uparrow$	$D_{\lambda}\downarrow$	$D_{s}\downarrow$	HQNR $\uparrow$	$D_{\lambda}\downarrow$	$D_{s}\downarrow$
BT-H 2017	0.8659 $\pm$ 0.0568	0.0656 $\pm$ 0.0262	0.0742 $\pm$ 0.0383	0.8300 $\pm$ 0.0430	0.0860 $\pm$ 0.0301	0.0925 $\pm$ 0.0208
C-BDSD 2014	0.8562 $\pm$ 0.0233	0.0874 $\pm$ 0.0236	0.0618 $\pm$ 0.0128	0.6956 $\pm$ 0.0461	0.2253 $\pm$ 0.0488	0.1019 $\pm$ 0.0296
BDSD-PC 2019	0.8673 $\pm$ 0.0543	0.0634 $\pm$ 0.0246	0.0749 $\pm$ 0.0359	0.8286 $\pm$ 0.0432	0.1413 $\pm$ 0.0320	${0.0356\pm 0.0213}$
MTF-GLP 2006	0.9026 $\pm$ 0.0444	0.0373 $\pm$ 0.0124	0.0628 $\pm$ 0.0359	0.8549 $\pm$ 0.0475	0.0582 $\pm$ 0.0221	0.0930 $\pm$ 0.0320
MTF-GLP-FS 2018	0.9127 $\pm$ 0.0348	0.0357 $\pm$ 0.0106	0.0537 $\pm$ 0.0273	0.8658 $\pm$ 0.0415	0.0563 $\pm$ 0.0212	0.0830 $\pm$ 0.0260

Table 1. Performance comparison on real-world data of WorldView-3 (WV3) and WorldView-2 (WV2) datasets. The reported values represent the average performance over 20 test images. ( Red : best; Blue : second best)

Figure 5. Visual fusion images and related HQNR maps on a full resolution WV3 example.

Figure 6. Visual fusion images and the related HQNR maps on a full resolution WV2 example.

Figure 7. Comparison of HQNR for FusionMamba and LAGNet with and without the FMG-Pan framework across two real-world datasets (WV3, WV2). Note: * denotes LAGNet, without * denotes FusionMamba.

Name	HQNR $\uparrow$	$\boldsymbol{D_{\lambda}}\downarrow$	$\boldsymbol{D_{s}}\downarrow$
FMG-Pan	0.9637 $\pm$ 0.0116	0.0213 $\pm$ 0.0059	0.0153 $\pm$ 0.0077
$\text{FMG-Pan}^{\text{L}}$	0.9636 $\pm$ 0.0150	0.0227 $\pm$ 0.0056	0.0140 $\pm$ 0.0107
w/o pretrained model	0.9491 $\pm$ 0.0128	0.0227 $\pm$ 0.0059	0.0288 $\pm$ 0.0085
w/o spectral loss	0.9308 $\pm$ 0.0298	0.0232 $\pm$ 0.0097	0.0473 $\pm$ 0.0217
w/o physical loss	0.9602 $\pm$ 0.0078	0.0209 $\pm$ 0.0058	0.0193 $\pm$ 0.0039

Table 2. Average results of the ablation study for our FMG-Pan framework on 20 full-resolution WV3 examples. ( Red : best; Blue : second best)

Dataset	Method	HQNR $\uparrow$	$D_{\lambda}\downarrow$	$D_{s}\downarrow$	Time $\downarrow$ (s)
WV3	PsDip	0.9215	0.0191	0.0607	285.37
	ZS-Pan	0.9449	0.0254	0.0306	67.24
	$\text{FMG-Pan}_{\text{LAGNet}}$	0.9522	0.0230	$\pagecolor{blue!10}{0.0255}$	2.84
	$\text{FMG-Pan}_{\text{FusionMamba}}$	0.9637	0.0213	0.0153	2.93
WV2	PsDip	0.8980	0.0385	0.0659	280.23
	ZS-Pan	0.9112	0.0476	0.0435	68.52

Table 3. Average performance comparison of zero-shot methods and the proposed FMG-Pan framework on 20 full resolution test images for both WV3 and WV2 datasets. ( Red : best; Blue : second best)

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8. Time comparison (in seconds) between the proposed FMG-Pan framework (with LAGNet and FusionMamba as pretrained models) and previous zero-shot methods (PsDip, ZS-Pan) on WV3 and WV2 data.

Figure 9. Runtime composition of the proposed FMG-Pan framework using FusionMamba as the pretrained model. The inner ring corresponds to WV3 and the outer ring to WV2, showing the proportion of time spent on training, pretrained model inference, adaptive-model inference, PF module precomputation, and other processes.

Figure 10. Visual fusion images and the related HQNR maps on a full resolution GF2 example. A full-resolution GF2 qualitative comparison with HQNR maps.

Figure 11. Visual fusion images and the related HQNR maps on a full resolution QB example. A full-resolution QB qualitative comparison with HQNR maps.

Figure 12. Comparison of HQNR for FusionMamba with and without the FMG-Pan framework across four real-world datasets (WV3, WV2, GF2, QB)

Method	GF2 (Real-World Data): Avg $\pm$ std			QB (Real-World Data): Avg $\pm$ std
Method	HQNR $\uparrow$	$D_{\lambda}\downarrow$	$D_{s}\downarrow$	HQNR $\uparrow$	$D_{\lambda}\downarrow$	$D_{s}\downarrow$
BT-H 2017	0.8923 $\pm$ 0.0312	0.0832 $\pm$ 0.0180	0.0814 $\pm$ 0.0110	0.8901 $\pm$ 0.0980	0.0457 $\pm$ 0.0136	0.0693 $\pm$ 0.0109
C-BDSD 2014	0.8870 $\pm$ 0.0210	0.0119 $\pm$ 0.0094	0.0689 $\pm$ 0.0079	0.8317 $\pm$ 0.0750	0.0776 $\pm$ 0.0385	0.0689 $\pm$ 0.0089
BDSD-PC 2019	0.8710 $\pm$ 0.0087	0.0633 $\pm$ 0.0078	0.0742 $\pm$ 0.0095	0.8699 $\pm$ 0.0670	0.0671 $\pm$ 0.0074	0.0780 ${\pm}$ 0.0082
MTF-GLP 2006	0.8247 $\pm$ 0.0462	0.0598 $\pm$ 0.0072	0.0545 $\pm$ 0.0011	0.8574 $\pm$ 0.0840	0.0311 $\pm$ 0.0079	0.0546 $\pm$ 0.0087
MTF-GLP-FS 2018	0.8179 $\pm$ 0.0912	0.0371 $\pm$ 0.0015	0.0149 $\pm$ 0.0099	0.8230 $\pm$ 0.0970	0.0495 $\pm$ 0.0058	0.1411 $\pm$ 0.0083

Appendix

Table 4. Performance comparison on real-world GaoFen-2 (GF2) and QuickBird (QB) data. The reported values represent the average performance over 20 test images. ( Red : best; Blue : second best)

Dataset	Method	HQNR $\uparrow$	$D_{\lambda}\downarrow$	$D_{s}\downarrow$	Runtime $\downarrow$ (s)
WV3	$\text{FMG-Pan}_{\text{FusionMamba}}$	0.9637 $\pm$ 0.0116	0.0213 $\pm$ 0.0059	0.0153 $\pm$ 0.0077	2.93 $\pm$ 0.12
	$\text{FMG-Pan}_{\text{FusionMamba}}^{\text{L}}$	0.9636 $\pm$ 0.0150	0.0227 $\pm$ 0.0056	0.0140 $\pm$ 0.0107	2.82 $\pm$ 0.13
	$\text{FMG-Pan}_{\text{FusionMamba}}^{\text{F}}$	0.9644 $\pm$ 0.0140	0.0205 $\pm$ 0.0052	0.0154 $\pm$ 0.0105	1.47 $\pm$ 0.08
WV2	$\text{FMG-Pan}_{\text{FusionMamba}}$	0.9471 $\pm$ 0.0193	0.0315 $\pm$ 0.0173	0.0221 $\pm$ 0.0490	9.75 $\pm$ 0.32
	$\text{FMG-Pan}_{\text{FusionMamba}}^{\text{L}}$	0.9460 $\pm$ 0.0256	0.0325 $\pm$ 0.0179	0.0223 $\pm$ 0.0096	9.55 $\pm$ 0.29
	$\text{FMG-Pan}_{\text{FusionMamba}}^{\text{F}}$	0.9439 $\pm$ 0.0252	0.0330 $\pm$ 0.0182	0.0239 $\pm$ 0.0084	3.43 $\pm$ 0.16

Appendix

Table 5. Comparison in terms of HQNR,

D_{\lambda}

D_{s}

, and runtime (s), of FMG-Pan

{}_{\text{FusionMamba}}

, its lighter-weight variant (FMG-Pan

{}^{\text{L}}_{\text{FusionMamba}}

), and the fine-tuned variant (FMG-Pan

{}^{\text{F}}_{\text{FusionMamba}}

) on 20 full resolution test images for both WV3 and WV2 datasets. ( Red : best; Blue : second best)

Dataset	Pretrained model	Baseline				FMG-Pan
Dataset	Pretrained model	SAM $\downarrow$	ERGAS $\downarrow$	SCC $\uparrow$	Q2n $\uparrow$	SAM $\downarrow$	ERGAS $\downarrow$	SCC $\uparrow$	Q2n $\uparrow$
WV3	FusionMamba	2.8245 $\pm$ 0.5534	2.0890 $\pm$ 0.5008	0.9874 $\pm$ 0.0039	0.9212 $\pm$ 0.0834	3.0803 $\pm$ 0.6404	2.2188 $\pm$ 0.5275	0.9849 $\pm$ 0.0048	0.9129 $\pm$ 0.0845
WV3	LAGNet	3.8258 $\pm$ 0.7352	2.8503 $\pm$ 0.7677	0.9738 $\pm$ 0.0091	0.8722 $\pm$ 0.1139	3.5971 $\pm$ 0.7639	2.7380 $\pm$ 0.6882	0.9757 $\pm$ 0.0084	0.8971 $\pm$ 0.0872
WV2	FusionMamba	5.5330 $\pm$ 0.5352	4.2925 $\pm$ 0.4078	0.9272 $\pm$ 0.0078	0.8408 $\pm$ 0.0888	5.3988 $\pm$ 0.5930	4.1908 $\pm$ 0.4356	0.9288 $\pm$ 0.0073	0.8438 $\pm$ 0.0882
WV2	LAGNet	8.6509 $\pm$ 1.0960	7.3129 $\pm$ 0.8368	0.7987 $\pm$ 0.0452	0.6693 $\pm$ 0.0941	5.9802 $\pm$ 0.7432	5.0153 $\pm$ 0.4367	0.8846 $\pm$ 0.0175	0.8046 $\pm$ 0.0801
GF2	FusionMamba	0.7076 $\pm$ 0.1329	0.6206 $\pm$ 0.1052	0.9923 $\pm$ 0.0014	0.9830 $\pm$ 0.0076	0.9437 $\pm$ 0.1791	0.7955 $\pm$ 0.1371	0.9868 $\pm$ 0.0023	0.9696 $\pm$ 0.0096

Appendix

Table 6. Performance comparison between the baseline pretrained models and the proposed FMG-Pan framework on simulated data. The reported values represent the average performance over 20 test images( Red : The improved results compared to the baseline.)

Figure 13. Comparison of efficiency and reconstruction quality on WV3. (a) Runtime, (b) GPU memory usage (c) HQNR. FMG achieves substantially lower runtime and memory consumption compared to Full and LoRA, while maintaining competitive reconstruction quality.

Explore similar work

G-ZAP: A Generalizable Zero-Shot Framework for Arbitrary-Scale Pansharpening
Mar 15, 2026Zhiqi Yang, Shan Yin, Jingze Liang +1Zero-Shot Learning Remote Sensing
University of Electronic Science and Technology of China
GSPan: A Continuous Gaussian Primitive Representation for Arbitrary-Scale Pansharpening
Jun 16, 2026Fangyi Li, Xiaoyuan Yang, Yixiao Li +3Remote Sensing Image Super-Resolution Remote Sensing
Beihang University, School of Mathematical Sciences, Beijing, 102206, China · Tsinghua University, Department of Automation, Beijing, 100084, China · National Research Council - Institute of Methodologies for Environmental Analysis, CNR-IMAA, Tito, 85050, Italy
Universal Pansharpening Model
Mar 4, 2026Hebaixu Wang, Jing Zhang, Haonan Guo +4Multi-Source Image Fusion Remote Sensing
School of Electronic Information, Wuhan University, Hubei, 430072, Wuhan, China. · Zhongguancun Academy, Beijing, 100094,China. · School of Computer Science, Wuhan University, Hubei, 430072, Wuhan, China. +2

Mar 15, 2026J/K move · Enter open · S save

G-ZAP: A Generalizable Zero-Shot Framework for Arbitrary-Scale Pansharpening

Zhiqi Yang, Shan Yin, Jingze Liang, Liang-Jian Deng

University of Electronic Science and Technology of China

Zero-Shot Learning Remote Sensing

Open paper AskPDF arXiv

Pansharpening aims to fuse a high-resolution panchromatic (PAN) image and a low-resolution multispectral (LRMS) image to produce a high-resolution multispectral (HRMS) image. Recent deep models have achieved strong performance, yet they typically rely on large-scale pretraining and often generalize poorly to unseen real-world image pairs. Prior zero-shot approaches improve real-scene generalization but require per-image optimization, hindering weight reuse, and the above methods are usually limited to a fixed scale. To address this issue, we propose G-ZAP, a generalizable zero-shot framework for arbitrary-scale pansharpening, designed to handle cross-resolution, cross-scene, and cross-sensor generalization. G-ZAP adopts a feature-based implicit neural representation (INR) fusion network as the backbone and introduces a multi-scale, semi-supervised training scheme to enable robust generalization. Extensive experiments on multiple real-world datasets show that G-ZAP achieves state-of-the-art results under PAN-scale fusion in both visual quality and quantitative metrics. Notably, G-ZAP supports weight reuse across image pairs while maintaining competitiveness with per-pair retraining, demonstrating strong potential for efficient real-world deployment.

Abstract

Explore similar work

G-ZAP: A Generalizable Zero-Shot Framework for Arbitrary-Scale Pansharpening

GSPan: A Continuous Gaussian Primitive Representation for Arbitrary-Scale Pansharpening

Universal Pansharpening Model