Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observations

Xiaoyu Dong; Jiahuan Li; Ziteng Cui; Naoto Yokoya

cs.CVFeb 21, 2026

Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observations

Authors: Xiaoyu Dong, Jiahuan Li, Ziteng Cui, Naoto Yokoya

Organizations: The University of Tokyo, Japan · RIKEN AIP, Japan

Abstract

Cross-modal super-resolution (SR) on real-world misaligned data is challenging, as only unlabeled low-resolution (LR) source and high-resolution (HR) guide images with complex spatial misalignment are available. Previous methods either rely on simulated training data or adopt suboptimal alignment strategies that overlook cross-modal dependencies, limiting their practical performance. To address these issues, we propose RobSelf, a self-supervised model that jointly optimizes a misalignment-aware feature translator and a content-aware reference filter online. The translator resolves unsupervised cross-modal and cross-resolution alignment via weakly-supervised, misalignment-aware translation, yielding an aligned guide feature. Guided by this feature, the filter performs reference-based discriminative self-enhancement on the source, enabling SR prediction with high resolution and high fidelity. Experiments on synthesized data and collected real-world data demonstrate that RobSelf achieves state-of-the-art performance, outperforming existing self-supervised and supervised methods. Moreover, it achieves superior efficiency, being up to 15.3 $\times$ faster than prior self-supervised methods.

Figures & tables

Figure 1: Real-world misaligned RGB-guided depth SR ( $\times 4$ ). Our model achieves state-of-the-art performance, requiring no training data, ground-truth supervision, or pre-alignment. (a) LR source; (b) HR guide; (c) pre-aligned guide by MINIMA [ 36 ] ; (d) SSGNet [ 40 ] + pre-alignment; (e) SGNet [ 49 ] + pre-alignment; (f) RobSelf-Re (Ours).

Figure 2: RobSelf is solely supervised by the LR source. Within this framework, the translator is optimized under weak supervision to map $\mathbf{F}_{guide}$ into $\mathbf{I}^{\rm Trans}_{pred}$ that mimics the source, while yielding $\mathbf{F}^{\rm Aligned}_{guide}$ . Guided by $\mathbf{F}^{\rm Aligned}_{guide}$ , the filter performs reference-based discriminative self-enhancement on $\mathbf{F}_{source}$ to obtain $\mathbf{F}^{\rm Enhanced}_{source}$ , from which $\mathbf{I}^{\rm SR}_{pred}$ is generated. RobSelf has t wo variants depending on the the translator design.

Figure 3: Misalignment-aware feature translator. Each encoder layer downsamples by $\times 2$ ; each decoder layer upsamples by $\times 2$ .

Figure 4: Content-aware reference filter.

Panel of Figure 5 — Figure 5: Examples from our collected real-world misaligned data. LR sources are overlaid on HR guides for better visualization of misalignments.

	P2P [ 30 ]	CMSR [ 38 ]	DCTNet [ 69 ]	MMSR [ 12 ]	SSGNet [ 40 ]	SGNet [ 49 ]	C2PD [ 19 ]	DORNet [ 47 ]	RobSelf-Re	RobSelf-De
Self-Supervised	$\checkmark$	$\checkmark$	✗	$\checkmark$	$\checkmark$	✗	✗	✗	$\checkmark$	$\checkmark$
Pre-Alignment	$\checkmark$	✗	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	✗	✗
Params (M)	0.2	1.18	0.48	0.25	0.31	36.39	65.06	3.05	0.76/0.87 ∗	0.81/0.92 ∗
$\times 4$ (RMSE $\downarrow$ )	2.81/2.33	1.91	2.19/2.16	2.22/1.88	2.20/1.92	2.44/2.33	2.12/2.06	2.57/2.46	1 .52	1.43
$\times 8$ (RMSE $\downarrow$ )	3.33/2.91	3.03	3.34/3.33	2.99/2.79	2.99/2.83	3.48/3.34	3.27/3.15	3.58/3.54	2 .57	2.49

Table 1: Synthesized misaligned RGB-guided depth SR. For methods lacking alignment strategies, results without and with pre-alignment are reported (wo/w). ∗ Params vary with SR factors (

\times 4

\times 8

) due to level

i

. The best and s econd best are highlighted.

Panel of Figure 6 — Figure 6: Synthesized misaligned RGB-guided depth SR ( $\times 8$ ). Image patches are shown due to space limit. *Pre-alignment applied.

	P2P [ 30 ]	CMSR [ 38 ]	DCTNet [ 69 ]	MMSR [ 12 ]	SSGNet [ 40 ]	SGNet [ 49 ]	DCNAS [ 72 ]	DORNet [ 47 ]	RobSelf-Re	RobSelf-De
Self-Supervised	$\checkmark$	$\checkmark$	✗	$\checkmark$	$\checkmark$	✗	✗	✗	$\checkmark$	$\checkmark$
Pre-Alignment	$\checkmark$	✗	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	✗	✗
Params (M)	0.2	1.18	0.48	0.25	0.31	9.22	0.60	3.05	0.65/0.76 ∗	0.70/0.81 ∗
$\times 2$ (RMSE $\downarrow$ )	3.75/3.51	2.53	3.27/3.14	2.93/2.76	2.93/2.74	3.18/2.98	3.49/3.27	3.24/3.04	2 .23	2.18
$\times 4$ (DSS $\uparrow$ )	1.7/2.2	6.7	4.1/4.5	5.0/5.6	5.0/5.5	4.7/5.1	3.6/4.1	4.3/4.9	8 .4	8.6

Table 2: Real-World misaligned RGB-guided depth SR. For methods lacking alignment strategies, results without and with pre-alignment are reported (wo/w). ∗ Params vary with SR factors (

\times 2

\times 4

) due to level

i

. The best and s econd best are highlighted.

Panel of Figure 7 — Figure 7: Real-World misaligned RGB-guided depth SR (upper: $\times 2$ ; lower: $\times 4$ ). Image patches are shown due to space limit. *Pre-alignment applied.

	P2P [ 30 ]	CMSR [ 38 ]	MMSR [ 12 ]	SSGNet [ 40 ]	Fusion-ArF [ 35 ]	Fusion-CAP [ 26 ]	PAN [ 8 ]	HSI SR [ 63 ]	RobSelf-Re	RobSelf-De
Self-Supervised	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	✗	✗	✗	✗	$\checkmark$	$\checkmark$
Pre-Alignment	$\checkmark$	✗	$\checkmark$	$\checkmark$	✗	✗	✗	✗	✗	✗
Params (M)	0.2	1.18	0.25	0.31	47.39	0.38	7.17	10.97	0.91/1.02 ∗	0.96/1.07 ∗
$\times 2$ (RMSE $\downarrow$ )	18.72/18.91	3.42	8.94/8.91	8.96/9.04	6.58	4.40	5.19	4.52	3.09	3 .12
$\times 4$ (NIQE $\downarrow$ )	6.32/ 5.32	8.05	10.95/10.50	10.30/10.23	9.54	8.95	9.24	9.01	7.63	7 .60

Table 3: Real-World misaligned RGB-guided NIR SR. For methods lacking alignment strategies, results without and with pre-alignment are reported (wo/w). ∗ Params vary with SR factors (

\times 2

\times 4

) due to level

i

. The best and s econd best are highlighted.

Panel of Figure 8 — Figure 8: Real-world misaligned RGB-guided NIR SR ( $\times 2$ ). Image patches are shown due to space limit. *Pre-alignment applied.

Panel of Figure 9 — Figure 9: Effectiveness of our translator and filter.

Translator	✗	$\checkmark$	✗	$\checkmark$
Filter	✗	✗	$\checkmark$	$\checkmark$
RMSE $\downarrow$	2.55	2.37	2.31	2.18

Table 4: Effect of our translator and filter. For first two settings,

\{\mathbf{F}_{source},\mathbf{F}_{guide}\}

\{\mathbf{F}_{source},\mathbf{F}^{\rm Aligned}_{guide}\}

are fused using a

1\times 1

conv. No feature fusion in last two settings.

Panel of Figure 10 — Figure 10: Guide alignment performance: our translator vs. modules in [ 52 ] . Visualizations for real-world misaligned RGB-guided depth (left) and NIR (right) SR.

Panel of Figure 11 — Figure 11: Our translator can “synthesize” missing guide structures. See the right part of the square pot. Visualizations by RobSelf-De for real-world NIR SR ( $\times 2$ ).

Panel of Figure 12 — Figure 12: Performance on well-aligned data: synthesized RGB-guided depth SR ( $\times 4$ ).

	P2P* [ 30 ]	MMSR* [ 12 ]	SSGNet* [ 40 ]	RobSelf-De	RobSelf-Re
Task I ( $\times 4$ ). Guide: $512\times 576\times 3$ ; Upsampled source: $512\times 576\times 1$ .
RMSE $\downarrow$	2.33	1.88	1.92	1.43	1 .52
Time	325s	137s	151s	1 00s	71s
Task II ( $\times 2$ ). Guide: $576\times 640\times 3$ ; Upsampled source: $576\times 640\times 1$ .
RMSE $\downarrow$	3.51	2.76	2.74	2.18	2 .23
Time	982s	164s	181s	1 22s	89s

Table 5: Runtime comparison of self-supervised methods. *Pre-alignment applied.

Image Super-Resolution

Image super-resolution reconstructs high-resolution images from lower-resolution observations, balancing detail, fidelity, and computational cost.

See all 248 papers

Adaptive Context Matters: Towards Provable Multi-Modality Guidance for Super-Resolution
May 11, 2026Jinyi Luo, Minghao Liu, Yifan Li +2Mixture of Experts Multimodal Fusion
Wangxuan Institute of Computer Technology, Peking University
Language-Assisted Super-Resolution from Real-World Low-Resolution Patches
Jun 30, 2026Joonkyu Park, Kyoung Mu LeeImage Super-Resolution
Dept. of ECE&ASRI, 2IPAI, Seoul National University
Simon-SR: Spatially Adaptive Modulation and Visual Prompt Adaptation for Text-Reinforced Super-Resolution
Jul 10, 2026Haotong Cheng, Yuxuan Li, Zijie CuiMultimodal Fusion Visual Prompt Learning
College of Electronic Science and Engineering, Jilin University, Changchun China

May 11, 2026J/K move · Enter open · S save

Adaptive Context Matters: Towards Provable Multi-Modality Guidance for Super-Resolution

Jinyi Luo, Minghao Liu, Yifan Li, Zejia Fan+1

Wangxuan Institute of Computer Technology, Peking University

Mixture of Experts Multimodal Fusion Image Super-Resolution Generalization Bounds

Open paper AskPDF arXiv

Super-resolution (SR) is a severely ill-posed problem with inherent ambiguity, as widely recognized in both empirical and theoretical studies. Although recent semantic-guided and multi-modal SR methods exploit large models or external priors to enhance semantic alignment, the fusion of heterogeneous modalities remains insufficiently understood in practice and theory. In this work, we provide the first theoretical modeling of multi-modal SR, revealing that prior methods are bottlenecked by sub-optimal modality utilization. Our analysis shows that the generalization risk bound can be improved by strengthening the alignment between modality weights and their effective contributions, while reducing representation complexity. This theoretical insight inspires us to propose the novel Multi-Modal Mixture-of-Experts Super-Resolution framework (M

^3

ESR) that employs generalization-oriented dynamic modality fusion for accurate risk control and modality contribution optimization. In detail, we propose a novel spatially dynamic modality weighting module and a temporally adaptive modality temperature scheduling mechanism, enabling flexible and adaptive spatial-temporal modality weighting for effective risk control. Extensive experiments demonstrate that our M

^3

ESR significantly boosts generalization and semantic consistency performances, which confirms our superiority.

Abstract

Image Super-Resolution

Explore similar work

Adaptive Context Matters: Towards Provable Multi-Modality Guidance for Super-Resolution

Language-Assisted Super-Resolution from Real-World Low-Resolution Patches

Simon-SR: Spatially Adaptive Modulation and Visual Prompt Adaptation for Text-Reinforced Super-Resolution