Cross-modal super-resolution (SR) on real-world misaligned data is challenging, as only unlabeled low-resolution (LR) source and high-resolution (HR) guide images with complex spatial misalignment are available. Previous methods either rely on simulated training data or adopt suboptimal alignment strategies that overlook cross-modal dependencies, limiting their practical performance. To address these issues, we propose RobSelf, a self-supervised model that jointly optimizes a misalignment-aware feature translator and a content-aware reference filter online. The translator resolves unsupervised cross-modal and cross-resolution alignment via weakly-supervised, misalignment-aware translation, yielding an aligned guide feature. Guided by this feature, the filter performs reference-based discriminative self-enhancement on the source, enabling SR prediction with high resolution and high fidelity. Experiments on synthesized data and collected real-world data demonstrate that RobSelf achieves state-of-the-art performance, outperforming existing self-supervised and supervised methods. Moreover, it achieves superior efficiency, being up to 15.3× faster than prior self-supervised methods.
Figures & tables
Figure 1: Real-world misaligned RGB-guided depth SR ( ×4 ). Our model achieves state-of-the-art performance, requiring no training data, ground-truth supervision, or pre-alignment. (a) LR source; (b) HR guide; (c) pre-aligned guide by MINIMA [ 36 ] ; (d) SSGNet [ 40 ] + pre-alignment; (e) SGNet [ 49 ] + pre-alignment; (f) RobSelf-Re (Ours).
Figure 2: RobSelf is solely supervised by the LR source. Within this framework, the translator is optimized under weak supervision to map Fguide into IpredTrans that mimics the source, while yielding FguideAligned . Guided by FguideAligned , the filter performs reference-based discriminative self-enhancement on Fsource to obtain FsourceEnhanced , from which IpredSR is generated. RobSelf has t wo variants depending on the the translator design.
Figure 3: Misalignment-aware feature translator. Each encoder layer downsamples by ×2 ; each decoder layer upsamples by ×2 .
Figure 4: Content-aware reference filter.
Figure 5: Examples from our collected real-world misaligned data. LR sources are overlaid on HR guides for better visualization of misalignments.
P2P [ 30 ]
CMSR [ 38 ]
DCTNet [ 69 ]
MMSR [ 12 ]
SSGNet [ 40 ]
SGNet [ 49 ]
C2PD [ 19 ]
DORNet [ 47 ]
RobSelf-Re
RobSelf-De
Self-Supervised
✓
✓
✗
✓
✓
✗
✗
✗
✓
✓
Pre-Alignment
✓
✗
✓
✓
✓
✓
✓
✓
✗
✗
Params (M)
0.2
1.18
0.48
0.25
0.31
36.39
65.06
3.05
0.76/0.87 ∗
0.81/0.92 ∗
×4 (RMSE ↓ )
2.81/2.33
1.91
2.19/2.16
2.22/1.88
2.20/1.92
2.44/2.33
2.12/2.06
2.57/2.46
1 .52
1.43
×8 (RMSE ↓ )
3.33/2.91
3.03
3.34/3.33
2.99/2.79
2.99/2.83
3.48/3.34
3.27/3.15
3.58/3.54
2 .57
2.49
Table 1: Synthesized misaligned RGB-guided depth SR. For methods lacking alignment strategies, results without and with pre-alignment are reported (wo/w). ∗ Params vary with SR factors ( ×4 / ×8 ) due to level i . The best and s econd best are highlighted.
Figure 6: Synthesized misaligned RGB-guided depth SR ( ×8 ). Image patches are shown due to space limit. *Pre-alignment applied.
P2P [ 30 ]
CMSR [ 38 ]
DCTNet [ 69 ]
MMSR [ 12 ]
SSGNet [ 40 ]
SGNet [ 49 ]
DCNAS [ 72 ]
DORNet [ 47 ]
RobSelf-Re
RobSelf-De
Self-Supervised
✓
✓
✗
✓
✓
✗
✗
✗
✓
✓
Pre-Alignment
✓
✗
✓
✓
✓
✓
✓
✓
✗
✗
Params (M)
0.2
1.18
0.48
0.25
0.31
9.22
0.60
3.05
0.65/0.76 ∗
0.70/0.81 ∗
×2 (RMSE ↓ )
3.75/3.51
2.53
3.27/3.14
2.93/2.76
2.93/2.74
3.18/2.98
3.49/3.27
3.24/3.04
2 .23
2.18
×4 (DSS ↑ )
1.7/2.2
6.7
4.1/4.5
5.0/5.6
5.0/5.5
4.7/5.1
3.6/4.1
4.3/4.9
8 .4
8.6
Table 2: Real-World misaligned RGB-guided depth SR. For methods lacking alignment strategies, results without and with pre-alignment are reported (wo/w). ∗ Params vary with SR factors ( ×2 / ×4 ) due to level i . The best and s econd best are highlighted.
Figure 7: Real-World misaligned RGB-guided depth SR (upper: ×2 ; lower: ×4 ). Image patches are shown due to space limit. *Pre-alignment applied.
P2P [ 30 ]
CMSR [ 38 ]
MMSR [ 12 ]
SSGNet [ 40 ]
Fusion-ArF [ 35 ]
Fusion-CAP [ 26 ]
PAN [ 8 ]
HSI SR [ 63 ]
RobSelf-Re
RobSelf-De
Self-Supervised
✓
✓
✓
✓
✗
✗
✗
✗
✓
✓
Pre-Alignment
✓
✗
✓
✓
✗
✗
✗
✗
✗
✗
Params (M)
0.2
1.18
0.25
0.31
47.39
0.38
7.17
10.97
0.91/1.02 ∗
0.96/1.07 ∗
×2 (RMSE ↓ )
18.72/18.91
3.42
8.94/8.91
8.96/9.04
6.58
4.40
5.19
4.52
3.09
3 .12
×4 (NIQE ↓ )
6.32/ 5.32
8.05
10.95/10.50
10.30/10.23
9.54
8.95
9.24
9.01
7.63
7 .60
Table 3: Real-World misaligned RGB-guided NIR SR. For methods lacking alignment strategies, results without and with pre-alignment are reported (wo/w). ∗ Params vary with SR factors ( ×2 / ×4 ) due to level i . The best and s econd best are highlighted.
Figure 8: Real-world misaligned RGB-guided NIR SR ( ×2 ). Image patches are shown due to space limit. *Pre-alignment applied.
Figure 9: Effectiveness of our translator and filter.
Translator
✗
✓
✗
✓
Filter
✗
✗
✓
✓
RMSE ↓
2.55
2.37
2.31
2.18
Table 4: Effect of our translator and filter. For first two settings, {Fsource,Fguide} or {Fsource,FguideAligned} are fused using a 1×1 conv. No feature fusion in last two settings.
Figure 10: Guide alignment performance: our translator vs. modules in [ 52 ] . Visualizations for real-world misaligned RGB-guided depth (left) and NIR (right) SR.
Figure 11: Our translator can “synthesize” missing guide structures. See the right part of the square pot. Visualizations by RobSelf-De for real-world NIR SR ( ×2 ).
Figure 12: Performance on well-aligned data: synthesized RGB-guided depth SR ( ×4 ).
Super-resolution (SR) is a severely ill-posed problem with inherent ambiguity, as widely recognized in both empirical and theoretical studies. Although recent semantic-guided and multi-modal SR methods exploit large models or external priors to enhance semantic alignment, the fusion of heterogeneous modalities remains insufficiently understood in practice and theory. In this work, we provide the first theoretical modeling of multi-modal SR, revealing that prior methods are bottlenecked by sub-optimal modality utilization. Our analysis shows that the generalization risk bound can be improved by strengthening the alignment between modality weights and their effective contributions, while reducing representation complexity. This theoretical insight inspires us to propose the novel Multi-Modal Mixture-of-Experts Super-Resolution framework (M3ESR) that employs generalization-oriented dynamic modality fusion for accurate risk control and modality contribution optimization. In detail, we propose a novel spatially dynamic modality weighting module and a temporally adaptive modality temperature scheduling mechanism, enabling flexible and adaptive spatial-temporal modality weighting for effective risk control. Extensive experiments demonstrate that our M3ESR significantly boosts generalization and semantic consistency performances, which confirms our superiority.
Jinyi Luo, Minghao Liu, Yifan Li +2
Wangxuan Institute of Computer Technology, Peking University
Single image super-resolution aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. Training SR models typically requires paired HR-LR data, which is difficult to obtain in reality. As a result, most methods synthesize LR images by artificially degrading HR images with handcrafted kernels or camera ISP adjustments. However, these synthetic degradations fail to capture the complexity of real LR images, leading to poor generalization in practice. To address this, we observe that even within a single high-quality image, regions at different depths exhibit varying resolutions, where distant regions act as LR patches and closer ones as HR patches. This allows the extraction of real, degradation-induced LR patches from real images. Since these LR patches lack paired HR counterparts, we propose LA-SR (Language Assistant for SR), a novel framework for unpaired SR. The key idea of LA-SR is to redefine unpaired SR in the language space, using vision-language models to bridge the LR-HR gap. LA-SR projects images into a semantically rich space representing both content and quality, and applies two language-guided losses: linguistic content loss to preserve semantic fidelity, and linguistic quality loss to enhance perceptual realism. With this alignment, LA-SR effectively super-resolves real LR inputs, producing realistic outputs that overcome the limitations of synthetic-data-trained methods.
Joonkyu Park, Kyoung Mu Lee
Dept. of ECE&ASRI, 2IPAI, Seoul National University
Severe downsampling makes single image super-resolution (SISR) an ill-posed problem, in which the language-guided multi-modal methods are especially vulnerable to erroneous priors and require costly textual annotations. To address these issues, we propose Simon-SR, a multi-modal SISR framework leveraging learnable prompts for efficient semantic mining and robust text-image fusion. Simon-SR treats textual semantics as learnable latent variables. Specifically, the Contrastive Prompt Learning (CPL) mines instance-level semantics from unannotated images with frozen CLIP encoders, and Prompt-Guided Spatially Adaptive Refinement (PSAR) injects them through attention-gated multi-modal fusion. On CUB and COCO2017 at ×4 and ×16, Simon-SR gains up to 0.50 dB PSNR and 0.0133 SSIM over current SOTA. The project is available at https://github.com/CHT05017/ICAIS26-SimonSR
Haotong Cheng, Yuxuan Li, Zijie Cui
College of Electronic Science and Engineering, Jilin University, Changchun China