Guided depth super-resolution (GDSR) is a multi-modal approach for depth map super-resolution that relies on a low-resolution depth map and a high-resolution RGB image to restore finer structural details. However, the misleading color and texture cues indicating depth discontinuities in RGB images often lead to artifacts and blurred depth boundaries in the generated depth map. Recent methods counter this by drawing priors from large pretrained models, but these priors enter the network as decoded predictions such as relative depth, surface normal, or segmentation maps, coupling restoration quality to the accuracy of the decoded prior and requiring auxiliary objectives. We propose a solution that introduces global contextual semantic priors, generated from pretrained vision transformer token embeddings, injecting them directly into the depth branch. Our Guided Token Attention (GTA) module lets multi-level depth encodings act as queries of cross-attention over semantic tokens drawn from progressively deeper layers of the pretrained visual transformer (ViT), scaled by a zero-initialized gate that admits semantic evidence only to the extent that it reduces reconstruction error. The resulting token-to-depth correspondence is aligned implicitly, through learned attention under a single reconstruction loss, rather than through an explicit alignment or distribution-matching objective. Building on this, we present Neural Attention with Implicit Multi-token Alignment (NAIMA), which, to the best of our knowledge, is the first GDSR framework guided by undecoded semantic tokens. NAIMA remains competitive in-distribution while achieving the strongest cross-dataset generalization.
Figures & tables
Figure 1: Blurred depth discontinuities caused by RGB noise when performing super-resolution without semantic guidance. In contrast, our semantics-aware approach leverages global contextual information to better preserve structural fidelity.
Figure 2: Edge maps overlaid on the RGB image, the ground-truth depth map, and depth maps predicted by different super-resolution techniques. The semantically guided prediction preserves sharper boundaries and avoids the boundary-bleeding problem, while leveraging object semantics to recover true geometric boundaries, even where the low-resolution input is missing information.
Figure 3: Overview of NAIMA. a. Full architecture: a frozen DINOv2-ViT-S/14 provides multi-level semantic tokens, injected into four Guided Token Attention (GTA) blocks that progressively refine depth features alongside a recurrent RGB stream. b. A single GTA block: depth features query L2-normalized projections of the semantic tokens via cross-attention, scaled by a zero-initialized gate α , added back through the identity path, and fused with spatial RGB features.
Figure 4: Visualization of feature maps extracted from different layers of the NAIMA architecture, for the model trained for an 8 × scaling factor. The top row shows RGB features encoded at each level, while the bottom row shows the corresponding refined feature maps obtained by fusing RGB features with semantically guided depth representations. Feature levels increase from left to right. The semantically corrected features have less noise compared to the features coming directly from RGB spatial encoding.
Methods
In-Distribution
Out-of-Distribution
NYU_v2
RGBDD
Middlebury
LU
4 ×
8 ×
16 ×
4 ×
8 ×
16 ×
4 ×
8 ×
16 ×
4 ×
8 ×
16 ×
DKN 2021 [ 9 ]
1.62
3.26
6.51
1.30
1.96
3.42
1.23
2.12
4.24
0.96
2.16
5.11
FDKN 2021 [ 9 ]
1.86
3.58
6.96
1.18
1.91
3.41
1.08
2.17
4.50
0.82
2.10
5.05
FDSR 2021 [ 7 ]
1.61
3.18
5.84
1.18
1.74
3.05
1.13
2.08
4.39
1.29
2.19
5.00
JIIF 2021 [ 20 ]
1.37
2.76
5.27
1.17
1.79
2.87
1.09
1.82
3.31
0.85
1.73
4.16
Table 1: Results of comparison of existing techniques with NAIMA, across 4 largely used evaluation datasets, and 3 different scaling factors. All baseline results are reported from prior works, where models are trained on the NYU_v2 dataset. Total examples for each dataset are as follows: NYU_v2: 449, RGBDD: 405, Middlebury: 30, LU: 6. Best results in red , second-best in blue . NYU_v2 is the in-distribution setting; the remaining datasets are out-of-distribution dataset evaluations. NAIMA remains competitive in-distribution and obtains the most best or second-best results in the cross-dataset setting, particularly at larger scaling factors.
Methods
x4
x8
x16
DCTNet 2022
2.6
4.33
6.34
SGNet 2024
2.11
3.27
4.73
NAIMA (Ours)
2.09
3.18
4.72
Table 2: Comparison on the TOFDSR dataset (2024), which contains 560 pairs for validation. For fair evaluation, only methods with publicly available pretrained weights are included. NAIMA outperforms the existing architectures for all three scaling factors.
Figure 5: Comparison of depth maps at 8 × scaling factor across multiple evaluation datasets. Bicubic denotes the bicubic-upsampled low-resolution input, while GT represents the ground-truth depth map. The bounding box denotes the corresponding area in the RGB image. It can be observed that NAIMA not only improves depth boundary reconstruction but also adheres to structural details while reconstructing the depth maps.
Figure 6: Comparison of error maps for a selected patch from the Middlebury dataset at 8 × upscaling. The bounding box indicates the corresponding region in the RGB image.
Methods
Middlebury
LU
NYU_v2
RGBDD
TOFDSR
NAIMA
1.62
1.44
2.39
1.61
3.18
NAIMA-L1
1.67
1.45
2.42
1.63
3.22
NAIMA -
1.70
1.50
2.49
1.68
3.26
NAIMA +
1.73
1.50
2.51
1.72
3.28
Table 3: Ablation of NAIMA components, trained on NYU_v2 at 8× and evaluated across all datasets. NAIMA + replaces cross-attention with element-wise addition, NAIMA - removes the semantic branch entirely, and NAIMA-L1 removes the gradient term from the loss. Every variant degrades relative to the full model, confirming the importance of semantic priors, the attention-based injection, and the gradient loss.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Loss plots for NAIMA models at scaling factors of 4x, 8x, and 16x. All three models converge well within the 200 epochs.
Scale
Params (M)
Train. (M)
Non-Train. (M)
FLOPs (T)
4 ×
61.459
40.968
21.876
2.722
8 ×
80.260
59.769
21.924
5.009
16 ×
140.422
119.931
21.924
11.374
Appendix
Table 4: Model complexity of NAIMA across different scaling factors. Parameters are reported in millions (M), and FLOPs in tera floating point operations (T).
Figure 8: Comparison of depth maps at 16 × scaling factor across multiple evaluation datasets. Bicubic denotes the bicubic-upsampled low-resolution input, while GT represents the ground-truth depth map. The bounding box denotes the corresponding area in the RGB image. For the 16 × scale, where the low-resolution depth input contains severely limited structural information, and most cues must be inferred from the RGB guidance, NAIMA successfully reconstructs sharp depth discontinuities and recovers fine structural details. Notably, it accurately captures the chair structure in the first example, preserves the object’s edge boundaries in the second, and finely reconstructs the fins on the toy in the third.
Figure 9: Comparison of depth maps at 4 × scaling factor across multiple evaluation datasets. Bicubic denotes the bicubic-upsampled low-resolution input, while GT represents the ground-truth depth map. The bounding box denotes the corresponding area in the RGB image. NAIMA provides more accurate depth estimation for fine structures, such as the thin plant vine in the first example. In the second example, it better preserves depth discontinuity in the window’s structure compared to competing methods. In the third example, NAIMA reconstructs sharper and more precise depth boundaries for the flower pattern, closely matching the ground truth.
Guided depth super-resolution (GDSR) typically extracts RGB guidance features through convolutional hierarchies, inheriting their fine-to-coarse bias. Thus, low-level spatial cues surface in early layers, leaving the deeper layers to suppress those that do not correspond to true depth boundaries, which risks artifacts and blurred edges. The same fine-to-coarse bias persists in semantics-based methods that consume low-level tokens early and global tokens late. We present WAVE, which introduces a multi-level discrete wavelet transform (ML-DWT) as an explicit and interpretable feature-control mechanism, enabling a coarse-to-fine reconstruction by consuming sub-bands and semantic tokens in reverse of their generation order. WAVE further exploits these sub-bands to treat high- and low-frequency content separately, isolating the misleading RGB color and texture cues before any feature is learned so that their admission can be controlled explicitly, offering an intuitive alternative to the suppression learned implicitly by an opaque network. WAVE separates structure and detail reconstruction into dedicated modules that: i) model interactions within and across wavelet sub-bands, depth features, and semantic priors, ii) apply semantic gating to the high-frequency bands, and iii) fuse modalities through an invertible coupling mechanism that prevents collapse onto a single modality. Extensive experiments across multiple benchmarks demonstrate that WAVE matches or outperforms existing methods, with the largest gains at high upsampling factors, where low-resolution depth contains the least structure.
Tayyab Nasir, Daochang Liu, Ajmal Mian
The University of Western Australia, Perth, Western Australia
Guided depth super-resolution (GDSR) reconstructs HR depth maps from LR inputs with HR RGB guidance. Existing methods either model each modality independently or rely on computationally expensive attention mechanisms with quadratic complexity, hindering the establishment of efficient and semantically interactive joint representations. In this paper, we observe that feature maps from different modalities exhibit semantic-level correlations during feature extraction. This motivates us to develop a more flexible approach enabling dense, semantically-aware deep interactions between modalities. To this end, we propose a novel GDSR framework centered around the Interactive State Space Model. Specifically, we design a cross-modal local scanning mechanism that enables fine-grained semantic interactions between RGB and depth features. Leveraging the Mamba architecture, our framework achieves global modeling with linear complexity. Furthermore, a cross-modal matching transform module is introduced to enhance interactive modeling quality by utilizing representative features from both modalities. Extensive experiments demonstrate competitive performance against state-of-the-art methods.
Chen Wu, Ling Wang, Zhuoran Zheng +4
National University of Defense Technology · The Hong Kong University of Science and Technology (Guangzhou) · Sun Yat-sen University +2
Despite recent advances, single-image super-resolution (SR) remains challenging, especially in real-world scenarios with complex degradations. Diffusion-based SR methods, particularly those built on Stable Diffusion, leverage strong generative priors but commonly rely on text conditioning derived from semantic captioning. Such textual descriptions provide only high-level semantics and lack the spatially aligned visual information required for faithful restoration, leading to a representation gap between abstract semantics and spatially aligned visual details. To address this limitation, we propose GramSR, a one-step diffusion-based SR framework that replaces text conditioning with dense visual features extracted from the low-resolution input using a pre-trained DINOv3 encoder. GramSR adopts a three-stage LoRA architecture, where pixel-level, semantic-level, and texture-level LoRA modules are trained sequentially. The pixel-level module focuses on degradation removal using ℓ2 loss, the semantic-level module enhances perceptual details via LPIPS and CSD losses, and the texture-level module enforces feature correlation consistency through a Gram matrix loss computed from DINOv3 features. At inference, independent guidance scales enable flexible control over degradation removal, semantic enhancement, and texture preservation. Extensive experiments on standard SR benchmarks demonstrate that GramSR consistently outperforms existing one-step diffusion-based methods, achieving superior structural fidelity and texture realism. The code for this work is available at: https://github.com/aimagelab/GramSR.