NAIMA: Semantics Aware RGB Guided Depth Super-Resolution

Tayyab Nasir; Daochang Liu; Ajmal Mian

eess.IVApr 6, 2026

NAIMA: Semantics Aware RGB Guided Depth Super-Resolution

Authors: Tayyab Nasir, Daochang Liu, Ajmal Mian

Organizations: The University of Western Australia, Perth, Western Australia

Abstract

Guided depth super-resolution (GDSR) is a multi-modal approach for depth map super-resolution that relies on a low-resolution depth map and a high-resolution RGB image to restore finer structural details. However, the misleading color and texture cues indicating depth discontinuities in RGB images often lead to artifacts and blurred depth boundaries in the generated depth map. Recent methods counter this by drawing priors from large pretrained models, but these priors enter the network as decoded predictions such as relative depth, surface normal, or segmentation maps, coupling restoration quality to the accuracy of the decoded prior and requiring auxiliary objectives. We propose a solution that introduces global contextual semantic priors, generated from pretrained vision transformer token embeddings, injecting them directly into the depth branch. Our Guided Token Attention (GTA) module lets multi-level depth encodings act as queries of cross-attention over semantic tokens drawn from progressively deeper layers of the pretrained visual transformer (ViT), scaled by a zero-initialized gate that admits semantic evidence only to the extent that it reduces reconstruction error. The resulting token-to-depth correspondence is aligned implicitly, through learned attention under a single reconstruction loss, rather than through an explicit alignment or distribution-matching objective. Building on this, we present Neural Attention with Implicit Multi-token Alignment (NAIMA), which, to the best of our knowledge, is the first GDSR framework guided by undecoded semantic tokens. NAIMA remains competitive in-distribution while achieving the strongest cross-dataset generalization.

Figures & tables

Figure 1: Blurred depth discontinuities caused by RGB noise when performing super-resolution without semantic guidance. In contrast, our semantics-aware approach leverages global contextual information to better preserve structural fidelity.

Figure 2: Edge maps overlaid on the RGB image, the ground-truth depth map, and depth maps predicted by different super-resolution techniques. The semantically guided prediction preserves sharper boundaries and avoids the boundary-bleeding problem, while leveraging object semantics to recover true geometric boundaries, even where the low-resolution input is missing information.

Figure 3: Overview of NAIMA. a. Full architecture: a frozen DINOv2-ViT-S/14 provides multi-level semantic tokens, injected into four Guided Token Attention (GTA) blocks that progressively refine depth features alongside a recurrent RGB stream. b. A single GTA block: depth features query L2-normalized projections of the semantic tokens via cross-attention, scaled by a zero-initialized gate $\alpha$ , added back through the identity path, and fused with spatial RGB features.

Figure 4: Visualization of feature maps extracted from different layers of the NAIMA architecture, for the model trained for an 8 $\times$ scaling factor. The top row shows RGB features encoded at each level, while the bottom row shows the corresponding refined feature maps obtained by fusing RGB features with semantically guided depth representations. Feature levels increase from left to right. The semantically corrected features have less noise compared to the features coming directly from RGB spatial encoding.

Methods	In-Distribution			Out-of-Distribution
	NYU_v2			RGBDD			Middlebury			LU
	4 $\times$	8 $\times$	16 $\times$	4 $\times$	8 $\times$	16 $\times$	4 $\times$	8 $\times$	16 $\times$	4 $\times$	8 $\times$	16 $\times$
DKN 2021 [ 9 ]	1.62	3.26	6.51	1.30	1.96	3.42	1.23	2.12	4.24	0.96	2.16	5.11
FDKN 2021 [ 9 ]	1.86	3.58	6.96	1.18	1.91	3.41	1.08	2.17	4.50	0.82	2.10	5.05
FDSR 2021 [ 7 ]	1.61	3.18	5.84	1.18	1.74	3.05	1.13	2.08	4.39	1.29	2.19	5.00
JIIF 2021 [ 20 ]	1.37	2.76	5.27	1.17	1.79	2.87	1.09	1.82	3.31	0.85	1.73	4.16

Table 1: Results of comparison of existing techniques with NAIMA, across 4 largely used evaluation datasets, and 3 different scaling factors. All baseline results are reported from prior works, where models are trained on the NYU_v2 dataset. Total examples for each dataset are as follows: NYU_v2: 449, RGBDD: 405, Middlebury: 30, LU: 6. Best results in red , second-best in blue . NYU_v2 is the in-distribution setting; the remaining datasets are out-of-distribution dataset evaluations. NAIMA remains competitive in-distribution and obtains the most best or second-best results in the cross-dataset setting, particularly at larger scaling factors.

Methods	x4	x8	x16
DCTNet 2022	2.6	4.33	6.34
SGNet 2024	2.11	3.27	4.73
NAIMA (Ours)	2.09	3.18	4.72

Table 2: Comparison on the TOFDSR dataset (2024), which contains 560 pairs for validation. For fair evaluation, only methods with publicly available pretrained weights are included. NAIMA outperforms the existing architectures for all three scaling factors.

Figure 5: Comparison of depth maps at 8 $\times$ scaling factor across multiple evaluation datasets. Bicubic denotes the bicubic-upsampled low-resolution input, while GT represents the ground-truth depth map. The bounding box denotes the corresponding area in the RGB image. It can be observed that NAIMA not only improves depth boundary reconstruction but also adheres to structural details while reconstructing the depth maps.

Figure 6: Comparison of error maps for a selected patch from the Middlebury dataset at 8 $\times$ upscaling. The bounding box indicates the corresponding region in the RGB image.

Methods	Middlebury	LU	NYU_v2	RGBDD	TOFDSR
NAIMA	1.62	1.44	2.39	1.61	3.18
NAIMA-L1	1.67	1.45	2.42	1.63	3.22
NAIMA -	1.70	1.50	2.49	1.68	3.26
NAIMA +	1.73	1.50	2.51	1.72	3.28

Table 3: Ablation of NAIMA components, trained on NYU_v2 at

8\times

and evaluated across all datasets. NAIMA + replaces cross-attention with element-wise addition, NAIMA - removes the semantic branch entirely, and NAIMA-L1 removes the gradient term from the loss. Every variant degrades relative to the full model, confirming the importance of semantic priors, the attention-based injection, and the gradient loss.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Loss plots for NAIMA models at scaling factors of 4x, 8x, and 16x. All three models converge well within the 200 epochs.

Scale	Params (M)	Train. (M)	Non-Train. (M)	FLOPs (T)
4 $\times$	61.459	40.968	21.876	2.722
8 $\times$	80.260	59.769	21.924	5.009
16 $\times$	140.422	119.931	21.924	11.374

Appendix

Table 4: Model complexity of NAIMA across different scaling factors. Parameters are reported in millions (M), and FLOPs in tera floating point operations (T).

Figure 8: Comparison of depth maps at 16 $\times$ scaling factor across multiple evaluation datasets. Bicubic denotes the bicubic-upsampled low-resolution input, while GT represents the ground-truth depth map. The bounding box denotes the corresponding area in the RGB image. For the 16 $\times$ scale, where the low-resolution depth input contains severely limited structural information, and most cues must be inferred from the RGB guidance, NAIMA successfully reconstructs sharp depth discontinuities and recovers fine structural details. Notably, it accurately captures the chair structure in the first example, preserves the object’s edge boundaries in the second, and finely reconstructs the fins on the toy in the third.

Figure 9: Comparison of depth maps at 4 $\times$ scaling factor across multiple evaluation datasets. Bicubic denotes the bicubic-upsampled low-resolution input, while GT represents the ground-truth depth map. The bounding box denotes the corresponding area in the RGB image. NAIMA provides more accurate depth estimation for fine structures, such as the thin plant vine in the first example. In the second example, it better preserves depth discontinuity in the window’s structure compared to competing methods. In the third example, NAIMA reconstructs sharper and more precise depth boundaries for the flower pattern, closely matching the ground truth.

WAVE: Wavelet-Controlled Coarse-to-Fine Guidance for Depth Super-Resolution
Aug 26, 2026Tayyab Nasir, Daochang Liu, Ajmal MianMulti-Scale Feature Fusion Image Super-Resolution
The University of Western Australia, Perth, Western Australia
Interactive State Space Model with Cross-Modal Local Scanning for Depth Super-Resolution
May 12, 2026Chen Wu, Ling Wang, Zhuoran Zheng +4Cross-Modal Representation Learning Visual SSMs
National University of Defense Technology · The Hong Kong University of Science and Technology (Guangzhou) · Sun Yat-sen University +2
GramSR: Visual Feature Conditioning for Diffusion-Based Super-Resolution
Apr 28, 2026Fabio D'Oronzio, Federico Putamorsi, Leonardo Zini +2Diffusion Models for Image Restoration Diffusion Model Fine-Tuning
University of Modena and Reggio Emilia, Italy

Aug 26, 2026J/K move · Enter open · S save

WAVE: Wavelet-Controlled Coarse-to-Fine Guidance for Depth Super-Resolution

Tayyab Nasir, Daochang Liu, Ajmal Mian

The University of Western Australia, Perth, Western Australia

Multi-Scale Feature Fusion Image Super-Resolution

Open paper AskPDF arXiv

Guided depth super-resolution (GDSR) typically extracts RGB guidance features through convolutional hierarchies, inheriting their fine-to-coarse bias. Thus, low-level spatial cues surface in early layers, leaving the deeper layers to suppress those that do not correspond to true depth boundaries, which risks artifacts and blurred edges. The same fine-to-coarse bias persists in semantics-based methods that consume low-level tokens early and global tokens late. We present WAVE, which introduces a multi-level discrete wavelet transform (ML-DWT) as an explicit and interpretable feature-control mechanism, enabling a coarse-to-fine reconstruction by consuming sub-bands and semantic tokens in reverse of their generation order. WAVE further exploits these sub-bands to treat high- and low-frequency content separately, isolating the misleading RGB color and texture cues before any feature is learned so that their admission can be controlled explicitly, offering an intuitive alternative to the suppression learned implicitly by an opaque network. WAVE separates structure and detail reconstruction into dedicated modules that: i) model interactions within and across wavelet sub-bands, depth features, and semantic priors, ii) apply semantic gating to the high-frequency bands, and iii) fuse modalities through an invertible coupling mechanism that prevents collapse onto a single modality. Extensive experiments across multiple benchmarks demonstrate that WAVE matches or outperforms existing methods, with the largest gains at high upsampling factors, where low-resolution depth contains the least structure.

Abstract

Explore similar work

WAVE: Wavelet-Controlled Coarse-to-Fine Guidance for Depth Super-Resolution

Interactive State Space Model with Cross-Modal Local Scanning for Depth Super-Resolution

GramSR: Visual Feature Conditioning for Diffusion-Based Super-Resolution