FoundDSR: A Generalizable Foundation Model with Guided 2D Gaussian Splatting for Depth Super-Resolution

Zhengxue Wang; Zhiqiang Yan; Yuan Wu; Guangwei Gao; Xiang Li; Jian Yang

cs.CVSep 26, 2026

FoundDSR: A Generalizable Foundation Model with Guided 2D Gaussian Splatting for Depth Super-Resolution

Authors: Zhengxue Wang, Zhiqiang Yan, Yuan Wu, Guangwei Gao, Xiang Li, Jian Yang

Organizations: PCA Lab, Nanjing University of Science and Technology · National University of Singapore · Nankai University · Nanjing University

Abstract

We introduce FoundDSR, a generalizable foundation model for robust depth reconstruction across unseen data distributions using RGB-D pairs. FoundDSR begins with a guided 2D Gaussian Splatting strategy to model depth representations with Gaussian primitives. This strategy employs high-resolution RGB as prompts to optimize the Gaussian parameters, thereby encouraging each Gaussian primitive to anisotropically deform along high-frequency structural directions. The resulting Gaussian-upsampled representations are then mapped to high-resolution depth through an effective depth reconstruction branch. Furthermore, to mitigate training instability and bias toward dominant sources caused by distribution gaps in large-scale heterogeneous data, we introduce heterogeneous federated learning that allocates each data source to an independent client for local optimization and global aggregation. This design effectively endows FoundDSR with stable scalability to diverse and large-scale training data. Extensive zero-shot evaluations on synthetic, real-world, arbitrary-scale, and noisy conditions demonstrate that FoundDSR consistently outperforms existing state-of-the-art approaches, confirming its strong robustness and generalization to unknown scenes.

Figures & tables

Figure 1: (a) t-SNE visualization of feature embeddings from different data sources. (b) Average RMSE across all benchmarks at different scales.

Figure 2: Pipeline of FoundDSR. In federated learning, global data are distributed to independent clients for local optimization, while the server aggregates and redistributes model weights. For each client, LR depth is encoded into initial Gaussian $\mathcal{G}$ , which are modulated by RGB gradients for anisotropic structural deformation. The transformed Gaussian $\mathcal{\hat{G}}$ are then rendered into $\bm{D}_{gs}$ . Finally, $\bm{D}_{gs}$ and RGB $\bm{I}$ are fed into a depth reconstruction to predict HR depth $\bm{D}_{HR}$ .

Figure 3: Visualization of Gaussian primitive modulation with and without RGB guidance.

Methods	Datasets	$\times 2.5$	$\times 3.0$	$\times 4.0$	$\times 5.7$	$\times 6.0$	$\times 7.1$	$\times 8.0$	$\times 9.3$	$\times 11.0$	$\times 12.5$	$\times 13.0$	$\times 14.9$	$\times 16.0$	$\times 19.3$
PromptDA	NYU-v2	17.34	17.06	17.64	17.57	17.61	17.58	18.44	18.01	18.36	18.56	18.66	18.49	19.24	19.00
Marigold-DC		9.12	9.15	9.26	9.48	9.53	9.80	10.01	10.30	10.90	11.38	11.55	12.13	12.53	13.79
InfiniDepth		8.91	8.73	8.70	8.68	8.71	9.02	9.15	9.54	10.15	10.46	10.86	11.20	11.65	13.56
LDCM		4.54	4.57	4.95	6.15	6.50	7.04	7.58	8.44	9.18	9.89	10.28	11.12	11.52	13.12
FDSR		3.90	3.64	4.03	3.85	4.08	4.28	4.99	5.13	5.60	6.30	6.82	9.00	9.98	12.29
DCTNet		3.33	3.41	3.54	4.05	4.17	4.64	5.01	5.65	6.57	7.56	7.93	9.22	9.98	11.97

Table 1: Zero-shot comparisons on

5

synthetic benchmarks under integer and non-integer scales. All foundation models use their large variants. The best and second-best results are highlighted.

Figure 4: Visual comparisons on the $\times 16.0$ NYU‑v2 dataset. The zoomed‑in patches in (d)–(j) highlight the error maps within the white boxes, where darker purple shades indicate higher errors.

RMSE	PromptDA	Marigold-DC	InfiniDepth	LDCM	FDSR	DCTNet	DORNet	C2PD	DuCos	FoundDSR-S	FoundDSR
RGB-D-D	7.06	6.85	6.87	7.65	7.41	7.82	7.49	7.26	7.44	6.93	6.68
TOFDSR	8.79	7.80	8.48	8.22	7.23	7.45	7.34	7.23	7.42	7.05	6.92

Table 2: Zero-shot comparisons on the real-world RGB-D-D and TOFDSR datasets.

Methods	NYU-v2			Lu			Middlebury			RGB-D-D			iBims-1
Methods	0.02	0.03	0.04	0.02	0.03	0.04	0.02	0.03	0.04	0.02	0.03	0.04	0.02	0.03	0.04
PromptDA	17.42	17.34	17.36	22.49	22.42	22.40	15.12	15.52	11.93	5.23	5.35	5.53	38.83	39.06	39.48
Marigold-DC	8.71	8.79	8.97	36.42	36.04	35.94	5.75	5.83	5.65	2.93	2.93	2.96	33.90	33.80	33.89
LDCM	6.00	7.40	9.24	4.64	6.24	7.99	4.33	5.88	7.46	3.00	3.62	4.34	23.76	26.85	30.44
FDSR	5.82	6.56	7.05	3.18	3.66	3.96	3.00	3.49	3.86	2.26	2.56	2.92	18.61	19.71	20.62
DCTNet	6.24	6.94	7.81	3.32	3.94	4.30	3.35	3.96	4.22	2.58	2.95	3.44	20.08	21.83	23.36

Table 3: Zero-shot comparisons on multiple

\times 4.0

benchmarks with noise levels from

0.02

0.04

Figure 5: Zero-shot comparisons of continuous upsampling factors with prior foundation models and DSR methods on NYU-v2.

Table 9

Figure 6: Ablation study on (a) training data scaling and (b) heterogeneous federated learning. RMSE is averaged over five synthetic datasets.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Type	Resolution	Statistic
DIML	Real-World	$1344\times 756$	1.6K
Hypersim	Synthetic	$1024\times 768$	46.1K
DynamicReplica	Synthetic	$1280\times 720$	336.9K
LingBot-Depth	Synthetic	$1280\times 960$	1.0M
Total	-	-	1.4M

Appendix

Table 6: Overview of the training datasets.

Methods	Datasets	$\times 2.5$	$\times 3.0$	$\times 4.0$	$\times 5.7$	$\times 6.0$	$\times 7.1$	$\times 8.0$	$\times 9.3$	$\times 11.0$	$\times 12.5$	$\times 13.0$	$\times 14.9$	$\times 16.0$	$\times 19.3$
PromptDA	NYU-v2	7.58	7.54	7.67	7.74	7.73	7.67	8.04	7.82	8.01	9.16	8.16	8.23	8.50	8.49
Marigold-DC		4.18	4.20	4.27	4.39	4.41	4.56	4.68	4.83	5.14	5.38	5.53	5.79	6.04	6.73
InfiniDepth		3.33	3.37	3.42	3.53	3.59	3.78	3.91	4.60	4.52	4.72	4.94	5.11	5.45	6.53
LDCM		1.57	1.63	1.82	2.32	2.47	2.64	3.01	3.40	3.86	4.25	4.51	4.95	5.24	6.19
FDSR		1.12	1.10	1.19	1.28	1.33	1.48	1.68	1.84	2.10	2.49	2.74	3.70	4.34	5.54
DCTNet		1.26	1.29	1.36	1.58	1.63	1.82	2.01	2.30	2.84	3.36	3.56	4.17	4.62	5.65

Appendix

Table 7: Quantitative comparisons on five synthetic benchmarks using the MAE

\downarrow

(cm) metric.

Figure 7: Visual comparisons on the real-world RGB-D-D dataset. The zoomed‑in patches in (d)–(j) highlight the error maps within the white boxes, where darker purple shades indicate higher errors.

Methods	Datasets	$\times 2.5$	$\times 3.0$	$\times 4.0$	$\times 5.7$	$\times 6.0$	$\times 7.1$	$\times 8.0$	$\times 9.3$	$\times 11.0$	$\times 12.5$	$\times 13.0$	$\times 14.9$	$\times 16.0$	$\times 19.3$
PromptDA	NYU-v2	92.39	92.46	92.20	92.13	92.16	92.26	91.57	91.90	91.59	91.38	91.45	91.23	90.66	90.73
Marigold-DC		95.91	95.87	95.69	95.38	95.33	94.97	94.56	94.21	93.47	92.87	92.58	91.85	91.12	89.73
InfiniDepth		94.37	94.21	94.06	93.80	93.59	93.29	92.79	92.74	91.63	91.15	90.81	90.31	89.30	87.52
LDCM		98.88	98.78	98.39	97.43	97.23	96.55	95.94	95.24	94.23	93.39	93.09	92.09	91.28	89.74
FDSR		99.00	99.10	99.00	99.05	98.98	98.88	98.62	98.46	98.15	97.56	97.13	95.04	93.45	90.99
DCTNet		99.19	99.16	99.10	98.83	98.76	98.51	98.25	97.84	96.84	95.66	95.28	93.81	92.62	90.66

Appendix

Table 8: Quantitative comparisons on five synthetic benchmarks using the

\mathbf{\delta_{1.05}\uparrow}

metric.

Figure 8: Visual comparisons on NYU-v2 across integer scales. (a)–(j) denote RGB, LR, GT, Marigold-DC, LDCM, DCTNet, DORNet, DuCos, FoundDSR-S, and FoundDSR, respectively.

Figure 9: Visual comparisons on RGB-D-D across non-integer scales. (a)–(j) denote RGB, LR, GT, Marigold-DC, LDCM, DCTNet, DORNet, DuCos, FoundDSR-S, and FoundDSR, respectively.

Figure 10: Visual comparisons on $\times 4.0$ RGB-D-D under various noise levels. (a)–(j): RGB, LR, GT, Marigold-DC, LDCM, DCTNet, DORNet, DuCos, FoundDSR-S, and FoundDSR.

Figure 11: Ablation study of (a) different losses and (b) guided 2D GS and depth reconstruction (DR) on

\times 13.0

\times 14.9

, and

\times 16.0

NYU-v2.

Explore similar work

SR3R: Rethinking Super-Resolution 3D Reconstruction With Feed-Forward Gaussian Splatting
Feb 27, 2026Xiang Feng, Xiangbo Wang, Tieshi Zhong +73D Reconstruction 3D Gaussian Splatting
Hangzhou Dianzi University · ShanghaiTech University · Peking University +1
Any to Full: Prompting Depth Anything for Depth Completion in One Stage
Mar 5, 2026Zhiyuan Zhou, Ruofeng Liu, Taichi Liu +4Depth Completion Prompt Tuning
Rutgers University, USA · Michigan State University, USA · JD Logistics, China +1
DegBins: Degradation-Driven Binning for Depth Super-Resolution
May 10, 2026Zhiqiang Yan, Zhengxue Wang, Jian Yang +1Depth Estimation Residual Learning
Department of Computer Science, National University of Singapore · Nanjing University of Science and Technology

Feb 27, 2026J/K move · Enter open · S save

SR3R: Rethinking Super-Resolution 3D Reconstruction With Feed-Forward Gaussian Splatting

Xiang Feng, Xiangbo Wang, Tieshi Zhong, Chengkai Wang+6

Hangzhou Dianzi University · ShanghaiTech University · Peking University+1

3D Reconstruction 3D Gaussian Splatting Feed-Forward 3D Reconstruction Sparse-View 3D Reconstruction

Open paper AskPDF arXiv

3D super-resolution (3DSR) aims to reconstruct high-resolution (HR) 3D scenes from low-resolution (LR) multi-view images. Existing methods rely on dense LR inputs and per-scene optimization, which restricts the high-frequency priors for constructing HR 3D Gaussian Splatting (3DGS) to those inherited from pretrained 2D super-resolution (2DSR) models. This severely limits reconstruction fidelity, cross-scene generalization, and real-time usability. We propose to reformulate 3DSR as a direct feed-forward mapping from sparse LR views to HR 3DGS representations, enabling the model to autonomously learn 3D-specific high-frequency geometry and appearance from large-scale, multi-scene data. This fundamentally changes how 3DSR acquires high-frequency knowledge and enables robust generalization to unseen scenes. Specifically, we introduce SR3R, a feed-forward framework that directly predicts HR 3DGS representations from sparse LR views via the learned mapping network. To further enhance reconstruction fidelity, we introduce Gaussian offset learning and feature refinement, which stabilize reconstruction and sharpen high-frequency details. SR3R is plug-and-play and can be paired with any feed-forward 3DGS reconstruction backbone: the backbone provides an LR 3DGS scaffold, and SR3R upscales it to an HR 3DGS. Extensive experiments across three 3D benchmarks demonstrate that SR3R surpasses state-of-the-art (SOTA) 3DSR methods and achieves strong zero-shot generalization, even outperforming SOTA per-scene optimization methods on unseen scenes.

Abstract

Explore similar work

SR3R: Rethinking Super-Resolution 3D Reconstruction With Feed-Forward Gaussian Splatting

Any to Full: Prompting Depth Anything for Depth Completion in One Stage

DegBins: Degradation-Driven Binning for Depth Super-Resolution