We introduce FoundDSR, a generalizable foundation model for robust depth reconstruction across unseen data distributions using RGB-D pairs. FoundDSR begins with a guided 2D Gaussian Splatting strategy to model depth representations with Gaussian primitives. This strategy employs high-resolution RGB as prompts to optimize the Gaussian parameters, thereby encouraging each Gaussian primitive to anisotropically deform along high-frequency structural directions. The resulting Gaussian-upsampled representations are then mapped to high-resolution depth through an effective depth reconstruction branch. Furthermore, to mitigate training instability and bias toward dominant sources caused by distribution gaps in large-scale heterogeneous data, we introduce heterogeneous federated learning that allocates each data source to an independent client for local optimization and global aggregation. This design effectively endows FoundDSR with stable scalability to diverse and large-scale training data. Extensive zero-shot evaluations on synthetic, real-world, arbitrary-scale, and noisy conditions demonstrate that FoundDSR consistently outperforms existing state-of-the-art approaches, confirming its strong robustness and generalization to unknown scenes.
Figures & tables
Figure 1: (a) t-SNE visualization of feature embeddings from different data sources. (b) Average RMSE across all benchmarks at different scales.
Figure 2: Pipeline of FoundDSR. In federated learning, global data are distributed to independent clients for local optimization, while the server aggregates and redistributes model weights. For each client, LR depth is encoded into initial Gaussian G , which are modulated by RGB gradients for anisotropic structural deformation. The transformed Gaussian G^ are then rendered into Dgs . Finally, Dgs and RGB I are fed into a depth reconstruction to predict HR depth DHR .
Figure 3: Visualization of Gaussian primitive modulation with and without RGB guidance.
Methods
Datasets
×2.5
×3.0
×4.0
×5.7
×6.0
×7.1
×8.0
×9.3
×11.0
×12.5
×13.0
×14.9
×16.0
×19.3
PromptDA
NYU-v2
17.34
17.06
17.64
17.57
17.61
17.58
18.44
18.01
18.36
18.56
18.66
18.49
19.24
19.00
Marigold-DC
9.12
9.15
9.26
9.48
9.53
9.80
10.01
10.30
10.90
11.38
11.55
12.13
12.53
13.79
InfiniDepth
8.91
8.73
8.70
8.68
8.71
9.02
9.15
9.54
10.15
10.46
10.86
11.20
11.65
13.56
LDCM
4.54
4.57
4.95
6.15
6.50
7.04
7.58
8.44
9.18
9.89
10.28
11.12
11.52
13.12
FDSR
3.90
3.64
4.03
3.85
4.08
4.28
4.99
5.13
5.60
6.30
6.82
9.00
9.98
12.29
DCTNet
3.33
3.41
3.54
4.05
4.17
4.64
5.01
5.65
6.57
7.56
7.93
9.22
9.98
11.97
Table 1: Zero-shot comparisons on 5 synthetic benchmarks under integer and non-integer scales. All foundation models use their large variants. The best and second-best results are highlighted.
Figure 4: Visual comparisons on the ×16.0 NYU‑v2 dataset. The zoomed‑in patches in (d)–(j) highlight the error maps within the white boxes, where darker purple shades indicate higher errors.
RMSE
PromptDA
Marigold-DC
InfiniDepth
LDCM
FDSR
DCTNet
DORNet
C2PD
DuCos
FoundDSR-S
FoundDSR
RGB-D-D
7.06
6.85
6.87
7.65
7.41
7.82
7.49
7.26
7.44
6.93
6.68
TOFDSR
8.79
7.80
8.48
8.22
7.23
7.45
7.34
7.23
7.42
7.05
6.92
Table 2: Zero-shot comparisons on the real-world RGB-D-D and TOFDSR datasets.
Methods
NYU-v2
Lu
Middlebury
RGB-D-D
iBims-1
0.02
0.03
0.04
0.02
0.03
0.04
0.02
0.03
0.04
0.02
0.03
0.04
0.02
0.03
0.04
PromptDA
17.42
17.34
17.36
22.49
22.42
22.40
15.12
15.52
11.93
5.23
5.35
5.53
38.83
39.06
39.48
Marigold-DC
8.71
8.79
8.97
36.42
36.04
35.94
5.75
5.83
5.65
2.93
2.93
2.96
33.90
33.80
33.89
LDCM
6.00
7.40
9.24
4.64
6.24
7.99
4.33
5.88
7.46
3.00
3.62
4.34
23.76
26.85
30.44
FDSR
5.82
6.56
7.05
3.18
3.66
3.96
3.00
3.49
3.86
2.26
2.56
2.92
18.61
19.71
20.62
DCTNet
6.24
6.94
7.81
3.32
3.94
4.30
3.35
3.96
4.22
2.58
2.95
3.44
20.08
21.83
23.36
Table 3: Zero-shot comparisons on multiple ×4.0 benchmarks with noise levels from 0.02 to 0.04 .
Figure 5: Zero-shot comparisons of continuous upsampling factors with prior foundation models and DSR methods on NYU-v2.
Table 9
Figure 6: Ablation study on (a) training data scaling and (b) heterogeneous federated learning. RMSE is averaged over five synthetic datasets.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Type
Resolution
Statistic
DIML
Real-World
1344×756
1.6K
Hypersim
Synthetic
1024×768
46.1K
DynamicReplica
Synthetic
1280×720
336.9K
LingBot-Depth
Synthetic
1280×960
1.0M
Total
-
-
1.4M
Appendix
Table 6: Overview of the training datasets.
Methods
Datasets
×2.5
×3.0
×4.0
×5.7
×6.0
×7.1
×8.0
×9.3
×11.0
×12.5
×13.0
×14.9
×16.0
×19.3
PromptDA
NYU-v2
7.58
7.54
7.67
7.74
7.73
7.67
8.04
7.82
8.01
9.16
8.16
8.23
8.50
8.49
Marigold-DC
4.18
4.20
4.27
4.39
4.41
4.56
4.68
4.83
5.14
5.38
5.53
5.79
6.04
6.73
InfiniDepth
3.33
3.37
3.42
3.53
3.59
3.78
3.91
4.60
4.52
4.72
4.94
5.11
5.45
6.53
LDCM
1.57
1.63
1.82
2.32
2.47
2.64
3.01
3.40
3.86
4.25
4.51
4.95
5.24
6.19
FDSR
1.12
1.10
1.19
1.28
1.33
1.48
1.68
1.84
2.10
2.49
2.74
3.70
4.34
5.54
DCTNet
1.26
1.29
1.36
1.58
1.63
1.82
2.01
2.30
2.84
3.36
3.56
4.17
4.62
5.65
Appendix
Table 7: Quantitative comparisons on five synthetic benchmarks using the MAE ↓ (cm) metric.
Figure 7: Visual comparisons on the real-world RGB-D-D dataset. The zoomed‑in patches in (d)–(j) highlight the error maps within the white boxes, where darker purple shades indicate higher errors.
Methods
Datasets
×2.5
×3.0
×4.0
×5.7
×6.0
×7.1
×8.0
×9.3
×11.0
×12.5
×13.0
×14.9
×16.0
×19.3
PromptDA
NYU-v2
92.39
92.46
92.20
92.13
92.16
92.26
91.57
91.90
91.59
91.38
91.45
91.23
90.66
90.73
Marigold-DC
95.91
95.87
95.69
95.38
95.33
94.97
94.56
94.21
93.47
92.87
92.58
91.85
91.12
89.73
InfiniDepth
94.37
94.21
94.06
93.80
93.59
93.29
92.79
92.74
91.63
91.15
90.81
90.31
89.30
87.52
LDCM
98.88
98.78
98.39
97.43
97.23
96.55
95.94
95.24
94.23
93.39
93.09
92.09
91.28
89.74
FDSR
99.00
99.10
99.00
99.05
98.98
98.88
98.62
98.46
98.15
97.56
97.13
95.04
93.45
90.99
DCTNet
99.19
99.16
99.10
98.83
98.76
98.51
98.25
97.84
96.84
95.66
95.28
93.81
92.62
90.66
Appendix
Table 8: Quantitative comparisons on five synthetic benchmarks using the δ1.05↑ metric.
Figure 8: Visual comparisons on NYU-v2 across integer scales. (a)–(j) denote RGB, LR, GT, Marigold-DC, LDCM, DCTNet, DORNet, DuCos, FoundDSR-S, and FoundDSR, respectively.
Figure 9: Visual comparisons on RGB-D-D across non-integer scales. (a)–(j) denote RGB, LR, GT, Marigold-DC, LDCM, DCTNet, DORNet, DuCos, FoundDSR-S, and FoundDSR, respectively.
Figure 10: Visual comparisons on ×4.0 RGB-D-D under various noise levels. (a)–(j): RGB, LR, GT, Marigold-DC, LDCM, DCTNet, DORNet, DuCos, FoundDSR-S, and FoundDSR.
Figure 11: Ablation study of (a) different losses and (b) guided 2D GS and depth reconstruction (DR) on ×13.0 , ×14.9 , and ×16.0 NYU-v2.
3D super-resolution (3DSR) aims to reconstruct high-resolution (HR) 3D scenes from low-resolution (LR) multi-view images. Existing methods rely on dense LR inputs and per-scene optimization, which restricts the high-frequency priors for constructing HR 3D Gaussian Splatting (3DGS) to those inherited from pretrained 2D super-resolution (2DSR) models. This severely limits reconstruction fidelity, cross-scene generalization, and real-time usability. We propose to reformulate 3DSR as a direct feed-forward mapping from sparse LR views to HR 3DGS representations, enabling the model to autonomously learn 3D-specific high-frequency geometry and appearance from large-scale, multi-scene data. This fundamentally changes how 3DSR acquires high-frequency knowledge and enables robust generalization to unseen scenes. Specifically, we introduce SR3R, a feed-forward framework that directly predicts HR 3DGS representations from sparse LR views via the learned mapping network. To further enhance reconstruction fidelity, we introduce Gaussian offset learning and feature refinement, which stabilize reconstruction and sharpen high-frequency details. SR3R is plug-and-play and can be paired with any feed-forward 3DGS reconstruction backbone: the backbone provides an LR 3DGS scaffold, and SR3R upscales it to an HR 3DGS. Extensive experiments across three 3D benchmarks demonstrate that SR3R surpasses state-of-the-art (SOTA) 3DSR methods and achieves strong zero-shot generalization, even outperforming SOTA per-scene optimization methods on unseen scenes.
Xiang Feng, Xiangbo Wang, Tieshi Zhong +7
Hangzhou Dianzi University · ShanghaiTech University · Peking University +1
Accurate, dense depth estimation is crucial for robotic perception, but commodity sensors often yield sparse or incomplete measurements due to hardware limitations. Existing RGBD-fused depth completion methods learn priors jointly conditioned on training RGB distribution and specific depth patterns, limiting domain generalization and robustness to various depth patterns. Recent efforts leverage monocular depth estimation (MDE) models to introduce domain-general geometric priors, but current two-stage integration strategies relying on explicit relative-to-metric alignment incur additional computation and introduce structured distortions. To this end, we present Any2Full, a one-stage, domain-general, and pattern-agnostic framework that reformulates completion as a scale-prompting adaptation of a pretrained MDE model. To address varying depth sparsity levels and irregular spatial distributions, we design a Scale-Aware Prompt Encoder. It distills scale cues from sparse inputs into unified scale prompts, guiding the MDE model toward globally scale-consistent predictions while preserving its geometric priors. Extensive experiments demonstrate that Any2Full achieves superior robustness and efficiency. It outperforms OMNI-DC by 32.2% in average AbsREL and delivers a 1.4× speedup over PriorDA with the same MDE backbone, establishing a new paradigm for universal depth completion. Codes and checkpoints are available at https://github.com/zhiyuandaily/Any2Full.
Zhiyuan Zhou, Ruofeng Liu, Taichi Liu +4
Rutgers University, USA · Michigan State University, USA · JD Logistics, China +1
Depth super-resolution (DSR) aims to recover a high-resolution (HR) depth map from its low-resolution (LR) counterpart. With color image guidance, this task is typically formulated as learning the residual between HR and LR in a low-dimensional feature space. However, this additive formulation is insufficient to accurately capture the complex relationship between HR and LR, especially under spatially varying degradations. In this paper, we introduce DegBins, a novel DSR framework that leverages degradation-driven binning to adaptively enhance residual modeling. Specifically, DegBins reformulates the regression-based DSR as a hybrid classification-regression problem, where the residual depth is represented as a linear combination of discrete depth bins weighted by their learned probability distribution, yielding more flexible and expressive representations. Furthermore, DegBins models the degradation relationship between HR and LR in a high-dimensional feature space, enabling adaptive bin range adjustment and probability optimization conditioned on local degradation characteristics. To progressively improve reconstruction quality, DegBins adopts a multi-stage refinement scheme, where each stage performs finer-grained bin partitioning and probability updating based on the former estimation. This coarse-to-fine design facilitates more accurate depth recovery, particularly in regions with severe degradations or complex structural variations. Extensive experiments across five benchmarks demonstrate that DegBins consistently outperforms existing state-of-the-art methods in terms of accuracy, robustness, and generalization.
Zhiqiang Yan, Zhengxue Wang, Jian Yang +1
Department of Computer Science, National University of Singapore · Nanjing University of Science and Technology