Organizations: Hokkaido University, Sapporo, Japan · Westlake University, Hangzhou, China · Zhejiang University, Hangzhou, China · The University of Tokyo, Tokyo, Japan
A typical 2D-to-3D pipeline takes multi-view images as input, where a Vision Foundation Model (VFM) extracts features that are spatially upsampled to dense representations for 3D reconstruction. If dense features across views preserve geometric consistency, differentiable rendering can recover an accurate 3D representation, making the feature upsampler a critical component. Recent learnable upsampling methods mainly aim to enhance spatial details, such as sharper geometry or richer textures, yet their impact on 3D awareness remains underexplored. To address this gap, we introduce a spectral diagnostic framework with six complementary metrics that characterize amplitude redistribution, structural spectral alignment, and directional stability. Across classical interpolation and learnable upsampling methods on CLIP and DINO backbones, we observe three key findings. First, structural spectral consistency (SSC/CSC) is the strongest predictor of NVS quality, whereas High-Frequency Spectral Slope Drift (HFSS) often correlates negatively with reconstruction performance, indicating that emphasizing high-frequency details alone does not necessarily improve 3D reconstruction. Second, geometry and texture respond to different spectral properties: Angular Energy Consistency (ADC) correlates more strongly with geometry-related metrics, while SSC/CSC influence texture fidelity slightly more than geometric accuracy. Third, although learnable upsamplers often produce sharper spatial features, they rarely outperform classical interpolation in reconstruction quality, and their effectiveness depends on the reconstruction model. Overall, our results indicate that reconstruction quality is more closely related to preserving spectral structure than to enhancing spatial detail, highlighting spectral consistency as an important principle for designing upsampling strategies in 2D-to-3D pipelines.
Figures & tables
Figure 1: Overview of the spectral probing pipeline. Feature upsamplers reshape spectral structure, which in turn affects reconstruction quality. Our framework measures this relationship through six spectral diagnostics and NVS evaluation. Specifically, multi-view images are resized to H×W and encoded into patch-grid features ( h×w ). Different upsampling strategies produce dense maps at H×W , which are used to regress 3D Gaussian parameters via differentiable rendering. Spectral changes are correlated with NVS quality (PSNR, SSIM, LPIPS) to probe 3D awareness.
Dataset
Scene Type
Complexity
View Range
Views
LLFF [ 23 ]
Indoor
Simple
Small
2
DL3DV [ 21 ]
Indoor / Outdoor
Moderate
Medium
5–6
Casual [ 8 ]
Daily Scenario
Moderate
Medium
4–7
MipNeRF360 [ 3 ]
Unbounded
Moderate
360
6
MVImgNet [ 31 ]
Outdoor Object
Moderate
180–360
2–4
T&T [ 18 ]
Indoor / Outdoor
High
Large
6
Table 1: Datasets for Evaluation.
All
Geometry
Texture
Feature
Upsamplers
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
DINO
NSM
23.62
0.8383
0.1628
18.13
0.7811
0.2212
22.57
0.8204
0.1415
Bilinear
24.08
0.8400
0.1601
24.32
0.8483
0.1368
22.58
0.8193
0.1413
NN
23.94
0.8396
0.1606
22.54
0.8214
0.1790
22.71
0.8211
0.1392
Lanczos
24.07
0.8408
0.1582
24.29
0.8475
0.1342
22.71
0.8215
0.1392
Bicubic
24.15
0.8416
0.1577
24.36
0.8477
0.1343
22.83
0.8237
0.1370
Table 2: Cross-dataset average NVS performance using DUSt3R [ 28 ] as the 3D reconstructor. Results are reported under three probing modes: All (geometry+texture), Geometry-only, and Texture-only. Metrics are averaged over 30 scenes from six datasets. Best results are highlighted in bold.
LLFF
DL3DV
G eometry
T exture
A ll
G eometry
T exture
A ll
Upsamplers
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
NSM
19.21
.8350
.1593
24.11
.8904
.0779
25.13
.9028
.0858
18.26
.8207
.1967
20.62
.8334
.1555
22.09
.8616
.1596
Bilinear
25.05
.9021
.0745
23.96
.8882
.0803
24.99
.8899
.0919
22.27
.8614
.1514
20.86
.8373
.1528
22.00
.8618
.1590
NN
23.04
.8529
.1256
24.00
.8882
.0802
25.02
.8887
.0923
20.74
.8474
.1712
21.03
.8388
.1520
22.03
.8607
.1598
Lanczos
24.91
.8996
.0750
24.14
.8906
.0779
24.96
.8880
.0915
22.36
.8609
.1502
20.63
.8329
.1560
21.93
.8604
.1598
Table 4: Per-dataset NVS performance using DUSt3R [ 28 ] with the CLIP backbone on six multi-view datasets under three probing modes: Geometry-only, Texture-only, and All (geometry+texture). Best results are highlighted in bold. More results are provided in the Supplementary Material.
Figure 2: NVS visualizations using CLIP+DUSt3R [ 28 ] . Classical interpolation methods often achieve performance comparable to learned upsamplers. Best and second-best results are shown in bold and underlined , respectively.
Figure 3: Spectral–reconstruction correlations. Each row shows the Spearman correlation heatmap. Results of other upsamplers are provided in the Supplementary Material.
Figure 4: Spectral diagnostics under geometry-only (AG) and texture-only (AT) settings (CLIP+DUSt3R [ 28 ] ). Additional results are in the Supplementary Material.
LLFF
DL3DV
G eometry
T exture
A ll
G eometry
T exture
A ll
Upsamplers
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
NSM
18.54
.8298
.1655
23.04
.8804
.0949
21.97
.8657
.1230
16.11
.7658
.2303
20.98
.8408
.1498
22.22
.8628
.1575
Bilinear
25.01
.9061
.0696
23.97
.8879
.0804
24.08
.8791
.1052
22.62
.8681
.1453
20.92
.8395
.1509
22.16
.8619
.1582
NN
22.63
.8501
.1275
24.00
.8882
.0803
24.02
.8797
.1050
20.93
.8489
.1695
20.91
.8394
.1513
21.72
.8570
.1620
Lanczos
25.04
.9060
.0703
24.12
.8903
.0782
25.04
.8884
.0904
22.58
.8662
.1458
20.95
.8399
.1506
22.00
.8618
.1582
Table 5: Per-dataset NVS performance using DUSt3R [ 28 ] with the DINO backbone. Results are reported on six multi-view datasets (LLFF, DL3DV, Casual, MipNeRF360, MVImgNet, and T&T) under three probing modes: Geometry-only, Texture-only, and All (geometry + texture). We evaluate PSNR ( ↑ ), SSIM ( ↑ ), and LPIPS ( ↓ ). NSM denotes the no-upsampling baseline. Best results within each dataset and probing mode are highlighted in bold.
LLFF
DL3DV
G eometry
T exture
A ll
G eometry
T exture
A ll
Upsamplers
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
NSM
17.60
.8187
.1717
15.13
.7452
.1924
17.48
.8140
.1707
15.83
.7822
.2176
13.75
.6983
.2611
16.04
.7866
.2243
Bilinear
14.06
.7144
.2571
13.87
.7051
.2647
16.03
.7493
.2252
14.35
.7114
.2658
13.13
.6677
.2897
17.53
.7830
.2266
NN
16.85
.7931
.1808
15.12
.7463
.1902
17.31
.8060
.1802
16.19
.7817
.2213
13.67
.6961
.2633
15.81
.7755
.2259
Lanczos
15.17
.7256
.2290
15.19
.7250
.2523
17.97
.8126
.1723
15.87
.7641
.2317
13.70
.6976
.2620
15.44
.7838
.2251
Table 6: Per-dataset NVS performance using MASt3R [ 20 ] with the DINO backbone. Results are reported on six multi-view datasets (LLFF, DL3DV, Casual, MipNeRF360, MVImgNet, and T&T) under three probing modes: Geometry-only, Texture-only, and All (geometry + texture). We evaluate PSNR ( ↑ ), SSIM ( ↑ ), and LPIPS ( ↓ ). NSM denotes the no-upsampling baseline. Best results within each dataset and probing mode are highlighted in bold.
LLFF
DL3DV
G eometry
T exture
A ll
G eometry
T exture
A ll
Upsamplers
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
NSM
17.46
.8081
.1773
13.75
.7039
.2465
16.79
.7851
.1899
16.26
.7996
.2065
13.94
.7168
.2494
16.24
.7976
.2136
Bilinear
15.59
.7424
.2066
15.11
.7220
.2308
15.03
.7064
.2687
14.53
.7368
.2476
13.50
.6992
.2647
16.50
.7489
.2376
NN
16.52
7701
.2153
15.18
.7259
.2491
16.84
.7833
.1919
16.82
.7947
.2127
13.88
.7146
.2523
16.17
.7987
.2115
Lanczos
14.19
.7220
.2489
14.38
.7190
.2206
16.19
.7781
.2049
15.36
.7411
.2489
14.06
.7157
.2538
17.04
.7968
.2181
Table 7: Per-dataset NVS performance using MASt3R [ 20 ] with the CLIP backbone. Results are reported on six multi-view datasets (LLFF, DL3DV, Casual, MipNeRF360, MVImgNet, and T&T) under three probing modes: Geometry-only, Texture-only, and All (geometry + texture). We evaluate PSNR ( ↑ ), SSIM ( ↑ ), and LPIPS ( ↓ ). NSM denotes the no-upsampling baseline. Best results within each dataset and probing mode are highlighted in bold.
Figure 6: Spectral–reconstruction correlations. Each row shows the Spearman correlation heatmap.
Figure 7: Spectral–reconstruction correlations. Each row shows the Spearman correlation heatmap.
Figure 8: Spearman correlations between spectral diagnostics and reconstruction quality under geometry-only (AG) and texture-only (AT) probing settings (DINO). Left panels show correlations with −RPEmean (geometry quality), and right panels show correlations with −LPIPS (texture quality).
Figure 9: Spearman correlations between spectral diagnostics and reconstruction quality under geometry-only (AG) and texture-only (AT) probing settings. Left panels show correlations with −RPEmean (geometry quality), and right panels show correlations with −LPIPS (texture quality).
Figure 10: Spearman correlations between spectral diagnostics and reconstruction quality under geometry-only (AG) and texture-only (AT) probing settings. Left panels show correlations with −RPEmean (geometry quality), and right panels show correlations with −LPIPS (texture quality).
Figure 11: Spearman correlations between spectral diagnostics and reconstruction quality under geometry-only (AG) and texture-only (AT) probing settings. Left panels show correlations with −RPEmean (geometry quality), and right panels show correlations with −LPIPS (texture quality).
Figure 12: Spearman correlations between spectral diagnostics and reconstruction quality under geometry-only (AG) and texture-only (AT) probing settings. Left panels show correlations with −RPEmean (geometry quality), and right panels show correlations with −LPIPS (texture quality).
Pre-trained Vision Foundation Models (VFMs) have become central to modern computer vision due to their powerful semantic representations and strong generalization ability. However, their patchified or pooled outputs are inherently low-resolution, limiting their effectiveness in tasks requiring fine-grained, pixel-level reasoning. Existing feature upsampling approaches either degrade semantic fidelity or rely on VFM-specific retraining and heavy architectures, hindering efficiency and scalability. To address these challenges, we propose RaysUp, an ultra-lightweight, task-agnostic, and VFM-agnostic feature upsampling framework that reconstructs high-resolution feature maps at arbitrary resolutions. Unlike conventional 2D interpolation or attention-based schemes, RaysUp lifts feature reconstruction into a geometry-aware ray domain. Specifically, we introduce a Spatially Decoupled Guidance Encoder for direction-aware guidance encoding, an Any-Resolution Cross-Attention mechanism for resolution-flexible reconstruction, and a novel Ray Positional Encoding (RayPE) that injects implicit 3D geometric priors via 6D Plucker ray coordinates. Finally, a Geometry-Aware Neighborhood Attention module further ensures content-adaptive bilateral aggregation while preserving geometric consistency. Extensive experiments across diverse dense prediction tasks demonstrate that RaysUp achieves state-of-the-art performance while using only 16% of the parameters of AnyUp and delivering approximately 7x faster inference. These results highlight a substantially improved accuracy-efficiency trade-off and establish RaysUp as a practical and scalable solution for universal feature upsampling. Code is available at https://github.com/MAP-RaysUp/RaysUp.
Yuchuan Ding, Linfei Li, Lin Zhang +1
School of Computer Science and Technology, Tongji University, Shanghai, China
Recently, feature upsampling has gained increasing attention owing to its effectiveness in enhancing vision foundation models (VFMs) for pixel-level understanding tasks. Existing methods typically rely on high-resolution features from the same foundation model to achieve upsampling via self-reconstruction. However, relying solely on intra-model features forces the upsampler to overfit to the source model's inherent location misalignment and high-norm artifacts. To address this fundamental limitation, we propose DiveUp, a novel framework that breaks away from single-model dependency by introducing multi-VFM relational guidance. Instead of naive feature fusion, DiveUp leverages diverse VFMs as a panel of experts, utilizing their structural consensus to regularize the upsampler's learning process, effectively preventing the propagation of inaccurate spatial structures from the source model. To reconcile the unaligned feature spaces across different VFMs, we propose a universal relational feature representation, formulated as a local center-of-mass (COM) field, that extracts intrinsic geometric structures, enabling seamless cross-model interaction. Furthermore, we introduce a spikiness-aware selection strategy that evaluates the spatial reliability of each VFM, effectively filtering out high-norm artifacts to aggregate guidance from only the most reliable expert at each local region. DiveUp is a unified, encoder-agnostic framework; a jointly-trained model can universally upsample features from diverse VFMs without requiring per-model retraining. Extensive experiments demonstrate that DiveUp achieves state-of-the-art performance across various downstream dense prediction tasks, validating the efficacy of multi-expert relational guidance. Our code and models are available at: https://github.com/Xiaoqiong-Liu/DiveUp
Xiaoqiong Liu, Heng Fan
Department of Computer Science and Engineering, University of North Texas
Pre-trained vision foundation models (VFMs) provide strong semantic representations, yet their patch-level features are inherently coarse, limiting their effectiveness on tasks requiring fine-grained localization, dense prediction, and point-wise correspondence. In this work, we revisit feature upsampling for VFMs from the perspective of inverse problem and propose Weighted Reverse Convolution (WRC), a spatially adaptive inverse operator for densifying high-level visual descriptors. Specifically, we formulate feature upsampling as a weighted Tikhonov-regularized least-squares problem, where spatially varying weights modulate both data fidelity and prior strength at each spatial location. This allows WRC to adapt the reconstruction to spatially varying feature characteristics, thereby preserving critical structures while mitigating over-smoothing. Moreover, WRC retains an efficient, fully differentiable closed-form FFT solution, making it a practical drop-in upsampling operator. Integrated into a lightweight self-supervised densification framework, WRC consistently improves dense feature quality across various downstream benchmarks, including segmentation, depth estimation, video object segmentation, object discovery, and keypoint correspondence, while maintaining high computational efficiency.
Wentong Li, Zhiyuan Qi, Zichen Zhao +2
Nanjing University of Aeronautics and Astronautics · The Hong Kong Polytechnic University · Nanjing University