Recently, feature upsampling has gained increasing attention owing to its effectiveness in enhancing vision foundation models (VFMs) for pixel-level understanding tasks. Existing methods typically rely on high-resolution features from the same foundation model to achieve upsampling via self-reconstruction. However, relying solely on intra-model features forces the upsampler to overfit to the source model's inherent location misalignment and high-norm artifacts. To address this fundamental limitation, we propose DiveUp, a novel framework that breaks away from single-model dependency by introducing multi-VFM relational guidance. Instead of naive feature fusion, DiveUp leverages diverse VFMs as a panel of experts, utilizing their structural consensus to regularize the upsampler's learning process, effectively preventing the propagation of inaccurate spatial structures from the source model. To reconcile the unaligned feature spaces across different VFMs, we propose a universal relational feature representation, formulated as a local center-of-mass (COM) field, that extracts intrinsic geometric structures, enabling seamless cross-model interaction. Furthermore, we introduce a spikiness-aware selection strategy that evaluates the spatial reliability of each VFM, effectively filtering out high-norm artifacts to aggregate guidance from only the most reliable expert at each local region. DiveUp is a unified, encoder-agnostic framework; a jointly-trained model can universally upsample features from diverse VFMs without requiring per-model retraining. Extensive experiments demonstrate that DiveUp achieves state-of-the-art performance across various downstream dense prediction tasks, validating the efficacy of multi-expert relational guidance. Our code and models are available at: https://github.com/Xiaoqiong-Liu/DiveUp
Figures & tables
Figure 1: Overview of the DiveUp, which leverages diverse VFMs to learn feature upsampling. Best viewed in color and by zooming in for all the figures in this paper .
Figure 2: The DiveUp Multi-VFM Relational Guidance Framework. (1) Relational Feature Representation. We compute local self-affinity (a) to evaluate spatial reliability via entropy (b) and extract the local center-of-mass (COM) field (c) . In homogeneous regions, the expected spatial offset is negligible ( μ(p)≈0 ), whereas it shifts directionally at semantic boundaries ( μ(p)=0 ), effectively capturing intrinsic geometric structures. (2) Spikiness-Aware COM Field Fusion. DiveUp extracts these relational representations from diverse VFMs (d) and employs a spikiness-aware source selection (e) . By adaptively evaluating the spatial reliability of each VFM, we generate dynamic gating weights to fuse multiple COM fields into a unified Consensus COM Field (f) . This serves as the robust location alignment target for high-resolution feature reconstruction.
Methods
V.A.
Semantic Segmentation (Pascal VOC)
Depth Estimation (NYUv2)
SigLIP-B
DINOv2-S
RADIOv2.5
DINOv3-B
SigLIP-B
DINOv2-S
RADIOv2.5
DINOv3-B
Bilinear
✓
71.88
80.70
84.46
86.99
71.49
82.51
83.06
85.39
LoftUp [ICCV’25] [ 16 ]
✗
46.77
83.66
-
-
48.54
80.85
-
-
JAFAR [NeurIPS’25] [ 5 ]
✗
77.36
84.24
85.93
87.10
75.50
83.40
83.51
84.38
AnyUp-S [ICLR’26] [ 42 ]
✓
78.48
83.85
85.51
86.62
74.46
83.61
84.83
86.30
AnyUp-M [ICLR’26] [ 42 ]
✓
77.71
84.11
85.67
86.14
75.81
83.70
84.93
86.54
Table 1: Linear probing results of semantic segmentation (mIoU ↑ ) on Pascal VOC [ 9 ] and depth estimation ( δ1↑ ) on NYUv2 [ 34 ] . We evaluate features extracted from various VFMs. ‘V.A.’ indicates VFM-agnostic models. ‘ † ’ denotes results of NAF [ 2 ] are obtained using an additional refinement stage. ‘-’ indicates results not available. The best and the second best results are highlighted in bold and underlined , respectively, for all tables in the paper.
Semantic Segmentation (mIoU ↑ )
Depth Est. ( δ1↑ )
Methods
V.A.
COCO
Pascal VOC
ADE20K
NYUv2
Bilinear
✓
59.03
80.70
39.23
82.51
FeatUp [ICLR’24] [ 12 ]
✗
60.10
81.08
38.82
82.53
LoftUp [ICCV’25] [ 16 ]
✗
62.23
83.66
42.01
80.85
JAFAR [NeurIPS’25] [ 5 ]
✗
60.78
84.24
40.49
83.40
AnyUp-S [ICLR’26] [ 42 ]
✓
61.10
83.85
42.26
83.61
Table 2: Comparison across tasks on different datasets. The linear probing results of all methods are obtained using DINOv2-S. ‘V.A.’ indicates VFM-agnostic models. ‘ † ’ denotes results of NAF [ 2 ] are obtained using an additional refinement stage.
Figure 3: Comparison of upsampled features from different methods.
Figure 4: Comparison of prediction results on semantic segmentation (a) in Pascal VOC and depth estimation (b) for different methods in NYUv2.
Target VFMs
Methods
CLIP-B
SwinT-L
Avg.
NAF [CVPR’26] [ 2 ]
53.27
50.83
52.05
AnyUp-S [ICLR’26] [ 42 ]
50.89
49.90
50.40
AnyUp-M [ICLR’26] [ 42 ]
50.43
48.96
49.70
DiveUp (ours)
53.65
52.58
53.12
Table 3: Generalization to unseen semantic spaces on Cityscapes [ 4 ] using mIoU ( ↑ ).
Methods
Pascal VoC (mIoU ↑ )
JAFAR [NeurIPS’25] [ 5 ] *
84.15
JAFAR with DiveUp
84.94 (+0.79)
NAF [CVPR’26] [ 2 ] *
84.29
NAF with DiveUp
84.92 (+0.63)
Table 4: Generality of DiveUp in existing upsamplers JAFAR and NAF. The experiments are conducted on semantic segmentation with DINOv2-S on Pascal VOC. * indicates re-trained results for fair comparison.
Methods
Params (M ↓ )
Tra. Time (hour ↓ )
FLOPs (G ↓ )
Inf. Speed (FPS ↑ )
Inf. GPU Mem. (G ↓ )
Pascal VOC (mIoU ↑ )
NYUv2 ( δ1↑ )
FeatUp [ICLR’24] [ 12 ]
0.17
23.2
83.0
17.5
2.6
81.08
82.53
JAFAR [NeurIPS’25] [ 5 ]
0.63
2.7
366.5
9.2
5.3
84.24
83.40
AnyUp [ICLR’26] [ 42 ]
0.88
10.1
328.5
5.5
6.0
83.85
83.61
NAF [CVPR’26] [ 2 ]
0.66
1.0
265.1
17.8
1.7
84.52
84.06
DiveUp (ours)
0.66
1.5
265.1
17.8
1.7
85.08
84.73
Table 5: Comparison on model efficacy and complexity for different methods for a ×16 upsampling of input features
DINOv3-L
RADIOv3-L
Pascal VOC (mIoU ↑ )
❶
76.42
❷
✓
77.18 (+0.76)
❸
✓
77.57 (+1.15)
❹
✓
✓
77.84 (+1.42)
Table 6: Ablation experiments on applying different VFMs in DiveUp using frozen CLIP as Flr .
VFM Fusion
Pascal VOC (mIoU ↑ )
❶
Addition
84.51
❷
Concatenation
84.66
❸
SA selection (ours)
84.92
Table 7: Ablation experiments on VFM fusion strategies in DiveUp using frozen DINOv2-S as Flr .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Target VFM
Semantic Segmentation (mIoU %)
Absolute Gains
NAF [ 2 ]
AnyUp-S [ 42 ]
AnyUp-M [ 42 ]
DiveUp (Ours)
vs. NAF
vs. AnyUp-M
ResNet-50 [ 14 ]
51.44
54.60
56.85
57.08
+5.64
+0.23
PE-Core-S [ 1 ]
72.35
73.69
73.96
75.31
+2.96
+1.35
DFN-CLIP-B [ 11 ]
77.66
77.62
78.11
78.81
+1.15
+0.70
PaliGemma2 [ 36 ]
79.58
79.64
80.24
81.33
+1.75
+1.09
MaskCLIP-B [ 45 ]
73.37
73.75
75.14
75.57
+2.20
+0.43
Appendix
Table 8: Zero-Shot Generalization Across Diverse and Noisy VFMs. Performance evaluated on the PASCAL VOC segmentation benchmark using mIoU (%) without per-model retraining. DiveUp shows robustness and architecture-agnostic generalization, outperforming baselines across all evaluated noisy backbones. The best and the second best results are highlighted in bold and underlined , respectively.
Target VFM
Semantic Segmentation (mIoU %)
Absolute Gain
NAF (Baseline)
NAF w/ DiveUp
PE-Core-S
74.37
75.91
+1.54
DFN-CLIP-B
78.04
78.78
+0.74
MaskCLIP-B
73.99
75.16
+1.17
ClearCLIP-B
70.98
73.70
+2.72
Appendix
Table 9: Plug-and-Play Robustness on Noisy Backbones. Performance (mIoU %) evaluated on the PASCAL VOC segmentation task. By upgrading the standard NAF baseline with our multi-VFM relational guidance (NAF w/ DiveUp), the upsampling performance is consistently improved.
Window Size ( w )
Semantic Segmentation
mIoU (%)
Acc (%)
3×3
78.94
94.70
5×5
79.55
94.88
7×7 (Default)
79.60
94.90
9×9
79.43
94.84
11×11
79.29
94.80
Appendix
Table 10: Ablation on Local Window Size ( w×w ). Performance evaluated on PASCAL VOC using the SigLIP-B backbone. The upsampling performance peaks at w=7 , which balances spatial context with the exclusion of distant noise.
Figure 5: Visualization of Relational Representations (Before vs. After DiveUp). We visualize the intermediate local entropy maps and COM fields generated by a noisy backbone (ClearClip-B), compared with those generated after integrating DiveUp’s multi-VFM relational guidance. DiveUp suppresses background interference and spatial uncertainty, achieving more localized geometric structures. This improved location alignment corresponds to the quality of the final segmentation results (see IoU gains).
Figure 6: Qualitative Comparison for Semantic Segmentation. Results are produced using the SigLIP-B backbone on the PASCAL VOC dataset. Compared to other VFM-agnostic and VFM-specific upsamplers, DiveUp eliminates semantic bleeding and generates sharper object boundaries, reflecting the intrinsic geometric structures.
Figure 7: Qualitative Comparison for Dense Depth Estimation. Results are produced using the DINOv2-S backbone on the NYUv2 dataset. DiveUp preserves depth discontinuities and complex object boundaries.
Self-supervised Vision Foundation Models (VFMs) have become essential backbones for downstream tasks due to their strong and transferable visual representations. However, their patch-token-level features are often too coarse for dense prediction tasks such as semantic segmentation and depth estimation when accurate fine-grained predictions are required. Feature upsampling methods have been developed to recover pixel-level detail but still face limitations. Learnable upsamplers are often designed for a specific encoders and must be retrained for different encoders. Image-guided methods that use shallow pixel encoders often introduce textural artifacts and lack the semantic guidance needed for accurate downstream predictions. We introduce PixelUp, a zero-shot VFM-agnostic upsampler achieving semantic awareness through a coarse-to-fine chain of windowed cross-attention architecture guided by multi-scale semantic features. We demonstrate that PixelUp outperforms both VFM-specific and VFM-agnostic upsamplers, achieving state-of-the-art performance on dense prediction tasks with an average improvement of +1.2 mIoU on semantic segmentation and +0.25 δ1, on NYUv2 depth estimation across VFMs. PixelUp further improves training-free open-vocabulary and unsupervised semantic segmentation by an average of +1.3 mIoU and +0.5 mIoU, respectively. Code available at https://pixelup-project.vercel.app/
Pre-trained Vision Foundation Models (VFMs) have become central to modern computer vision due to their powerful semantic representations and strong generalization ability. However, their patchified or pooled outputs are inherently low-resolution, limiting their effectiveness in tasks requiring fine-grained, pixel-level reasoning. Existing feature upsampling approaches either degrade semantic fidelity or rely on VFM-specific retraining and heavy architectures, hindering efficiency and scalability. To address these challenges, we propose RaysUp, an ultra-lightweight, task-agnostic, and VFM-agnostic feature upsampling framework that reconstructs high-resolution feature maps at arbitrary resolutions. Unlike conventional 2D interpolation or attention-based schemes, RaysUp lifts feature reconstruction into a geometry-aware ray domain. Specifically, we introduce a Spatially Decoupled Guidance Encoder for direction-aware guidance encoding, an Any-Resolution Cross-Attention mechanism for resolution-flexible reconstruction, and a novel Ray Positional Encoding (RayPE) that injects implicit 3D geometric priors via 6D Plucker ray coordinates. Finally, a Geometry-Aware Neighborhood Attention module further ensures content-adaptive bilateral aggregation while preserving geometric consistency. Extensive experiments across diverse dense prediction tasks demonstrate that RaysUp achieves state-of-the-art performance while using only 16% of the parameters of AnyUp and delivering approximately 7x faster inference. These results highlight a substantially improved accuracy-efficiency trade-off and establish RaysUp as a practical and scalable solution for universal feature upsampling. Code is available at https://github.com/MAP-RaysUp/RaysUp.
Yuchuan Ding, Linfei Li, Lin Zhang +1
School of Computer Science and Technology, Tongji University, Shanghai, China
Vision Transformers (ViTs) have become a dominant architecture for visual representation learning, providing exceptionally strong and broadly reusable backbone features. However, ViTs are commonly operated on relatively small patch-token grids due to the quadratic cost of global self-attention, which creates a persistent bottleneck for dense prediction tasks such as semantic segmentation and depth estimation. This has motivated the development of task-agnostic feature upsamplers. While recent state-of-the-art methods produce visually sharp dense representations, their reliance on shallow image encoders for guided upsampling can introduce feature leakage, fragmentation, and blur. We introduce ViT-Up, an implicit feature upsampling framework that replaces external image guidance with layer-wise query construction from intermediate ViT hidden states. This enables feature prediction at arbitrary continuous image coordinates while preserving alignment with the backbone feature space. Experiments demonstrate that ViT-Up consistently outperforms state-of-the-art image-guided upsamplers across dense prediction and semantic correspondence. On DINOv3-S+, ViT-Up improves over prior methods by up to +2.07 mIoU on Cityscapes and +4.17 [email protected] on SPair-71k. With the larger DINOv3-B backbone, these gains increase to +3.36 mIoU and +8.09 [email protected], demonstrating that ViT-Up scales favorably with backbone capacity.