DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Models

Xiaoqiong Liu; Heng Fan

cs.CVMar 13, 2026

DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Models

Authors: Xiaoqiong Liu, Heng Fan

Organizations: Department of Computer Science and Engineering, University of North Texas

Abstract

Recently, feature upsampling has gained increasing attention owing to its effectiveness in enhancing vision foundation models (VFMs) for pixel-level understanding tasks. Existing methods typically rely on high-resolution features from the same foundation model to achieve upsampling via self-reconstruction. However, relying solely on intra-model features forces the upsampler to overfit to the source model's inherent location misalignment and high-norm artifacts. To address this fundamental limitation, we propose DiveUp, a novel framework that breaks away from single-model dependency by introducing multi-VFM relational guidance. Instead of naive feature fusion, DiveUp leverages diverse VFMs as a panel of experts, utilizing their structural consensus to regularize the upsampler's learning process, effectively preventing the propagation of inaccurate spatial structures from the source model. To reconcile the unaligned feature spaces across different VFMs, we propose a universal relational feature representation, formulated as a local center-of-mass (COM) field, that extracts intrinsic geometric structures, enabling seamless cross-model interaction. Furthermore, we introduce a spikiness-aware selection strategy that evaluates the spatial reliability of each VFM, effectively filtering out high-norm artifacts to aggregate guidance from only the most reliable expert at each local region. DiveUp is a unified, encoder-agnostic framework; a jointly-trained model can universally upsample features from diverse VFMs without requiring per-model retraining. Extensive experiments demonstrate that DiveUp achieves state-of-the-art performance across various downstream dense prediction tasks, validating the efficacy of multi-expert relational guidance. Our code and models are available at: https://github.com/Xiaoqiong-Liu/DiveUp

Figures & tables

Figure 1: Overview of the DiveUp, which leverages diverse VFMs to learn feature upsampling. Best viewed in color and by zooming in for all the figures in this paper .

Figure 2: The DiveUp Multi-VFM Relational Guidance Framework. (1) Relational Feature Representation. We compute local self-affinity (a) to evaluate spatial reliability via entropy (b) and extract the local center-of-mass (COM) field (c) . In homogeneous regions, the expected spatial offset is negligible ( $\boldsymbol{\mu}(p)\approx\mathbf{0}$ ), whereas it shifts directionally at semantic boundaries ( $\boldsymbol{\mu}(p)\neq\mathbf{0}$ ), effectively capturing intrinsic geometric structures. (2) Spikiness-Aware COM Field Fusion. DiveUp extracts these relational representations from diverse VFMs (d) and employs a spikiness-aware source selection (e) . By adaptively evaluating the spatial reliability of each VFM, we generate dynamic gating weights to fuse multiple COM fields into a unified Consensus COM Field (f) . This serves as the robust location alignment target for high-resolution feature reconstruction.

Methods	V.A.	Semantic Segmentation (Pascal VOC)				Depth Estimation (NYUv2)
Methods	V.A.	SigLIP-B	DINOv2-S	RADIOv2.5	DINOv3-B	SigLIP-B	DINOv2-S	RADIOv2.5	DINOv3-B
Bilinear	✓	71.88	80.70	84.46	86.99	71.49	82.51	83.06	85.39
LoftUp [ICCV’25] [ 16 ]	✗	46.77	83.66	-	-	48.54	80.85	-	-
JAFAR [NeurIPS’25] [ 5 ]	✗	77.36	84.24	85.93	87.10	75.50	83.40	83.51	84.38
AnyUp-S [ICLR’26] [ 42 ]	✓	78.48	83.85	85.51	86.62	74.46	83.61	84.83	86.30
AnyUp-M [ICLR’26] [ 42 ]	✓	77.71	84.11	85.67	86.14	75.81	83.70	84.93	86.54

Table 1: Linear probing results of semantic segmentation (mIoU

\uparrow

) on Pascal VOC [ 9 ] and depth estimation (

\delta_{1}\uparrow

) on NYUv2 [ 34 ] . We evaluate features extracted from various VFMs. ‘V.A.’ indicates VFM-agnostic models. ‘

{\dagger}

’ denotes results of NAF [ 2 ] are obtained using an additional refinement stage. ‘-’ indicates results not available. The best and the second best results are highlighted in bold and underlined , respectively, for all tables in the paper.

		Semantic Segmentation (mIoU $\uparrow$ )			Depth Est. ( $\delta_{1}\uparrow$ )
Methods	V.A.	COCO	Pascal VOC	ADE20K	NYUv2
Bilinear	✓	59.03	80.70	39.23	82.51
FeatUp [ICLR’24] [ 12 ]	✗	60.10	81.08	38.82	82.53
LoftUp [ICCV’25] [ 16 ]	✗	62.23	83.66	42.01	80.85
JAFAR [NeurIPS’25] [ 5 ]	✗	60.78	84.24	40.49	83.40
AnyUp-S [ICLR’26] [ 42 ]	✓	61.10	83.85	42.26	83.61

Table 2: Comparison across tasks on different datasets. The linear probing results of all methods are obtained using DINOv2-S. ‘V.A.’ indicates VFM-agnostic models. ‘

{\dagger}

’ denotes results of NAF [ 2 ] are obtained using an additional refinement stage.

Figure 3: Comparison of upsampled features from different methods.

Figure 4: Comparison of prediction results on semantic segmentation (a) in Pascal VOC and depth estimation (b) for different methods in NYUv2.

	Target VFMs
Methods	CLIP-B	SwinT-L	Avg.
NAF [CVPR’26] [ 2 ]	53.27	50.83	52.05
AnyUp-S [ICLR’26] [ 42 ]	50.89	49.90	50.40
AnyUp-M [ICLR’26] [ 42 ]	50.43	48.96	49.70
DiveUp (ours)	53.65	52.58	53.12

Table 3: Generalization to unseen semantic spaces on Cityscapes [ 4 ] using mIoU (

\uparrow

Methods	Pascal VoC (mIoU $\uparrow$ )
JAFAR [NeurIPS’25] [ 5 ] *	84.15
JAFAR with DiveUp	84.94 (+0.79)
NAF [CVPR’26] [ 2 ] *	84.29
NAF with DiveUp	84.92 (+0.63)

Table 4: Generality of DiveUp in existing upsamplers JAFAR and NAF. The experiments are conducted on semantic segmentation with DINOv2-S on Pascal VOC. * indicates re-trained results for fair comparison.

Methods	Params (M $\downarrow$ )	Tra. Time (hour $\downarrow$ )	FLOPs (G $\downarrow$ )	Inf. Speed (FPS $\uparrow$ )	Inf. GPU Mem. (G $\downarrow$ )	Pascal VOC (mIoU $\uparrow$ )	NYUv2 ( $\delta_{1}$ $\uparrow$ )
FeatUp [ICLR’24] [ 12 ]	0.17	23.2	83.0	17.5	2.6	81.08	82.53
JAFAR [NeurIPS’25] [ 5 ]	0.63	2.7	366.5	9.2	5.3	84.24	83.40
AnyUp [ICLR’26] [ 42 ]	0.88	10.1	328.5	5.5	6.0	83.85	83.61
NAF [CVPR’26] [ 2 ]	0.66	1.0	265.1	17.8	1.7	84.52	84.06
DiveUp (ours)	0.66	1.5	265.1	17.8	1.7	85.08	84.73

Table 5: Comparison on model efficacy and complexity for different methods for a

\times 16

upsampling of input features

	DINOv3-L	RADIOv3-L	Pascal VOC (mIoU $\uparrow$ )
❶			76.42
❷	✓		77.18 (+0.76)
❸		✓	77.57 (+1.15)
❹	✓	✓	77.84 (+1.42)

Table 6: Ablation experiments on applying different VFMs in DiveUp using frozen CLIP as

F_{lr}

	VFM Fusion	Pascal VOC (mIoU $\uparrow$ )
❶	Addition	84.51
❷	Concatenation	84.66
❸	SA selection (ours)	84.92

Table 7: Ablation experiments on VFM fusion strategies in DiveUp using frozen DINOv2-S as

F_{lr}

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Target VFM	Semantic Segmentation (mIoU %)				Absolute Gains
Target VFM	NAF [ 2 ]	AnyUp-S [ 42 ]	AnyUp-M [ 42 ]	DiveUp (Ours)	vs. NAF	vs. AnyUp-M
ResNet-50 [ 14 ]	51.44	54.60	56.85	57.08	+5.64	+0.23
PE-Core-S [ 1 ]	72.35	73.69	73.96	75.31	+2.96	+1.35
DFN-CLIP-B [ 11 ]	77.66	77.62	78.11	78.81	+1.15	+0.70
PaliGemma2 [ 36 ]	79.58	79.64	80.24	81.33	+1.75	+1.09
MaskCLIP-B [ 45 ]	73.37	73.75	75.14	75.57	+2.20	+0.43

Appendix

Table 8: Zero-Shot Generalization Across Diverse and Noisy VFMs. Performance evaluated on the PASCAL VOC segmentation benchmark using mIoU (%) without per-model retraining. DiveUp shows robustness and architecture-agnostic generalization, outperforming baselines across all evaluated noisy backbones. The best and the second best results are highlighted in bold and underlined , respectively.

Target VFM	Semantic Segmentation (mIoU %)		Absolute Gain
Target VFM	NAF (Baseline)	NAF w/ DiveUp	Absolute Gain
PE-Core-S	74.37	75.91	+1.54
DFN-CLIP-B	78.04	78.78	+0.74
MaskCLIP-B	73.99	75.16	+1.17
ClearCLIP-B	70.98	73.70	+2.72

Appendix

Table 9: Plug-and-Play Robustness on Noisy Backbones. Performance (mIoU %) evaluated on the PASCAL VOC segmentation task. By upgrading the standard NAF baseline with our multi-VFM relational guidance (NAF w/ DiveUp), the upsampling performance is consistently improved.

Window Size ( $w$ )	Semantic Segmentation
Window Size ( $w$ )	mIoU (%)	Acc (%)
$3\times 3$	78.94	94.70
$5\times 5$	79.55	94.88
$7\times 7$ (Default)	79.60	94.90
$9\times 9$	79.43	94.84
$11\times 11$	79.29	94.80

Appendix

Table 10: Ablation on Local Window Size (

w\times w

). Performance evaluated on PASCAL VOC using the SigLIP-B backbone. The upsampling performance peaks at

w=7

, which balances spatial context with the exclusion of distant noise.

Figure 5: Visualization of Relational Representations (Before vs. After DiveUp). We visualize the intermediate local entropy maps and COM fields generated by a noisy backbone (ClearClip-B), compared with those generated after integrating DiveUp’s multi-VFM relational guidance. DiveUp suppresses background interference and spatial uncertainty, achieving more localized geometric structures. This improved location alignment corresponds to the quality of the final segmentation results (see IoU gains).

Figure 6: Qualitative Comparison for Semantic Segmentation. Results are produced using the SigLIP-B backbone on the PASCAL VOC dataset. Compared to other VFM-agnostic and VFM-specific upsamplers, DiveUp eliminates semantic bleeding and generates sharper object boundaries, reflecting the intrinsic geometric structures.

Figure 7: Qualitative Comparison for Dense Depth Estimation. Results are produced using the DINOv2-S backbone on the NYUv2 dataset. DiveUp preserves depth discontinuities and complex object boundaries.

Explore similar work

PixelUp: Zero-Shot Semantic Feature Upsampling for Fine-Grained Vision Tasks
Aug 3, 2026Deepank Singh, Anurag Nihal, Vedhus HoskereDepth Estimation Vision Foundation Model Adaptation
University of Houston
RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation
Jun 22, 2026Yuchuan Ding, Linfei Li, Lin Zhang +1Dense Prediction
School of Computer Science and Technology, Tongji University, Shanghai, China
ViT-Up: Faithful Feature Upsampling for Vision Transformers
Jun 12, 2026Krispin Wandel, Jingchuan Wang, Hesheng WangSemantic Correspondence Vision Transformer
Shanghai Jiao Tong University, Shanghai, China

Aug 3, 2026J/K move · Enter open · S save

PixelUp: Zero-Shot Semantic Feature Upsampling for Fine-Grained Vision Tasks

Deepank Singh, Anurag Nihal, Vedhus Hoskere

University of Houston

Depth Estimation Vision Foundation Model Adaptation Semantic Segmentation Unsupervised Image Segmentation

Open paper AskPDF arXiv

Self-supervised Vision Foundation Models (VFMs) have become essential backbones for downstream tasks due to their strong and transferable visual representations. However, their patch-token-level features are often too coarse for dense prediction tasks such as semantic segmentation and depth estimation when accurate fine-grained predictions are required. Feature upsampling methods have been developed to recover pixel-level detail but still face limitations. Learnable upsamplers are often designed for a specific encoders and must be retrained for different encoders. Image-guided methods that use shallow pixel encoders often introduce textural artifacts and lack the semantic guidance needed for accurate downstream predictions. We introduce PixelUp, a zero-shot VFM-agnostic upsampler achieving semantic awareness through a coarse-to-fine chain of windowed cross-attention architecture guided by multi-scale semantic features. We demonstrate that PixelUp outperforms both VFM-specific and VFM-agnostic upsamplers, achieving state-of-the-art performance on dense prediction tasks with an average improvement of +1.2 mIoU on semantic segmentation and +0.25

δ_1

, on NYUv2 depth estimation across VFMs. PixelUp further improves training-free open-vocabulary and unsupervised semantic segmentation by an average of +1.3 mIoU and +0.5 mIoU, respectively. Code available at https://pixelup-project.vercel.app/

Abstract

Explore similar work

PixelUp: Zero-Shot Semantic Feature Upsampling for Fine-Grained Vision Tasks

RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation

ViT-Up: Faithful Feature Upsampling for Vision Transformers