Vision foundation models produce semantically strong features on a coarse patch grid, and dense prediction needs them at pixel resolution. Existing learned upsamplers close that gap by regressing the high-resolution feature map from the low-resolution one, typically through cross-attention driven by the input image. We observe that most of what they regress needs no learning: plain bilinear interpolation already reproduces the smooth regions of the target, and recent benchmarks even report bilinear and bicubic on par with several learned upsamplers. The rest is mostly edges and fine details, which suggests that these learned upsamplers spend much of their capacity re-computing the smooth content that classical interpolation provides. We therefore introduce AnchorUp, which stops predicting the high-resolution feature map and predicts only its correction from the interpolation. It keeps a parameter-free classical upsampler as an anchor, obtains a second prediction from a small guidance network, and blends the two through a single learnable scalar gate. Because the anchor supplies the smooth content, the guidance network only has to correct what remains. An analysis of the learned correction confirms that it concentrates on edges and fine details, and so the network can be small: AnchorUp uses 167,425 parameters, a quarter of the budget of the strongest baseline, NAF, and runs at 64% of its latency. Trained once on DINOv3 with only the first of NAF's two training stages, and evaluated on that backbone plus six held-out ones, it is already on par with NAF trained through both stages on Pascal VOC and Cityscapes. Our method surpasses it on DAVIS-2017 video object segmentation; the gain holds on every backbone and is larger in contour accuracy F. Code is available at https://github.com/KotaroHorikawa/AnchorUp.
Figures & tables
Figure 1: Accuracy against latency on Cityscapes averaged over seven backbones. Circle size shows the number of parameters (Tables 2 and 4 ). Top: AnchorUp achieves a better accuracy-speed trade-off than baselines: NAF. Bottom: Because our learned part supplies only a correction to a classical interpolation rather than the whole feature map, it focuses solely on missing details (outlines, whiskers, and fur) while remaining flat over the background. Feature maps are projected onto the first three principal components of each image; in the correction, zero appears as mid-gray.
Figure 2: AnchorUp. (a) The low-resolution feature map is upsampled in two ways: by a classical upsampler with no parameters, which gives the anchor, and by a NAF-style upsampler, a guidance network followed by neighborhood cross-attention. In the attention, queries come from the encoded image, keys from its average pooling, and values from the feature map itself. Only the guidance network and the gate α=σ(γ) have learnable parameters; the gate controls how far the output moves from the anchor toward the NAF-style prediction. (b) The output is a point on the segment between the two predictions: the network picks the direction, the gate picks the distance. The dashed line is the remaining error to the target.
Images only
Images + text
Distilled
Method
Anchor
d
#Params
Stage 2
DINOv3
Franca
CLIP
SigLIP2
PEcore
PEspatial
RADIO
Mean
Δ
(a) Pascal VOC
NAF [ 2 ]
–
256
662,528
✓
87.81
84.30
80.25
77.83
72.61
78.04
86.61
81.06
–
NAF
–
128
167,424
✓
87.83
84.21
79.98
77.40
72.16
78.16
86.51
80.89
− 0.17
NAF
–
128
167,424
87.34
84.24
80.53
78.85
73.62
77.60
86.55
81.24
+ 0.18
AnchorUp
bilinear
128
167,425
87.71
84.27
80.23
77.80
72.63
78.18
86.61
81.06
0.00
Table 1: Cross-backbone linear probing. All upsamplers are trained once on DINOv3-B/16 and applied unchanged to the other six backbones. Anchor is the interpolation used in Eq. 1 (“–” for NAF, which has none), d is the guidance dimension (Section 3.3 ), #Params the number of learnable parameters, and a check under Stage 2 marks training through the second stage of NAF’s recipe (Section 3.5 ). Rows are ordered so that each differs from the row above in a single factor; the third NAF row therefore differs from AnchorUp only by Eq. 1 . The best result per column is in bold, the second best is underlined; Δ is the difference in mean from the two-stage NAF row.
Images only
Images + text
Distilled
Method
DINOv3
Franca
CLIP
SigLIP2
PEcore
PEspatial
RADIO
Mean
Δ
(a) Pascal VOC
Bilinear
86.97
82.12
76.28
71.94
66.66
76.62
84.47
77.87
–
FeatUp [ 8 ]
86.60
81.68
77.29
72.17
66.28
74.28
82.95
77.32
− 0.54
JAFAR [ 4 ]
87.08
83.65
79.80
77.51
74.23
74.97
85.97
80.46
+ 2.59
AnyUp [ 29 ]
86.56
83.26
79.76
78.56
74.09
76.77
85.63
80.66
+ 2.80
Table 2: Comparison with other feature upsamplers. Every method is trained here under its own published configuration. NAF and AnchorUp are trained once on DINOv3-B/16 and applied unchanged; FeatUp and JAFAR require one model per backbone (Section 4 ). The NAF and AnchorUp rows are the models of Table 1 . Best per column in bold, second best underlined; Δ is the difference in mean from bilinear.
Images only
Images + text
Distilled
Method
Anchor
d
Stage 2
DINOv3
Franca
CLIP
SigLIP2
PEspatial
RADIO
Mean
Δ
J -Mean ↑
NAF [ 2 ]
–
256
✓
66.00
68.70
62.80
62.20
67.00
66.40
65.52
–
NAF
–
128
✓
65.60
68.40
62.90
61.90
66.80
65.90
65.25
− 0.27
NAF
–
128
62.79
66.59
61.77
62.31
65.88
65.02
64.06
− 1.46
AnchorUp
bilinear
128
66.30
69.50
63.50
62.40
67.70
67.20
66.10
+ 0.58
Table 3: Semi-supervised video object segmentation on DAVIS-2017. Label propagation uses the upsampled features directly, without any training on video. Rows follow the order of Table 1 , each differing from the row above it in one factor, so the third NAF row differs from AnchorUp only by Eq. 1 . The third NAF row is trained using Stage 1 only, following the AnchorUp recipe. The gain is largest on F -Mean, the boundary-accuracy term (Section 5 shows the correction concentrates on boundaries). Best per column in bold, second best underlined; Δ is the difference in mean from the two-stage NAF row.
Method
Params ↓
Latency (ms) ↓
Peak mem. (MiB) ↓
Bilinear
0
0.30
297.4
Bicubic
0
0.84
297.4
FeatUp [ 8 ]
173,540
27.95
2674.0
JAFAR [ 4 ]
628,480
45.82
5410.9
AnyUp [ 29 ]
845,056
49.27
3253.8
NAF [ 2 ]
662,528
25.31
1685.5
Table 4: Cost of one upsampling pass for the upsampler alone, at 448×448 output from the 28×28 features of 384 channels that a 448×448 input produces, FP32 and batch 1 on an RTX PRO 6000 Blackwell. Latencies are means over 500 forwards and the standard deviation is below 0.07 ms for every row. AnchorUp is 1.6× faster than NAF, with a quarter of its parameters.
Method
VOC
Cityscapes
DAVIS
Control
81.36±0.10
59.44±0.70
65.32±1.05
AnchorUp
81.19±0.18
60.57±0.08
68.87±0.15
Difference
−0.16±0.09
+1.14±0.62
+3.54±0.92
Table 5: Comparison with the control over three training seeds. The control is the third NAF row of Tables 1 and 3 ( d=128 , stage 1 only), and AnchorUp uses the bilinear anchor. Each entry is the mean and standard deviation over seeds 0–2 of the mean across backbones (mIoU on VOC and Cityscapes, J&F on DAVIS); the difference is taken between models of the same seed. Seed 0 is the model from Tables 1 and 3 , with the linear probes retrained.
Figure 3: The correction is a boundary signal. Columns: input, low-resolution features, anchor Fanchor , learned estimate Flearn , correction Cθ=α(Flearn−Fanchor) , and AnchorUp output F^HR . Feature maps are projected onto the first three principal components of each image and shown as RGB; in the correction column, zero appears as mid-gray. The correction is flat inside regions and concentrates on contours and on the fine repeated structure the anchor cannot resolve (rows 2 and 3). Added to the anchor (third column), this is the detail the output (last column) recovers.
Figure 4: The correction is high-pass. Radially averaged Fourier power of each term of Eq. 1 : the anchor Fanchor , the learned estimate Flearn , the correction Cθ=α(Flearn−Fanchor) , and the output F^HR . Two readings matter. At the lowest frequencies the correction carries about two orders of magnitude less power than the anchor ( 106 against 2×108 ): the smooth content of the output comes from the anchor, not from the network. Between roughly 0.06 and 0.3 cycles per pixel the correction carries more power than the anchor: the learned part spends its output where the interpolation cannot reach. The measurement protocol is in the supplementary material.
Anchor
learned α
α⋆
difference
Bilinear
0.620
0.551
+0.069
Bicubic
0.771
0.748
+0.023
Table S1: The learned gate against the gate the loss would choose. α⋆ is Eq. 3 of the main paper solved on 200 Cityscapes validation images with DINOv3 features, no labels used, each row using its own trained model. The gate is trained by the same loss as the network, with no schedule and no separate term.
Figure S1 : The error is flat around the learned gate α , and its minimum moves with the anchor. The loss of Section 3.4 of the main paper as a function of α , averaged over 200 Cityscapes validation images, with the guidance network of the bilinear-anchor model frozen and the anchor swapped. Dots mark minima; the vertical line marks the learned gate. The norm sums over all feature elements of an image, setting the scale of the axis.
Backbone
Checkpoint
Patch
DINOv3
vit_base_patch16_dinov3.lvd1689m
16
CLIP
16
vit_base_patch16_clip_384.
laion2b_ft_in12k_in1k
SigLIP2
vit_base_patch16_siglip_512.v2_webli
16
Franca
franca_vitb14
14
Table S2 : Frozen backbones. All are base-size vision transformers; Franca uses a patch size of 14 and the rest 16.
Setting
Value
Data
ImageNet, 512×512 HR targets
Degradation
fixed, ×2
Guidance resolution
min(224,4×HR feature) per axis
Loss
ℓ2 feature reconstruction, no auxiliary terms
Batch size
4
Optimizer
AdamW, β=(0.9,0.999)
Table S3: Training setup in full. The position encoding is worth stating explicitly because a later default in the same codebase disables the RoPE rescale: these results use rescale 2.
Images only
Images + text
Distilled
Method
DINOv3
Franca
CLIP
SigLIP2
PEcore
PEspatial
RADIO
Mean
(a) Pascal VOC
Control
86.93±0.35
84.02±0.19
80.70±0.15
79.50±0.57
74.50±0.76
77.42±0.16
86.45±0.09
81.36±0.10
AnchorUp
87.58±0.13
84.32±0.02
80.47±0.20
78.38±0.56
73.26±0.64
77.68±0.02
86.67±0.02
81.19±0.18
Difference
+0.65±0.22
+0.30±0.18
−0.23±0.06
−1.12±0.01
−1.24±0.12
+0.26±0.17
+0.23±0.10
−0.16±0.09
(b) Cityscapes
Table S4 : Per-backbone values behind Table 5 of the main paper. Mean and standard deviation over training seeds 0–2. The control is the third NAF row of Tables 1 and 3 of the main paper ( d=128 , stage 1 only), and AnchorUp uses the bilinear anchor; the difference is taken between models of the same seed. DAVIS excludes PEcore, as in Table 3 of the main paper.
Backbone
Image–text
Δ VOC
Δ Cityscapes
PEcore
✓
+1.98
+0.42
SigLIP 2
✓
+1.73
+0.30
CLIP
✓
+0.55
−0.29
RADIO
+0.03
−1.12
PEspatial
−0.03
−0.79
Franca
−0.07
−0.51
Table S5: Bicubic anchor minus bilinear anchor, by backbone. A check under Image–text marks the backbones pretrained with image–text supervision. Rows are ordered by the VOC gain. The three image–text backbones take the top three places on both benchmarks and the four others take the bottom four, with no overlap. One caveat for the PEcore–PEspatial reading of Section 4.1 of the main paper: the two checkpoints differ in nominal resolution as well, 224 and 512.
Anchor
d
α
DINOv3
CLIP
SigLIP 2
PEcore
Mean
#Params (M)
Bilinear
32
0.579
65.66
56.01
57.19
54.06
58.23
0.011
64
0.607
66.09
56.53
57.81
54.69
58.78
0.043
128
0.620
66.45
57.07
58.28
55.25
59.26
0.167
256
0.630
66.49
57.09
58.53
55.32
59.36
0.663
Bicubic
32
0.772
65.38
56.03
57.40
54.30
58.28
0.011
64
0.771
65.74
56.42
57.87
54.78
58.70
0.043
Table S6: Guidance dimension and anchor. AnchorUp trained with the recipe of Section 3.5 of the main paper at four guidance dimensions for both anchors. Cityscapes mIoU is evaluated by linear probing; the mean is calculated over the four backbones shown, not the seven in Table 1 of the main paper. The d=128 rows correspond to the models in that table, whose costs are reported in Table 4 of the main paper. α is the trained gate.
Training steps
Benchmark
25,000
100,000
Difference
VOC mIoU
81.06
81.08
+0.02
Cityscapes mIoU
60.77
60.94
+0.17
DAVIS J&F
65.90
66.29
+0.39
Table S7: Training length. AnchorUp with a bilinear anchor, mean over the seven backbones of Table 1 of the main paper. The DAVIS row therefore includes PEcore and is not comparable to Table 3 of the main paper, which averages over six. The main paper reports the 25,000-step checkpoint throughout.
Figure S2 : Where the extra training goes. Radially averaged power of the AnchorUp prediction with a bilinear anchor, at six checkpoints of the same run. The band below 0.05 cycles per feature pixel is settled at the first checkpoint and moves by less than 0.5% thereafter; all of the growth is in the shaded band above 0.25 , which gains 29% between 25,000 and 100,000 steps and 56% between 25,000 and 300,000.
Figure S3 : The correction split at the crossings of Figure 4 of the main paper. Rows are the three scenes of its Figure 3. The middle band, where the correction carries more power than the anchor, holds the fine contour lines; the band below holds the broad response around the same boundaries; the band above holds almost nothing.
Figure S4 : Decomposition, all ten images. Columns as in Figure 3 of the main paper, which shows the first three rows.
Figure S5 : Decomposition on held-out images, rows 1–10 of 30. ImageNet validation images, unseen during training; columns and projection as in Figure S4 .
Figure S6 : Decomposition on held-out images, rows 11–20 of 30. Columns and projection as in Figure S4 .
Figure S7 : Decomposition on held-out images, rows 21–30 of 30. Columns and projection as in Figure S4 .
Self-supervised Vision Foundation Models (VFMs) have become essential backbones for downstream tasks due to their strong and transferable visual representations. However, their patch-token-level features are often too coarse for dense prediction tasks such as semantic segmentation and depth estimation when accurate fine-grained predictions are required. Feature upsampling methods have been developed to recover pixel-level detail but still face limitations. Learnable upsamplers are often designed for a specific encoders and must be retrained for different encoders. Image-guided methods that use shallow pixel encoders often introduce textural artifacts and lack the semantic guidance needed for accurate downstream predictions. We introduce PixelUp, a zero-shot VFM-agnostic upsampler achieving semantic awareness through a coarse-to-fine chain of windowed cross-attention architecture guided by multi-scale semantic features. We demonstrate that PixelUp outperforms both VFM-specific and VFM-agnostic upsamplers, achieving state-of-the-art performance on dense prediction tasks with an average improvement of +1.2 mIoU on semantic segmentation and +0.25 δ1, on NYUv2 depth estimation across VFMs. PixelUp further improves training-free open-vocabulary and unsupervised semantic segmentation by an average of +1.3 mIoU and +0.5 mIoU, respectively. Code available at https://pixelup-project.vercel.app/
Recently, feature upsampling has gained increasing attention owing to its effectiveness in enhancing vision foundation models (VFMs) for pixel-level understanding tasks. Existing methods typically rely on high-resolution features from the same foundation model to achieve upsampling via self-reconstruction. However, relying solely on intra-model features forces the upsampler to overfit to the source model's inherent location misalignment and high-norm artifacts. To address this fundamental limitation, we propose DiveUp, a novel framework that breaks away from single-model dependency by introducing multi-VFM relational guidance. Instead of naive feature fusion, DiveUp leverages diverse VFMs as a panel of experts, utilizing their structural consensus to regularize the upsampler's learning process, effectively preventing the propagation of inaccurate spatial structures from the source model. To reconcile the unaligned feature spaces across different VFMs, we propose a universal relational feature representation, formulated as a local center-of-mass (COM) field, that extracts intrinsic geometric structures, enabling seamless cross-model interaction. Furthermore, we introduce a spikiness-aware selection strategy that evaluates the spatial reliability of each VFM, effectively filtering out high-norm artifacts to aggregate guidance from only the most reliable expert at each local region. DiveUp is a unified, encoder-agnostic framework; a jointly-trained model can universally upsample features from diverse VFMs without requiring per-model retraining. Extensive experiments demonstrate that DiveUp achieves state-of-the-art performance across various downstream dense prediction tasks, validating the efficacy of multi-expert relational guidance. Our code and models are available at: https://github.com/Xiaoqiong-Liu/DiveUp
Xiaoqiong Liu, Heng Fan
Department of Computer Science and Engineering, University of North Texas
Vision Transformers (ViTs) have become a dominant architecture for visual representation learning, providing exceptionally strong and broadly reusable backbone features. However, ViTs are commonly operated on relatively small patch-token grids due to the quadratic cost of global self-attention, which creates a persistent bottleneck for dense prediction tasks such as semantic segmentation and depth estimation. This has motivated the development of task-agnostic feature upsamplers. While recent state-of-the-art methods produce visually sharp dense representations, their reliance on shallow image encoders for guided upsampling can introduce feature leakage, fragmentation, and blur. We introduce ViT-Up, an implicit feature upsampling framework that replaces external image guidance with layer-wise query construction from intermediate ViT hidden states. This enables feature prediction at arbitrary continuous image coordinates while preserving alignment with the backbone feature space. Experiments demonstrate that ViT-Up consistently outperforms state-of-the-art image-guided upsamplers across dense prediction and semantic correspondence. On DINOv3-S+, ViT-Up improves over prior methods by up to +2.07 mIoU on Cityscapes and +4.17 [email protected] on SPair-71k. With the larger DINOv3-B backbone, these gains increase to +3.36 mIoU and +8.09 [email protected], demonstrating that ViT-Up scales favorably with backbone capacity.