Vision transformers (ViTs) serve as the backbone for countless vision tasks, including dense prediction tasks, such as segmentation and depth estimation, even though the quadratic complexity of self-attention severely limits feature resolution and induces spatial artifacts. In this work, we introduce ElasticDino, a lightweight adapter which produces high-resolution feature maps, adhering to the fine details of the input image. As a key insight, we observe the consistency of artifacts across channels, and simultaneously resolve them to match the input image structure. Using a slim architecture based on spatial deformations, ElasticDino produces pixel-aligned, high-resolution feature maps efficiently. Leveraging deformations allows strictly preserving sharp semantics, and thereby avoiding the smoothness biases inherent to alternative approaches. Due to the absence of ground truth, we employ pixel-level semantic segmentation and depth estimation with linear heads to predict a set of downstream dense tasks, guiding the deformation process without leaking task-specific semantics into the model. We demonstrate superior accuracy over current methods on a range of popular dense task benchmarks, with better computational performance.
Figures & tables
Figure 1: ElasticDino upscales ViT features and corrects their artifacts, allowing to efficiently train a variety of dense vision tasks.
Figure 2: Visualizations of features values across different channels. We observe artifacts are consistent across channels (top row). In addition, low frequency regions of the image emphasize positional artifacts (bottom left), and the low resolution nature of the features typically induce thickening of object edges (bottom right).
Figure 3: Method overview. The input image is passed to the (DINO) backbone for the low resolution, artifact-full features. These are successively passed to a sequence feature-fixing layers and then bilinearly upsampled by a factor 2. Feature Deformation Layers project and concatenate both the image and the semantic features, which are fed to a series of depthwise residual blocks, that end up generating a deformation field that warps the feature map. Task heads induce the sole losses of the system. The are simple MLPs that operate on the individual pixel level. Each head is compared to the ground truth using a task-specific loss.
ED (ours)
AnyUp
LoftUp
JAFAR
UPLift
Avg Time (ms)
108.41
545.42
418.89
177.79
147.27
Peak mem (GB)
0.71
1.94
8.31
7.33
9.22
#Params (K)
27,095
878
4,303
628
789
Table 1: Forward pass performance measures of ElasticDino and competing feature upscaling works over DINOv2. Note that even though ElasticDino trains more parameters, it is faster and more memory efficient, due to the linear (as opposed to attention-based) nature of our operations
Model
Cscapes
COCO
ADE20k
Oneformer-sw-L
0.6412
0.6264
0.5109
Mask2Former-sw-L
0.5986
0.6239
0.4928
ElasticDino-L
0.6875
0.6270
0.5299
Table 2: Validation Mean IoU for semantic segmentation benchmarks. Note that results differ from the comparison benchmarks, as we use a convolutional head (rather than linear) and train for longer than specified in the JAFAR benchmark.
Figure 4: Qualitative evaluation. Left: The top three PCA components of the feature maps of input images are depicted, as generated originally by DINOv2 (Low-Res), previous approaches, and us. Top Right: Semantic segmentation maps of input images, as generated by humans (GT), three previous methods, and us are also depicted. Bottom right: depth maps for the same approaches and input setting are depicted as well. Throughout all results, ElasticDino depicts sharper edges that respect the input image resolution and mitigates semantic bleeding between regions, even when RGB colors are similar.
Figure 5: Ablating deformations weakens the preservation of original features by allowing the encoding of the task specific values in the output, instead of only blending them. From left to right: input image, low-resolution DINOv2 features, the result without deformations.
Figure 8
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Method
VOC
ADE20K
mIoU( ↑ )
Acc( ↑ )
mIoU( ↑ )
Acc( ↑ )
Bilinear
51.25
68.18
9.05
25.15
JAFAR
63.99
77.91
10.69
28.06
AnyUp
57.59
73.40
10.71
28.95
LoftUp
38.73
55.29
7.64
32.16
ElasticDino (ours)
58.43
73.89
10.75
28.28
Appendix
Table 1: Zero-Shot Open-Vocabulary segmentation on VOC and Ade20k datasets following the methodology used in JAFAR [ 5 ] and introduced in MaskCLIP [ 18 ] paper. Competitive results indicating good alignment with original internal CLIP features.
JAFAR
FeatUp
AnyUp
LoftUp
Bilinear
Ours
10−3MSE
5.3
11.5
4.2
64.0
3.2
4.5
Appendix
Table 10
Figure 1: Qualitative comparison of ElasticDino to dedicated segmentation approaches (Mask2Former [ 2 ] ,OneFormer [ 8 ] ) over the cityScapes dataset.
Figure 2: Intrinsic decomposition estimation trained on Hypersim [ 13 ] with a convolutional head at resolution 128x128. For each image, the decomposition is shown on the right: top row is the ground truth and bottom row is the prediction from ElasticDino. Left to right: albedo, diffuse shading, surface normals. Even in poor light conditions, we obtain strong qualitative results.
Figure 3: Left: Accuracy vs size. We perform better for larger segments. Right: sizes across datasets. Cityscapes consists of smaller objects.
Figure 4: Visualization of deformations along the different resolution stages of the upscaling process and their effect on the features while being upscaled. Deformations are visualized using color wheel optical flow [ 1 ] , showing where each pixel was sampled from. Features are visualized through the first three PCA components. For each image, top row shows the feature maps and bottom row shows the cumulative deformations so far. Zoomed-in viewing recommended.
Figure 5: Visualization of deformations along the different resolution stages of the upscaling process and their effect on the features while being upscaled. Deformations are visualized using color wheel optical flow [ 1 ] , showing where each pixel was sampled from. Features are visualized through the first three PCA components. For each image, top row shows the feature maps and bottom row shows the cumulative deformations so far. Zoomed-in viewing recommended.
Figure 6: Additional qualitative comparisons through visualizations of the first three PCA components of the resulting features. From left to right: Input image, Original DINOv2 low-res features, JAFAR [ 5 ] , Anyup [ 16 ] , LoftUp [ 7 ] , UPLift [ 15 ] , ours. Zoomed-in viewing recommended.
Figure 7: Additional qualitative comparisons through visualizations of the first three PCA components of the resulting features. From left to right: Input image, Original DINOv2 low-res features, JAFAR [ 5 ] , Anyup [ 16 ] , LoftUp [ 7 ] , UPLift [ 15 ] , ours. Zoomed-in viewing recommended.
Figure 8: Qualitative evaluation of our auxiliary training tasks. During training, our encoder is trained various tasks, each with an individual head and the minimal receptive field of one pixel. Their results are demonstrated here. Top to bottom: PCA of the upscaled features, RGB reconstruction, depth estimation, semantic segmentation, and four tasks of intrinsic decomposition (shading, albeo, residuals, normals). For each image, we show (left) its GT labels (or the features before upscaling for the PCA), intermediate results of our heads after upscaling to size 64×64 (middle), and the final results of size 256×256 (right). Zoomed-in viewing recommended.
Figure 9: Additional qualitative comparisons for the segmentation task over the COCO dataset. From left to right: Input image, JAFAR [ 5 ] , Anyup [ 16 ] , LoftUp [ 7 ] , UPLift [ 15 ] , ours, Ground Truth. Zoomed-in viewing recommended.
Figure 10: Additional qualitative comparisons for the segmentation task over the ADE20K dataset. From left to right: Input image, JAFAR [ 5 ] , Anyup [ 16 ] , LoftUp [ 7 ] , UPLift [ 15 ] , ours, Ground Truth. Zoomed-in viewing recommended.
Figure 11: Additional qualitative comparisons for the segmentation task over the VOC dataset. From left to right: Input image, JAFAR [ 5 ] , Anyup [ 16 ] , LoftUp [ 7 ] , UPLift [ 15 ] , ours, Ground Truth. Zoomed-in viewing recommended.
Figure 12: Additional qualitative comparisons for the segmentation task over the CityScapes dataset. From left to right: Input image, JAFAR [ 5 ] , Anyup [ 16 ] , LoftUp [ 7 ] , UPLift [ 15 ] , ours, Ground Truth. Zoomed-in viewing recommended.
Figure 13: Additional qualitative comparisons for the depth estimation task over the COCO dataset. From left to right: Input image, JAFAR [ 5 ] , Anyup [ 16 ] , LoftUp [ 7 ] , UPLift [ 15 ] , ours, Ground Truth. Zoomed-in viewing recommended.
Vision foundation model adaptation tailors pretrained image and video models to new tasks and domains while balancing accuracy, efficiency, and generalization.
Vision Transformers (ViTs) have become a dominant architecture for visual representation learning, providing exceptionally strong and broadly reusable backbone features. However, ViTs are commonly operated on relatively small patch-token grids due to the quadratic cost of global self-attention, which creates a persistent bottleneck for dense prediction tasks such as semantic segmentation and depth estimation. This has motivated the development of task-agnostic feature upsamplers. While recent state-of-the-art methods produce visually sharp dense representations, their reliance on shallow image encoders for guided upsampling can introduce feature leakage, fragmentation, and blur. We introduce ViT-Up, an implicit feature upsampling framework that replaces external image guidance with layer-wise query construction from intermediate ViT hidden states. This enables feature prediction at arbitrary continuous image coordinates while preserving alignment with the backbone feature space. Experiments demonstrate that ViT-Up consistently outperforms state-of-the-art image-guided upsamplers across dense prediction and semantic correspondence. On DINOv3-S+, ViT-Up improves over prior methods by up to +2.07 mIoU on Cityscapes and +4.17 [email protected] on SPair-71k. With the larger DINOv3-B backbone, these gains increase to +3.36 mIoU and +8.09 [email protected], demonstrating that ViT-Up scales favorably with backbone capacity.
Vision Transformers (ViTs) can learn strong image-level representations while their patch representations become less effective for dense prediction during prolonged training. We revisit this dense degradation phenomenon and argue that it is not fully explained by high-norm artifacts alone. Instead, we characterize semantic diffusion: an optimization shortcut in which global semantic information spreads through patch tokens beyond what is locally justified. Our analysis shows that dense representation quality is not captured by locality alone: shallow features can remain better aligned with foreground regions yet underperform deeper features, and [CLS] features remain complementary for dense prediction. These observations suggest that the goal should not be to remove global context, but to make token interactions more selective. We therefore study sparse attention as a minimal intervention, replacing softmax attention with entmax-1.5 while preserving global token connectivity. On DINOv1 ViT-S/16 trained for 200 epochs on ImageNet-1K, this change preserves ImageNet linear probing accuracy and substantially improves semantic segmentation performance: VOC mIoU increases from 42.80 to 48.78, ADE20K from 19.85 to 21.97, and Cityscapes from 36.79 to 37.87. These results suggest that selective token mixing is a simple and effective bias for improving dense ViT representations.
Self-supervised Vision Foundation Models (VFMs) have become essential backbones for downstream tasks due to their strong and transferable visual representations. However, their patch-token-level features are often too coarse for dense prediction tasks such as semantic segmentation and depth estimation when accurate fine-grained predictions are required. Feature upsampling methods have been developed to recover pixel-level detail but still face limitations. Learnable upsamplers are often designed for a specific encoders and must be retrained for different encoders. Image-guided methods that use shallow pixel encoders often introduce textural artifacts and lack the semantic guidance needed for accurate downstream predictions. We introduce PixelUp, a zero-shot VFM-agnostic upsampler achieving semantic awareness through a coarse-to-fine chain of windowed cross-attention architecture guided by multi-scale semantic features. We demonstrate that PixelUp outperforms both VFM-specific and VFM-agnostic upsamplers, achieving state-of-the-art performance on dense prediction tasks with an average improvement of +1.2 mIoU on semantic segmentation and +0.25 δ1, on NYUv2 depth estimation across VFMs. PixelUp further improves training-free open-vocabulary and unsupervised semantic segmentation by an average of +1.3 mIoU and +0.5 mIoU, respectively. Code available at https://pixelup-project.vercel.app/