What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers

Moritz Pawlowsky; Antonis Vamvakeros; Alexander Weiss; Anja Bielefeld; Samuel J. Cooper; Ronan Docherty

cs.CVMar 17, 2026

What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers

Authors: Moritz Pawlowsky, Antonis Vamvakeros, Alexander Weiss, Anja Bielefeld, Samuel J. Cooper, Ronan Docherty

Organizations: Center for Materials Research, Justus Liebig University, 35392 Giessen · Dyson School of Design Engineering, Imperial College London, London · Finden ltd, Building R71, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus · Department of Materials, Imperial College London, London

Abstract

Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich representations useful for many downstream tasks. However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content. This makes zero-shot adaption difficult in fields like material science, where images are often cross-sections of homogeneous microstructure (i.e. having no preferred direction). In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images.

Figures & tables

Figure 2: Linear probe analysis of DINOv2-S features. (a) We train linear probes to map from image features (or individual channels) to randomly sampled (red squares) ramp functions, reporting $R^{2}$ scores on holdout regions. Per-channel scores and predictions (which use all channels) are both averaged over a dataset of 15 homogenous microscopy images, i.e. images with no preferred direction. (b) The channels with the highest positional $R^{2}$ scores for a series of images, including a satellite image and an electron microscope image of a nickel superalloy [ 48 ] .

Figure 3: Per-channel per-layer ‘positional fingerprint’ of $R^{2}$ scores for DINOv2, DINOv3 and ALiBi-Dv2 for a left-right target ramp. DINOv2 begins with positional information spread across channels (its learned PE is added at the start of the network), which later decreases, whereas for DINOv3 the channels become more positional with layer depth (RoPE is applied at each layer). ALiBi-Dv2 has less positional information present across its channels and layers.

Model	d	PE	Supervised	$R^{2}_{\text{micro}}$	$R^{2}_{\text{texture}}$	$R^{2}_{\text{noise}}$
DINO	S	Learned	$\times$	0.57	0.53	0.53
DINO	B	Learned	$\times$	0.64	0.59	0.54
DINOv2	S	Learned	$\times$	0.83	0.71	0.89
DINOv2	B	Learned	$\times$	0.69	0.58	0.89
DINOv3	S+	RoPE	$\times$	0.97	0.90	0.98
ViT (MAE)	B	Learned	$\times$	0.71	0.61	0.71

Table 1:

(x,y)

linear probe

R^{2}

scores for a set of existing models which produce patch-level features. Supervised models - including those with an SSL backbone - have lower

R^{2}

scores. Scores are reported per-dataset: ‘micro’ for micrographs, ‘texture’ for DTD [ 49 ] texture images and ‘noise’ for random noise.

Model	d	PE	$R^{2}_{\text{micro}}$	$R^{2}_{\text{texture}}$	$R^{2}_{\text{noise}}$
DINOv2	S	Learned	0.83	0.71	0.89
DINOv2	B	Learned	0.68	0.58	0.89
DINOv2 (CB)	S	Learned	0.78	0.67	0.89
DVT	S	Learned	0.75	0.68	0.83
ALiBi-Dv2	S	ALiBI	-0.23	0.32	-0.85
ALiBi-Dv3	S+	ALiBI	-0.51	0.16	-0.93

Table 2:

(x,y)

linear probe

R^{2}

scores for DINOv2 variations. ‘CB’ refers to zeroing the 4 most positional channels of DINOv2 (see Fig. 2 ). Our ALiBi model has more homogeneous features and therefore has a lower

R^{2}

Figure 4: Training of a toy attention layer for a series of positional encodings: fixed sinuoidal, learned, RoPE (1D and 2D), ALiBi and NoPE. The goal is to test whether the layer can learn a mapping from a fixed random input to an $(x,y)$ grid using its positional encoding and weights. Absolute encodings (sinusoidal, learned) can, as can both RoPE variants. ALiBi and NoPE, by contrast, cannot - this makes them good candidates for post-hoc finetuning.

Figure 5: Feature PCA comparisons for DINOv2-S, DVT and our ALiBi-Dv2. ALiBi-Dv2 produces features which are semantically rich but that display less positional bias and artefacts. The token geometry is (locally) smooth, and in many cases similar to that of DVT. Of note is the ‘square-circle’ image, which retains ‘objectness’ across indistinguishable interior patches, the preserved vertical bias of the dog image (from depth-of-field) and the reduced positional biases on the X-ray CT cross-section of a pintype Li-ion battery [ 53 ] .

Figure 6: Feature PCA visualisations for DINOv2, DINOv3 and our ALiBi-Dv2 & ALiBi-Dv3. We retain desirable object decompositions (e.g. head vs body of the dogs), whilst having less positional features for the satellite image and the SEM images of a battery cathode [ 53 ] & biphase steel [ 54 ] .

Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [ 53 ] .

Figure 8: k-means clustering of features of DINOv2-S, DVT, DINOv3, and our ALiBi models. For some images, like the SEM micrograph of a eutectic alloy [ 56 ] , both DINOv2 and DVT decompose using positional information (radial or vertical), whereas the ALiBi variants produce clean, homogeneous results.

Model	VOC07	VOC12	ADE20k
DINOv2	0.680	0.762	0.240
NoPE	0.689	0.757	0.235
ALiBi-Dv2	0.692	0.769	0.239

Table 3: Validation mIoU (mean Intersection over Union) for frozen-feature linear probe semantic segmentation of benchmark datasets. We see our ALiBi-Dv2 performs similarly to, and in some cases outperforms, DINOv2.

Figure 9: Trainable segmentation of images: a set of features that describe each pixel (classical + bilinearly upsampled ViT features) are extracted, and a lightweight classifier (XGBoost) is trained to map from feature vectors to user labels. Our ALiBi-Dv2 has more homogenous features and so produces less positionally biased (and therefore higher quality) segmentations of complex microstructures. Segmentations using DINOv2 and DVT’s features are biased either radially, horizontally or vertically.

Figure S1: (a) Micrographs used for the linear probe. (b) Random samples from the DTD used for the linear probe [ 49 ] .

Figure S2: Comparison of sampling strategies. For each method (grid, holdout, random), the target ramps (left) and the mean model predictions across all channels (right) are shown. Red markers denote sampled training locations.

Figure S3: Linear probe analysis of DINOv2, DINOv3 and ALiBi-Dv2 with a target ramp of random noise, using the micrograph dataset.

Figure S4: Linear probe analysis of ViT-B (MAE) features over the micrograph dataset. Again we see clear ramp channels in the output features.

Figure S5: Linear probe analysis of our ALiBi-Dv2 features over the micrograph dataset.

Figure S6: Per-channel per-layer $R^{2}$ scores for an $(x,y)$ ramp for a series of models. We see self-supervised models (DINO-B, DINOv2-B, DINOv3-B, ViT-B) have stronger and more widespread positional channels.

ALiBi-Dv2	low resolution	high resolution
image size	(224, 224)	(518, 518)
learning rate	1e-4	1e-5
batch size	256	32
num epochs	15	5
optimizer	AdamW ( $\lambda=0.01$ )	AdamW ( $\lambda=0.01$ )
loss	cosine similarity	cosine similarity

Table S1: Training hyperparameters for ALiBi-Dv2 in the low- and high-resolution training setups.

Figure S7: Comparing the impact of learned per-head ALiBi scalars m (top row) and constant m $1$ (bottom row). We see learning the head scalars can lead to undesirable long-range positional biases.

Figure S8: PCA visualizations of patch embeddings demonstrate that models trained with multiscale augmentation (ALiBi-Dv2, NoPE) successfully distinguish semantic regions ( e.g. separating the head from the body) across varying input resolutions. Models lacking multiscale training (-no-ms) fail to separate these features, highlighting the necessity of multiscale training for adapting to varying feature scales.

Figure S9: Feature PCAs with different finetuned positional encodings. When using sinusoidal PE, positional biases re-emerge during training (even with channel-blanking); NoPE has no biases (by definition), but fails to model useful spatial relationships for similar patches (see the shoulder of the cat). ALiBi-Dv2 avoids these problems.

Figure S10: PCA of very low information images for (a) ALiBi-Dv2 and (b) DINOv2. with introduced random noise into the respective positional encodings.

Figure S11: PCA features for a series of models as a function of input size resolution. DINOv2 and DINOv3 retain semantics as the image is resized at all resolutions. ALiBi-Dv2 handles resizing well, but loses semantic coherence at $(1246,1246)$ i.e. around 2.5 times the largest image seen during training.

Figure S12: Feature PCAs with feature scaling before PCA. Although the positional biases are less strongly expressed in the visualisation, they are still present, and can still be fit to or clustered around during downstream tasks.

Figure S13: Shared feature PCAs for DINOv2 and ALiBi-Dv2. We see ALiBi-Dv2 retains DINOv2’s feature similarities across changes in style, pose, etc.

	VOC07	VOC12	ADE20K
image size	(518, 518)	(518, 518)	(518, 518)
learning rate	1e-3	1e-3	1e-3
batch size	64	64	32
num epochs	75	50	50
samples per epoch	209	1,4k	1k
optimizer	AdamW( $\lambda=0.01$ )	AdamW( $\lambda=0.01$ )	AdamW( $\lambda=0.01$ )

Table S2: Training hyperparameters of the linear probe setup

Model	none	flip-ud	roll	rot90
DINOv2	0.762	0.4759	0.7601	0.456
NoPE	0.757	0.482	0.757	0.436
ALiBi-Dv2	0.769	0.499	0.769	0.445

Table S3: Validation mIoU for frozen-feature linear probe semantic segmentation of VOC12 as a function of image transform. ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE.

Figure S14: Visual comparison of linear head segmentation predictions using various backbones (DINOv2, NoPE, ALiBi) evaluated on the VOC07, VOC12, and ADE20K benchmarks

Figure S15: Further examples of trainable segmentation of battery materials using different ViT features. Data from Refs. [ 53 ] , [ 47 ] , [ 66 ] and [ 53 ] , respectively.

Figure S16: Trainable segmentation quantitive benchmark. (a) For a set of 5 training images, N rounds of human-like labels were added. (b) The class-averaged mIoU over 17 test images (with no labels) as a function of number of labelling rounds. (c) Example predictions of XGBoost classifier trained to map features $\rightarrow$ labels after 5 labelling rounds for classical, classical + DINOv2 and classical + ALiBi-Dv2. ALiBi-Dv2 offers improved performance over DINOv2 with less labels due to the lack of positional bias in its features.

Explore similar work

Aug 4, 2026

When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings

We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval. Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.

Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt +2

Institute for Applied Informatics at Leipzig University (InfAI) · ScaDS.AI Dresden/Leipzig · Seltz +1

Language Model Pretraining Long-Context Retrieval Decoder-Only Language Models Positional Encoding Multi-Head Attention

Ask PDF

Jul 1, 2026

Active Spatial Guidance: Eliminating Injected Positional Mechanisms in Vision Transformers

Vision Transformers (ViTs) commonly rely on injected positional mechanisms to address self-attention's permutation invariance. Motivated by the spatial regularities of natural images, we ask whether spatial organization can be induced from data rather than explicitly injected. Under controlled, matched from-scratch training, we propose Active Spatial Guidance (Guidance), a training-only objective that disables positional injection and applies an auxiliary 2D coordinate-regression loss to the final-layer patch tokens. The guidance head is used only during training and removed for inference; the deployed model consists of a positional-injection-free ViT encoder and the task-specific prediction module. Using DINOv3 ViT backbones, Guidance consistently improves performance on ImageNet-100 classification, ADE20K semantic segmentation, and Hypersim monocular depth estimation, outperforming strong injected baselines such as learned absolute positional embeddings and rotary positional embeddings under identical training protocols. On ImageNet-100, broader comparisons against representative injected positional designs further support Guidance's effectiveness. Guidance also improves robustness under resolution transfer, and multi-resolution training further strengthens accuracy across input sizes. Overall, our results suggest that spatial inductive bias in ViTs need not be architecturally injected, but can be shaped through training-time supervision. The code used for training and evaluation is publicly available in https://github.com/cloudlc/asg.

Cong Liu, Xiaofang Li, Simon X. Yang

School of Engineering, Yancheng Institute of Technology, No. 1 Xiwang Middle Avenue, Yancheng, 224051, Jiangsu, PR China · School of Computer Information Engineering, Changzhou Institute of Technology, No. 666 Liaohe Road, Xinbei District, Changzhou, 213032, Jiangsu, PR China · Advanced Robotics and Intelligent Systems Laboratory, School of Engineering, University of Guelph, 50 Stone Road East, Guelph, N1G 2W1, Ontario, Canada

Vision Transformer

Ask PDF

May 22, 2026

Vision Transformers Need Better Token Interaction

Vision Transformers (ViTs) can learn strong image-level representations while their patch representations become less effective for dense prediction during prolonged training. We revisit this dense degradation phenomenon and argue that it is not fully explained by high-norm artifacts alone. Instead, we characterize semantic diffusion: an optimization shortcut in which global semantic information spreads through patch tokens beyond what is locally justified. Our analysis shows that dense representation quality is not captured by locality alone: shallow features can remain better aligned with foreground regions yet underperform deeper features, and [CLS] features remain complementary for dense prediction. These observations suggest that the goal should not be to remove global context, but to make token interactions more selective. We therefore study sparse attention as a minimal intervention, replacing softmax attention with entmax-1.5 while preserving global token connectivity. On DINOv1 ViT-S/16 trained for 200 epochs on ImageNet-1K, this change preserves ImageNet linear probing accuracy and substantially improves semantic segmentation performance: VOC mIoU increases from 42.80 to 48.78, ADE20K from 19.85 to 21.97, and Cityscapes from 36.79 to 37.87. These results suggest that selective token mixing is a simple and effective bias for improving dense ViT representations.

Linxiang Su

University of Szeged

Visual Representation Learning Vision Transformer Sparse Attention Semantic Segmentation

Ask PDF