cs.CVMar 17, 2026
SaveWhat DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers
Organizations: Center for Materials Research, Justus Liebig University, 35392 Giessen · Dyson School of Design Engineering, Imperial College London, London · Finden ltd, Building R71, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus · Department of Materials, Imperial College London, London
Abstract
Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich representations useful for many downstream tasks. However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content. This makes zero-shot adaption difficult in fields like material science, where images are often cross-sections of homogeneous microstructure (i.e. having no preferred direction). In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images.
Figures & tables
Figure 2: Linear probe analysis of DINOv2-S features. (a) We train linear probes to map from image features (or individual channels) to randomly sampled (red squares) ramp functions, reporting scores on holdout regions. Per-channel scores and predictions (which use all channels) are both averaged over a dataset of 15 homogenous microscopy images, i.e. images with no preferred direction. (b) The channels with the highest positional scores for a series of images, including a satellite image and an electron microscope image of a nickel superalloy [ 48 ] .
Figure 3: Per-channel per-layer ‘positional fingerprint’ of scores for DINOv2, DINOv3 and ALiBi-Dv2 for a left-right target ramp. DINOv2 begins with positional information spread across channels (its learned PE is added at the start of the network), which later decreases, whereas for DINOv3 the channels become more positional with layer depth (RoPE is applied at each layer). ALiBi-Dv2 has less positional information present across its channels and layers.
| Model | d | PE | Supervised | |||
|---|---|---|---|---|---|---|
| DINO | S | Learned | 0.57 | 0.53 | 0.53 | |
| B | Learned | 0.64 | 0.59 | 0.54 | ||
| DINOv2 | S | Learned | 0.83 | 0.71 | 0.89 | |
| B | Learned | 0.69 | 0.58 | 0.89 | ||
| DINOv3 | S+ | RoPE | 0.97 | 0.90 | 0.98 | |
| ViT (MAE) | B | Learned | 0.71 | 0.61 | 0.71 |
Table 1: linear probe scores for a set of existing models which produce patch-level features. Supervised models - including those with an SSL backbone - have lower scores. Scores are reported per-dataset: ‘micro’ for micrographs, ‘texture’ for DTD [ 49 ] texture images and ‘noise’ for random noise.
| Model | d | PE | |||
|---|---|---|---|---|---|
| DINOv2 | S | Learned | 0.83 | 0.71 | 0.89 |
| B | Learned | 0.68 | 0.58 | 0.89 | |
| DINOv2 (CB) | S | Learned | 0.78 | 0.67 | 0.89 |
| DVT | S | Learned | 0.75 | 0.68 | 0.83 |
| ALiBi-Dv2 | S | ALiBI | -0.23 | 0.32 | -0.85 |
| ALiBi-Dv3 | S+ | ALiBI | -0.51 | 0.16 | -0.93 |
Table 2: linear probe scores for DINOv2 variations. ‘CB’ refers to zeroing the 4 most positional channels of DINOv2 (see Fig. 2 ). Our ALiBi model has more homogeneous features and therefore has a lower .
Figure 4: Training of a toy attention layer for a series of positional encodings: fixed sinuoidal, learned, RoPE (1D and 2D), ALiBi and NoPE. The goal is to test whether the layer can learn a mapping from a fixed random input to an grid using its positional encoding and weights. Absolute encodings (sinusoidal, learned) can, as can both RoPE variants. ALiBi and NoPE, by contrast, cannot - this makes them good candidates for post-hoc finetuning.
Figure 5: Feature PCA comparisons for DINOv2-S, DVT and our ALiBi-Dv2. ALiBi-Dv2 produces features which are semantically rich but that display less positional bias and artefacts. The token geometry is (locally) smooth, and in many cases similar to that of DVT. Of note is the ‘square-circle’ image, which retains ‘objectness’ across indistinguishable interior patches, the preserved vertical bias of the dog image (from depth-of-field) and the reduced positional biases on the X-ray CT cross-section of a pintype Li-ion battery [ 53 ] .
Figure 6: Feature PCA visualisations for DINOv2, DINOv3 and our ALiBi-Dv2 & ALiBi-Dv3. We retain desirable object decompositions (e.g. head vs body of the dogs), whilst having less positional features for the satellite image and the SEM images of a battery cathode [ 53 ] & biphase steel [ 54 ] .
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [ 53 ] .
Figure 8: k-means clustering of features of DINOv2-S, DVT, DINOv3, and our ALiBi models. For some images, like the SEM micrograph of a eutectic alloy [ 56 ] , both DINOv2 and DVT decompose using positional information (radial or vertical), whereas the ALiBi variants produce clean, homogeneous results.
| Model | VOC07 | VOC12 | ADE20k |
|---|---|---|---|
| DINOv2 | 0.680 | 0.762 | 0.240 |
| NoPE | 0.689 | 0.757 | 0.235 |
| ALiBi-Dv2 | 0.692 | 0.769 | 0.239 |
Table 3: Validation mIoU (mean Intersection over Union) for frozen-feature linear probe semantic segmentation of benchmark datasets. We see our ALiBi-Dv2 performs similarly to, and in some cases outperforms, DINOv2.
Figure 9: Trainable segmentation of images: a set of features that describe each pixel (classical + bilinearly upsampled ViT features) are extracted, and a lightweight classifier (XGBoost) is trained to map from feature vectors to user labels. Our ALiBi-Dv2 has more homogenous features and so produces less positionally biased (and therefore higher quality) segmentations of complex microstructures. Segmentations using DINOv2 and DVT’s features are biased either radially, horizontally or vertically.
Figure S1: (a) Micrographs used for the linear probe. (b) Random samples from the DTD used for the linear probe [ 49 ] .
Figure S2: Comparison of sampling strategies. For each method (grid, holdout, random), the target ramps (left) and the mean model predictions across all channels (right) are shown. Red markers denote sampled training locations.
Figure S3: Linear probe analysis of DINOv2, DINOv3 and ALiBi-Dv2 with a target ramp of random noise, using the micrograph dataset.
Figure S4: Linear probe analysis of ViT-B (MAE) features over the micrograph dataset. Again we see clear ramp channels in the output features.
Figure S5: Linear probe analysis of our ALiBi-Dv2 features over the micrograph dataset.
Figure S6: Per-channel per-layer scores for an ramp for a series of models. We see self-supervised models (DINO-B, DINOv2-B, DINOv3-B, ViT-B) have stronger and more widespread positional channels.
| ALiBi-Dv2 | low resolution | high resolution |
|---|---|---|
| image size | (224, 224) | (518, 518) |
| learning rate | 1e-4 | 1e-5 |
| batch size | 256 | 32 |
| num epochs | 15 | 5 |
| optimizer | AdamW ( ) | AdamW ( ) |
| loss | cosine similarity | cosine similarity |
Table S1: Training hyperparameters for ALiBi-Dv2 in the low- and high-resolution training setups.
Figure S7: Comparing the impact of learned per-head ALiBi scalars m (top row) and constant m (bottom row). We see learning the head scalars can lead to undesirable long-range positional biases.
Figure S8: PCA visualizations of patch embeddings demonstrate that models trained with multiscale augmentation (ALiBi-Dv2, NoPE) successfully distinguish semantic regions ( e.g. separating the head from the body) across varying input resolutions. Models lacking multiscale training (-no-ms) fail to separate these features, highlighting the necessity of multiscale training for adapting to varying feature scales.
Figure S9: Feature PCAs with different finetuned positional encodings. When using sinusoidal PE, positional biases re-emerge during training (even with channel-blanking); NoPE has no biases (by definition), but fails to model useful spatial relationships for similar patches (see the shoulder of the cat). ALiBi-Dv2 avoids these problems.
Figure S10: PCA of very low information images for (a) ALiBi-Dv2 and (b) DINOv2. with introduced random noise into the respective positional encodings.
Figure S11: PCA features for a series of models as a function of input size resolution. DINOv2 and DINOv3 retain semantics as the image is resized at all resolutions. ALiBi-Dv2 handles resizing well, but loses semantic coherence at i.e. around 2.5 times the largest image seen during training.
Figure S12: Feature PCAs with feature scaling before PCA. Although the positional biases are less strongly expressed in the visualisation, they are still present, and can still be fit to or clustered around during downstream tasks.
Figure S13: Shared feature PCAs for DINOv2 and ALiBi-Dv2. We see ALiBi-Dv2 retains DINOv2’s feature similarities across changes in style, pose, etc.
| VOC07 | VOC12 | ADE20K | |
| image size | (518, 518) | (518, 518) | (518, 518) |
| learning rate | 1e-3 | 1e-3 | 1e-3 |
| batch size | 64 | 64 | 32 |
| num epochs | 75 | 50 | 50 |
| samples per epoch | 209 | 1,4k | 1k |
| optimizer | AdamW( ) | AdamW( ) | AdamW( ) |
Table S2: Training hyperparameters of the linear probe setup
| Model | none | flip-ud | roll | rot90 |
|---|---|---|---|---|
| DINOv2 | 0.762 | 0.4759 | 0.7601 | 0.456 |
| NoPE | 0.757 | 0.482 | 0.757 | 0.436 |
| ALiBi-Dv2 | 0.769 | 0.499 | 0.769 | 0.445 |
Table S3: Validation mIoU for frozen-feature linear probe semantic segmentation of VOC12 as a function of image transform. ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE.
Figure S14: Visual comparison of linear head segmentation predictions using various backbones (DINOv2, NoPE, ALiBi) evaluated on the VOC07, VOC12, and ADE20K benchmarks
Figure S15: Further examples of trainable segmentation of battery materials using different ViT features. Data from Refs. [ 53 ] , [ 47 ] , [ 66 ] and [ 53 ] , respectively.
Figure S16: Trainable segmentation quantitive benchmark. (a) For a set of 5 training images, N rounds of human-like labels were added. (b) The class-averaged mIoU over 17 test images (with no labels) as a function of number of labelling rounds. (c) Example predictions of XGBoost classifier trained to map features labels after 5 labelling rounds for classical, classical + DINOv2 and classical + ALiBi-Dv2. ALiBi-Dv2 offers improved performance over DINOv2 with less labels due to the lack of positional bias in its features.