Foundational vision models have become the de facto standard for many vision tasks due to their strong performance. However, they are notoriously opaque and remain hard to interpret. We present ALOE (ALign Once to Explain), a one-time, label-free feature alignment approach that efficiently converts foundational vision models into inherently interpretable B-cos variants. Once aligned, the B-cos backbone is used as a drop-in replacement across several downstream tasks-amortizing the cost of interpretability. ALOE is robust across pre-training paradigms (supervised, self-supervised, vision-language) and can preserve teacher-level generalization with orders of magnitude less data than training from scratch. On classification, it strongly outperforms vanilla B-cosification (e.g., +10.5 p.p. top-1 on ImageNet for DINOv3 ViT-B/16), retains strong linear probing, k-NN, and zero-shot transfer competitive with foundational backbones (DINOv3, SigLIP2) across diverse downstream datasets. It also preserves spatially structured features useful for dense prediction, while yielding well-localized and highly human-interpretable explanations by design.
Figures & tables
Figure 2: ALOE vs. B-cosification [ 4 ] . ALOE consistently outperforms vanilla B-cosification in ImageNet-1k linear-probe accuracy across model scales for both SigLIP2 [ 76 ] and DINOv3 [ 72 ] (see (a)). It reaches performance close to the original SigLIP2 and DINOv3 baselines while significantly improving localization scores (Grid-PG, see (b))—even compared against different SOTA post-hoc attribution methods.
Figure 3: Conventional vs. B-cos ViT. We replace linear/MLP layers with bias-free B-cos transforms and uncentered normalization, keeping self-attention and residuals intact. This yields a dynamic-linear B-cos ViT with inherently faithful explanations via W(x)x ( Sec. 3.1 ).
Figure 4: Align Once to Explain. (1) Transform to B-cos —convert a Foundation ViT encoder into a bias-free, dynamic-linear backbone ( Sec. 4.1 ). (2) Align once —label-free feature alignment with cosine distance objective between foundational teacher and B-cos student, on unlabeled images ( Sec. 4.2 ). (3) Deploy —freeze the aligned B-cos backbone for linear probing and zero-shot transfer; faithful, well-localized explanations follow directly from W(x) ( i.e . , model summary), amortizing interpretability across tasks ( Sec. 5.1 ).
Model
Global Feature
Layer-wise Features
Supervised [ 25 ]
Final [CLS]
Image tokens, [CLS]
DINOv3 [ 72 ]
Final [CLS]
Image tokens, [CLS] , register tokens
SigLIP2 [ 76 ]
Attention-pooled final embedding
Image tokens
Table 1: Tokens used for alignment ( Sec. 4.2 ) for each model.
Feature
Inh. Inter.
IN1k
Cal101
Flowers
Food
Aircr
DTD
Cars
SUN
C10
C100
Avg.
Sup. [ 25 ]
✗
81.19
97.43
99.33
86.22
41.31
74.44
57.39
74.19
97.07
86.56
79.51
B-cosif. [ 4 ]
✓
77.65
96.35
95.83
78.18
37.07
71.15
46.06
66.83
95.00
81.43
74.56
ALOE (ours)
✓
81.30
97.32
99.44
86.73
43.96
75.55
61.59
74.99
97.53
87.67
80.61
+3.65
+0.97
+3.61
+8.55
+6.89
+4.40
+15.53
+8.16
+2.53
+6.24
+6.05
SigLIP2 [ 76 ]
✗
84.21
98.88
99.21
94.44
75.48
85.74
95.42
81.70
96.95
84.91
89.69
B-cosif. [ 4 ]
✓
75.05
97.52
94.14
83.74
45.04
79.18
73.70
75.68
94.44
81.11
79.96
Table 2: Linear-probe accuracy on base architectures (ViT-B/16). ALOE substantially outperforms vanilla B-cosification while remaining competitive with the original foundation models. Teachers are shown in gray; best interpretable model in bold . Note: For DINOv3 B-cosification we follow the supervised fine-tuning protocol [ 4 ] since the original work did not have a self-supervised baseline. All B-cosification baselines use the original public implementation ( Sec. B ).
Table 3: Downstream and Zero-shot Evaluations. (a) k -NN results ( k=20 ). (b) Zero-shot ImageNet top-1 accuracy (SigLIP2 ViT-B/16, 80-prompt template). In both evaluation settings, ALOE substantially outperforms vanilla B-cosification and remains highly competitive with the original teacher models.
Figure 5: PCA Visualization. DINOv3 ( row 2 ) vs. ALOE ( row 3 ) last layer features visualized with 3 principal components and mapped to joint RGB space, preserves global feature geometry that indicates aligned semantics while being inherently interpretable.
Figure 6: Explanation quality. Comparison between model inherent (B-cos: W(x) [ 8 ] ) explanations after ALOE vs a popular strong method for visualizing vision transformers, AttnLRP [ 1 ] .
Figure 7: VLM Zero-Shot Explanations. ALOE aligned B-cos models (SigLIP2 teacher) yield zero-shot model-inherent explanations for VLMs, where the input prompt to the text encoder is “this is a photo of {class name}”. The explanations are visually well aligned with class-specific features.
Figure 8: Token-level Visual Grounding with ALOE † . By propagating relevance through the LLaVA-MORE [ 21 ] Gemma-2-9B [ 74 ] backbone (using AttnLRP) and extracting visual explanations from our B-cos SigLIP2 encoder, we achieve fine-grained localization for individual generated tokens. The relevance maps highlight well their corresponding semantic regions in the input image. Further examples and failure cases are shown in Sec. D . † Earlier CVPR-version checkpoint [ 47 ] ; see Appendix B .
Pool
+L
+{2/3,L}
+{1/3,2/3,L}
+ All
80.86
83.06
88.66
88.79
88.41
Table 4: Feature depth ablation † . Average LP accuracy over 10 datasets for SigLIP2 ViT-B/16 (cosine loss). † Earlier CVPR-version checkpoint [ 47 ] ; see Appendix B .
Figure 9: Data efficiency of ALOE. ImageNet top-1 linear-probe accuracy for ALOE aligned SigLIP2 (ViT-B/16) across varying amounts of YFCC15M data (0.1%–100%). Across these settings, including the full-data distillation, ALOE remains nearly flat (83.09%–83.87%). Already at 1% (150k images), it reaches 83.79%, within 0.42 p.p. of the SigLIP2 teacher (84.21%) and 8.74 p.p. above the B-cosification reference. The 100% setting reaches 83.87%, leaving only a 0.34 p.p. gap to the teacher.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Family
Architecture
Eval res.
Identifier
Fully Supervised Models [ 31 ]
Supervised [ 25 ]
ViT-B/16
224 × 224
google/vit-base-patch16-224
Vision–Language Models [ 29 , 30 , 28 ]
SigLIP2 [ 76 ]
ViT-B/16
224 × 224
google/siglip2-base-patch16-224
SigLIP2 [ 76 ]
ViT-L/16
256 × 256
google/siglip2-large-patch16-256
SigLIP2 [ 76 ]
ViT-so400m/16
256 × 256
google/siglip2-so400m-patch16-256
Appendix
Table A1: Teacher encoders used for alignment. We adopt each teacher’s native evaluation resolution and keep their weights frozen during alignment.
Feature
Arch
Inter.
IN1k
Cal101
Flowers
Food
Aircr
DTD
Cars
SUN
C10
C100
Avg.
Fully Supervised Pre-Training
Sup. [ 25 ]
ViT-B/16
✗
81.19
97.43
99.33
86.22
41.31
74.44
57.39
74.19
97.07
86.56
79.51
B-cosif. [ 4 ]
B-ViT-B/16
✓
77.65
96.35
95.83
78.18
37.07
71.15
46.06
66.83
95.00
81.43
74.56
ALOE
B-ViT-B/16
✓
81.30
97.32
99.44
86.73
43.96
75.55
61.59
74.99
97.53
87.67
80.61
+3.65
+0.97
+3.61
+8.55
+6.89
+4.40
+15.53
+8.16
+2.53
+6.24
+6.05
Vision Language Pre-training
Appendix
Table C1 : Linear-probe accuracy on frozen features. ALOE B-cos ViTs substantially outperform B-cosification while remaining competitive with the original foundation models on ImageNet-1k and on the 10-dataset average. All models use the same protocol and resolution. Teachers are shown in gray; best per block in bold (for B-cos models where a vanilla B-cosification baseline exists). ✓ : denotes inherently interpretable models (vs. not ✗ ).
Feature
Arch
Inter.
IN1k
Cal101
Flowers
Food
Aircr
DTD
Cars
SUN
C10
C100
Avg.
Fully Supervised Pre-Training
Sup. [ 25 ]
ViT-B/16
✗
80.70
92.07
87.61
78.98
22.02
62.89
29.96
68.62
96.38
82.42
70.17
B-cosif. [ 4 ]
B-ViT-B/16
✓
77.05
89.58
75.13
65.00
19.14
59.20
23.45
56.74
93.14
74.57
63.30
ALOE (ours)
B-ViT-B/16
✓
80.76
92.29
87.61
79.73
23.04
63.50
31.38
68.98
96.65
83.20
70.71
+3.71
+2.71
+12.48
+14.73
+3.90
+4.30
+7.93
+12.24
+3.51
+8.63
+7.41
Vision Language Pre-training
Appendix
Table C2 : k -NN accuracy on frozen features. For the k -NN ( k=20 ) evaluation setting, ALOE B-cos ViTs again significantly outperform B-cosification [ 4 ] while remaining competitive with the original foundation models on ImageNet-1k and on the 10-dataset average. All models use the same protocol and resolution (see Sec. 4.4 ). Teachers are shown in gray; best per block in bold (for B-cos models). ✓ : denotes inherently interpretable models (vs. not ✗ ).
Architecture
Inh. Inter.
Zero-shot Acc.
Base architecture
SigLIP2 [ 76 ] — ViT-B/16
✗
78.07
B - cosif. [ 4 ] — B - ViT - B/16
✓
58.66
ALOE (ours) — B - ViT - B/16
✓
77.17
Δ (ALOE vs. B - cosif.)
+18.51
Larger architectures
Appendix
Table C3: Zero-shot ImageNet-1k with SigLIP2 prompts. We replace the SigLIP2 image encoder with the ALOE-aligned B-cos counterpart and evaluate zero-shot classification with the OpenCLIP 80-prompt template [ 34 , 19 ] . Values are ImageNet top-1 accuracy (%). Teachers are shown in gray; ✓ denotes inherently interpretable B-cos models ( ✗ are not). For ViT-B/16, ALOE substantially outperforms B - cosification and remains competitive with the teacher; similar trends hold for larger models. The Δ ( row 4 ) reports ALOE minus B-cosification.
Method
Inh. Inter.
Relative
Absolute
δ1↑
RMSE ↓
δ1↑
RMSE ↓
DINOv3 [ 72 ]
✗
0.9542
0.2779
0.7998
0.4748
B - cosif. [ 4 ]
✓
0.8311
0.4604
0.6503
0.6804
ALOE (ours)
✓
0.9316
0.3275
0.7089
0.5845
Δ (ALOE vs. B - cosif.)
+0.1005
-0.1329
+0.0586
-0.0959
Appendix
Table C4: Dense linear probing for monocular depth (ViT-B/16). We report relative and absolute depth metrics; higher is better for δ1 and lower is better for RMSE. The Δ ( row 4 ) reports ALOE minus B-cosification.
Method
Inh. Inter.
δ1↑
δ2↑
δ3↑
RMSE ↓
DINOv3 [ 72 ]
✗
0.57
0.77
0.83
23.7
B - cosif.
✓
0.38
0.61
0.70
31.7
ALOE (ours)
✓
0.54
0.75
0.82
24.9
Δ (ALOE vs. B - cosif.)
+0.16
+0.14
+0.12
-6.8
Appendix
Table C5: Surface Normals Estimation (ViT-B/16). We report angular accuracy metrics ( δn : fraction of pixels whose angular error is below the n -th threshold) and RMSE using Probe3D [ 26 ] . Higher is better for δ metrics; lower is better for RMSE. The Δ ( row 4 ) reports ALOE minus B-cosification.
Figure C1 : Multiview correspondence. Evaluated on two different datasets (left: NAVI [ 35 ] , right: ScanNet [ 23 ] ) using Probe3D [ 26 ] . ALOE increases performance substantially over B-cosification and narrows the gap to the teacher model.
Figure C2 : Data efficiency of ALOE for DINOv3 on YFCC15M (ViT-B/16). We vary the amount of unlabeled YFCC15M data used for label-free alignment from 0.1% (15k images) to 100% (15M images) and report ImageNet-1k linear-probe accuracy. Across these settings, including the full-data distillation, ALOE remains nearly flat (83.71%–84.14%). At 10% (1.5M images), it already substantially outperforms B-cosification (+10.4 p.p.). The 100% setting reaches 84.14%, leaving only a 0.22 p.p. gap to the DINOv3 teacher (84.36%).
Model
Lin. Probe ↑
k -NN ↑
Grid-PG ↑
ΔGridPG
Teacher ( ✗ )
ALOE ( ✓ )
Teacher ( ✗ )
ALOE ( ✓ )
LRP
IG
LeG
Chf
ALOE ( ✓ )
Vision–language teacher: SigLIP2 [ 76 ]
ViT-B/16
84.21
83.87
80.39
79.97
55.47
39.22
34.10
54.27
84.42
+28.95
ViT-L/16
87.15
87.08
83.76
83.89
47.39
37.46
36.62
52.57
83.48
+36.09
ViT-so/16
87.98
87.83
84.51
84.56
49.81
37.82
46.09
55.03
82.75
+32.94
ViT-so/14@384
88.65
88.48
85.08
85.13
49.54
37.71
45.18
56.03
83.07
+33.53
Appendix
Table C6 : Localization (Grid-PG) vs. recognition. ALOE improves Grid-PG substantially across backbones and teachers while maintaining competitive linear-probe and k -NN accuracy on ImageNet-1k. Grid-PG is reported for Teacher using AttnLRP (LRP) [ 1 ] , Integrated Gradients (IG) [ 73 ] , LeGrad (LeG) [ 10 ] , and CheferCAM (Chf) [ 16 ] , while ALOE uses model inherent B-cos attributions. ΔGridPG is ALOE minus the teacher with AttnLRP (LRP). ✓ : denotes inherently interpretable models. Fields marked with - (LeGrad and CheferCAM on the supervised ViT) were not implemented for this architecture.
Figure C3 : Perturbation stability of explanations on ViT-B/16. Target-class probability vs. percentage of top-attributed 16×16 pixel blocks removed (lower curves are better). Across both teachers— DINOv3 (left) and SigLIP2 (right)—removing the blocks ranked highest by ALOE (ours) model-inherent B-cos attributions W(x)x [ 8 ] causes a rapid drop in model confidence, indicating faithful localization. Interestingly, LIME [ 63 ] attributions cause the fastest initial drop for both teachers and perform best overall for SigLIP2, while ALOE remains competitive.
Figure C4 : GridPG on standard DINOv3 and ALOE. We compute GridPG scores on DINOv3 ViT-S/B/L teachers and their aligned ALOE counterparts using the same post-hoc attribution methods. Applying post-hoc methods to ALOE models increases localization strongly across most methods, narrowing the gap to inherent B-cos explanations.
Figure C5 : Example for the User Study † . Users were given two attribution maps from randomly drawn methods and had to decide which attribution map helps them best to understand the model’s prediction. Blue shows positive attributions while yellow shows negative ones. † Earlier CVPR-version checkpoint [ 47 ] ; see Appendix B .
Method
BT Score ↑
ALOE Win Rate ↑
ALOE †
- 0.97
–
IntGrad
- 0.41
63.5%
AttnLRP
- 0.11
70.2%
CheferCAM
-0.11
74.5%
LeGrad
-0.59
82.6%
LIME
-0.79
85.3%
Appendix
Table C7: Human Evaluation † . ALOE achieves the highest Bradley–Terry (BT) score and outperforms all baselines in pairwise comparisons (Win Rate >50% ). † Earlier CVPR-version checkpoint [ 47 ] ; see Appendix B .
Loss
MSE
Cosine
SigLIP
InfoNCE
DINOv3
83.9
84.0
83.7
83.5
SigLIP2
75.5
75.8
76.0
75.7
Google ViT
81.0
81.1
81.1
81.0
Appendix
Table C8 : Alignment objective ablation † . IN1k top-1 (%) using only the global alignment term ( Lglobal , no layer-wise terms); the selected objective was then extended to intermediate depths ( Tab. 4 ). † Earlier CVPR-version checkpoint [ 47 ] ; see Appendix B .
Figure D1 : Zero-shot, model-inherent VLM explanations. RGB explanations of ALOE (ours) using W(x)x attributions, given the fixed text prompt “this is a photo of { class name } ” . Our explanations are sharply localized on class-relevant regions.
Figure D2 : Qualitative attribution comparisons. Example 1: Visualizations for ALOE (ours) —using model-inherent B-cos attributions W(x)x —versus popular post-hoc methods (AttnLRP, Integrated Gradients (IntGrad), LeGrad, CheferCAM, and LIME). ALOE produces sharper, better-localized, and color-faithful highlight maps with less background noise, focusing on class-relevant object regions consistently across examples.
Figure D3 : Qualitative attribution comparisons. Similar to Figure D2 .
Figure D4 : Qualitative attribution comparisons–not well-localized examples. Visualizations for ALOE (ours) —using model-inherent B-cos attributions W(x)x —versus popular post-hoc methods (AttnLRP, Integrated Gradients (IntGrad), LeGrad, CheferCAM, and LIME). We highlight failure cases where explanations lack localization, suggesting a reliance on background context. Given that B-cos attributions are mathematically faithful to the model’s linear transformation, these diffuse heatmaps could also expose model-level shortcuts rather than “explanation failure.”
Figure D5 : Depth Estimation. Visualization of predicted depth maps for ALOE (ours) , DINOv3 and B-cosification on the NYUv2 [ 71 ] depth-estimation dataset using the Probe3D [ 26 ] protocol. ALOE-aligned features yield depth maps with coherent geometry that visually are very similar to the DINOv3 teacher model. The depth maps from the vanilla B-cosification model seem to be more noisy and blurred.
Figure D6 : Token-level Visual Grounding via AttnLRP and ALOE † . We generate visual explanations by leveraging the inherent B-cos interpretability of the ALOE † SigLIP2 vision backbone, using AttnLRP to propagate relevance from LLaVA-MORE’s Gemma-2-9B language model. † Earlier CVPR-version checkpoint [ 47 ] ; see Appendix B .
Figure D7 : Token-level Visual Grounding via AttnLRP and ALOE † . Similar to Fig. D6 . † Earlier CVPR-version checkpoint [ 47 ] ; see Appendix B .
Figure D8 : Token-level Visual Grounding via AttnLRP and ALOE † . Similar to Figs. D6 and D7 . † Earlier CVPR-version checkpoint [ 47 ] ; see Appendix B .