What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers

Jesse Ponnock

cs.CVAug 19, 2026

What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers

Authors: Jesse Ponnock

Organizations: Johns Hopkins University

Abstract

Vision transformers (ViTs) trained to copy a pretrained teacher's attention maps recover most of fine-tuning's in-distribution accuracy yet fall measurably short of it under distribution shift, as recent work has shown. What the copy delivers has never been measured directly in the attention structure and tied to robustness. We build that instrumentation for ViT-S students of a self-supervised teacher on ImageNet-100, and report three findings that triangulate one conclusion. First, the transfer is essentially perfect and permanently so: the distilled student's attention ends up roughly two orders of magnitude closer to the teacher's than fine-tuning does, and does not drift with additional training. Second, the gap is real at 14 $\times$ fewer parameters and 10 $\times$ less data than previously studied, but it has a time axis. It tracks training maturity, and completing the schedules that the stopping rule interrupted closes it below our pre-registered threshold in two of three seeds, with comparisons at equal accuracy giving the same result. The endpoint gap at this scale is substantially a training-maturity artifact: robustness matures later than accuracy, and stopping rules tuned to accuracy undersample it. Third, forcing cross-row redundancy down by half the structural separation between the distilled and fine-tuned conditions produces no detectable robustness response under two registered ways of matching accuracy. Verified transfer, a gap that closes while the structure never moves, and a null under direct intervention are together consistent with the deficit residing in features, not in the visible attention structure. This is elimination plus intervention, and its scope is the regime we measured. In this regime, attention overlays show where a model looks, not what it knows.

Figures & tables

Figure 1: Study design: one probe-selected MAE teacher; three training conditions (scratch, weight transfer, attention transfer)

\times

three seeds under one pre-registered stopping rule; an ADL dose ladder on the distillation objective; full-trajectory checkpoint retention feeding the matched-accuracy analyses; and labeled full-schedule reruns as the calibration layer.

Figure 2: Three classes across the evaluation suites: the ID validation image and the same class as it appears in each natural-shift suite, plus an IN-C panel rendered from the ID image with the study’s reference corruption implementation (gaussian noise, severity 3). Selection is by fixed rule, with no per-image curation: the classes are the first three, in sorted WordNet-ID order, of the IN-100 classes covered by both IN-A and IN-R, and each panel is the first file in sorted filename order in its class directory, shown under the evaluation crop.

Student	KL(t $\\|$ s) (nats)	Cosine	$n$
Attention transfer	0.0170 (0.0124–0.0217)	0.989 (0.986–0.992)	3
+ ADL $\lambda{=}3.0$	0.0135 (0.0126–0.0141)	0.991 (0.990–0.991)	3
Weight transfer	1.6545 (1.6243–1.6800)	0.499 (0.494–0.507)	3
Scratch	2.1482 (2.0881–2.2270)	0.342 (0.339–0.345)	3

Table 1: Eval-time transplant fidelity on held-out validation images: mean per-layer/head KL(teacher

\|

student) and cosine similarity of attention maps. Mean (min–max) over seeds.

Model	Entropy	Top- $k$	Redundancy	Stability
Teacher (MAE)	3.596	0.418	0.4665	–
Attention transfer	3.610	0.415	0.4683	0.810
Weight transfer	4.136	0.305	0.5128	0.742
Scratch	4.338	0.257	0.5837	0.750

Table 2: Attention structure at rule-governed endpoints (seed means; per-seed dispersion in Appendix E ). Entropy and top-

k

mass measure per-row concentration; cross-row redundancy is Guo et al.’s diversity quantity (higher = rows more alike); stability is the clean–corrupt attention cosine.

Figure 3: Per-layer attention KL(teacher

\|

student) on held-out validation images (log scale; thin lines are seeds, bold lines seed means). The distilled students sit roughly two orders of magnitude below both contrasts at every layer; the maximum-dose ADL students (dashed) track the dose-zero distilled band (mean 0.0135 vs. 0.0170 nats).

Figure 4: Cross-row redundancy at rule-governed endpoints (mean cosine of each thresholded attention row to the head’s mean row, Guo Eq. 1–2; dots: seeds; open markers: seed means; dashed line: teacher). Distilled students sit at the teacher’s value, including after full 300-epoch schedules; supervised training drifts redundancy upward when unconstrained.

Figure 5: Mean attention received per patch (layer 6 of 12, averaged over heads and patch queries; CLS excluded as query), for three validation images: the first three of the study’s fixed eight-image attention snapshot, with no per-image selection. The color scale is shared within each column, so panels are comparable across models; per-column ranges are printed below. The layer was selected by a fixed rule (the layer whose teacher–distill similarity is the median across layers), not for visual effect. The comparison is one of similarity only: the attention-transfer row reproduces the teacher’s maps (cosine 0.998–0.999 on these images); weight transfer and scratch differ from both. Nothing in this figure bears on which attention is better ; Section 4.4 addresses that question causally.

Condition	Seed	Stop epoch	ID top-1	Eff. robustness [95% CI]
Scratch	s0	300 (cap)	0.8498	0.4199 [0.4070, 0.4332]
	s1	300 (cap)	0.8496	0.4168 [0.4038, 0.4301]
	s2	290 (early stop)	0.8462	0.4188 [0.4053, 0.4324]
Weight transfer (ft)	s0	200 (cap)	0.8916	0.4943 [0.4814, 0.5073]
	s1	200 (early stop)	0.8896	0.4913 [0.4785, 0.5044]
	s2	200 (cap)	0.8926	0.4964 [0.4836, 0.5092]

Table 3: Rule-governed endpoints per condition and seed, with labeled full-schedule reruns. Effective robustness is the 4-suite mean; brackets are paired image-level bootstrap 95% CIs (B=10,000). The evaluated checkpoint is each run’s min-delta-promoted best, which may precede the stop epoch shown.

Seed	Endpoint gap	Completed-schedule gap
s0	+0.0575 [0.0445, 0.0703]	+0.0044 [-0.0059, 0.0148]
s1	+0.0323 [0.0209, 0.0442]	+0.0013 [-0.0096, 0.0122]
s2	+0.0181 [0.0073, 0.0289]	+0.0110 [0.0003, 0.0217]

Table 4: The robustness gap (ft

-

distill, mean effective robustness) per seed, in the endpoint coordinate (rule-governed stops) and at completed schedules (full-schedule reruns). Pre-registered threshold 0.01. Paired bootstrap 95% CIs.

Figure 6: A: endpoint effective robustness against stop epoch for all 15 distill-family runs, with

r=0.978

computed on true stop epochs. Bands are min–max of the three weight-transfer and scratch seeds. Open squares are the full-schedule reruns at epoch 300, with dotted lines connecting each to its same-seed canonical endpoint; the s0 and s1 reruns coincide to

10^{-4}

and are drawn nested. For legibility, the six dosed runs that stop exactly at the 300-epoch cap are horizontally dodged by

\leq

5 epochs; all true stops are exactly 300. B: three clocks within seed 0, canonical solid and rerun dashed, where the rerun replays the canonical prefix exactly on shared epochs. Accuracy reaches 93% of its final value by epoch 119. Effective robustness’s highest probe arrives at epoch 294, at 0.490, with the final probe reading 0.487. Cross-row redundancy is flat from epoch 24 at the teacher’s value. The dotted vertical line marks the rule’s cut at epoch 145.

	Own-level rung		Top rung
Seed	level	gap	level	gap
s0	0.8248	+0.0237	0.8850	+0.0037
s1	0.8618	-0.0145	0.8794	-0.0030
s2	0.8812	+0.0049	0.8832	+0.0049

Table 5: The matched-accuracy ladder (wave-2): gap (ft

-

distill, mean effective robustness) at each seed’s own matched level and at the top matched level, trajectory checkpoints both sides; top-rung distill checkpoints come from the labeled full-schedule reruns. Selection is by nearest ID to the target level; for s1’s top rung this picks the rerun’s epoch-299 checkpoint rather than its promoted best, and the gap stays inside the band under either choice (Section 4.3 ). Bootstrap intervals do not exist for trajectory checkpoints (Section 3 ).

Figure 7: The matched-accuracy gap per seed at its own matched level (filled) and at the top matched level (open); connectors join each seed’s two measured rungs and are annotations, not measured paths. Shaded band = pre-registered

\pm

0.01 threshold. Gaps of both signs converge into the band as the matched level rises.

Dose	s0	s1	s2	Mean $\pm$ SD
$\lambda_{\mathrm{adl}}=0.1$	-0.0022	-0.0025	-0.0027	-0.0025 $\pm$ 0.0002
$\lambda_{\mathrm{adl}}=0.3$	-0.0035	-0.0048	-0.0030	-0.0038 $\pm$ 0.0009
$\lambda_{\mathrm{adl}}=1.0$	-0.0095	-0.0105	-0.0100	-0.0100 $\pm$ 0.0005
$\lambda_{\mathrm{adl}}=3.0$	-0.0246	-0.0252	-0.0241	-0.0247 $\pm$ 0.0006

Table 6: The causal dial: paired same-seed change in cross-row redundancy at each ADL dose (sweep endpoint minus its dose-zero baseline). Negative = less redundant.

	Per-pair				Iso-level
Dose	s0	s1	s2	mean	s0	s1	s2	mean
$\lambda_{\mathrm{adl}}=0.1$	+0.0154	+0.0017	-0.0259	-0.0029	+0.0154	+0.0054	-0.0380	-0.0057
$\lambda_{\mathrm{adl}}=0.3$	+0.0196	+0.0010	+0.0019	+0.0075	+0.0196	+0.0180	-0.0132	+0.0082
$\lambda_{\mathrm{adl}}=1.0$	+0.0236	-0.0176	+0.0010	+0.0023	+0.0236	+0.0025	-0.0373	-0.0037
$\lambda_{\mathrm{adl}}=3.0$	+0.0166	-0.0163	-0.0013	-0.0003	+0.0166	+0.0039	-0.0370	-0.0055

Table 7: The causal test: matched-skill

\Delta

effective robustness (dosed

-

dose-zero, same seed) at every ADL dose, in both registered coordinates: per-pair (each pair at its own matched level) and iso-level (all models at

L^{*}=0.8248

). Read against the frozen verdict definitions: flat. Seed 0’s own matched level coincides with

L^{*}

, so its two blocks repeat one measurement by construction. Bootstrap intervals do not exist for trajectory checkpoints (Section 3 ); seed-level 95% intervals on the dose means span

\pm

2.6–7.7pp.

Figure 8: The causal test. A: paired same-seed change in cross-row redundancy per dose, with points for seeds and the line for dose means, and dot shade tracking

\lambda

. The top dose displaces structure by half the distill–fine-tune separation. The dashed line is a magnitude ruler only: fine-tuning’s redundancy lies in the opposite direction, at

+0.0445

. B: matched-skill change in mean effective robustness for the same runs, per-pair (solid) and iso-accuracy (dashed) reads, on the same vertical scale as A. The shaded band is the pre-registered 1pp bar. The dial sweeps half a condition-width of structure, and robustness does not follow.

Model	Role	IN-C eff. rob.
distill_s0 (e145 endpoint)	canonical	0.6453
finetune_s0	canonical	0.7548
scratch_s0	canonical	0.7220
distill+ADL $\lambda$ 1.0_s0	canonical	0.7101
distill_s0 continuation (e300)	maturity contrast	0.7067
distill_s1 continuation (e300)	maturity contrast	0.7098

Table 8: Secondary synthetic-shift probe: ImageNet-C mean effective robustness on the frozen corruption set, role-labeled. Registered canonical rows plus the registered wave-2 maturity extension.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Validation trajectories of the three distilled seeds under the single pre-registered stopping rule, with rule-fire epochs and best checkpoints marked; line shade distinguishes seeds. Near-identical no-peak curves; stops 145 epochs apart.

Model	IN-A	IN-R	IN-Sk	IN-V2	Mean [95% CI]
scratch_vit_s_s0	0.1088	0.3871	0.3199	0.8637	0.4199 [0.4070, 0.4332]
scratch_vit_s_s1	0.1196	0.3919	0.3107	0.8451	0.4168 [0.4038, 0.4301]
scratch_vit_s_s2	0.1303	0.3922	0.3122	0.8402	0.4188 [0.4053, 0.4324]
finetune_vit_s_s0	0.1453	0.4953	0.4405	0.8961	0.4943 [0.4814, 0.5073]
finetune_vit_s_s1	0.1564	0.4805	0.4302	0.8982	0.4913 [0.4785, 0.5044]
finetune_vit_s_s2	0.1433	0.4965	0.4440	0.9019	0.4964 [0.4836, 0.5092]

Appendix

Table 9: Per-suite effective robustness for every evaluated endpoint model. Mean column carries the paired bootstrap 95% CI.

Analysis	Model	Val top-1	Epoch	Residual
dose per-pair $\lambda$ 0.1	sweep s0	0.8234	129	-0.0014
dose per-pair $\lambda$ 0.1	baseline s0	0.8248	119	+0.0000
dose per-pair $\lambda$ 0.1	sweep s1	0.8622	219	+0.0004
dose per-pair $\lambda$ 0.1	baseline s1	0.8618	219	+0.0000
dose per-pair $\lambda$ 0.1	sweep s2	0.8596	189	+0.0000
dose per-pair $\lambda$ 0.1	baseline s2	0.8598	224	+0.0002

Appendix

Table 10: Every matched-pair checkpoint selection: analysis, model, achieved validation top-1, selected epoch, and residual to the target level. All residuals

\leq 0.0034

except the s1 own-level ladder rung’s fine-tuned side (

-

0.0054, machine-flagged loose against the 0.005 tolerance).

Baseline population	$r$ (stop, residual)	ft $-$ distill endpoint residual gap
		s0	s1	s2
all 24 endpoint models	-0.562	-0.0181	-0.0122	+0.0525
fine-tune + scratch	-0.854	-0.0841	-0.0464	+0.0421
ft + scratch + dose-zero distill	-0.689	-0.0335	-0.0255	+0.0515

Appendix

Table 11: Residual-form sensitivity check (post-hoc, computed at final review; not pre-registered). Per suite, probit(OOD) is fit linearly on probit(covered-ID) over the stated baseline population; a model’s score is its mean residual over the four suites (probit units). Ratio-form references:

r=+0.978

; endpoint gaps +0.0575 / +0.0323 / +0.0181.

	Peak LR	Warmup (epochs)	Horizon (epochs)
Scratch, distill, and dose arms	$10^{-3}$	20	300
Fine-tune	$5\times 10^{-4}$	5	200
MAE teacher (pretraining)	$1.5\times 10^{-4}$	40	400 (probe-stopped)

Appendix

Table 12: Per-condition training settings; all other hyperparameters are shared as above.

Explore similar work

May 8, 2026

Attention Transfer Is Not Universally Effective for Vision Transformers

A recent work shows that Attention Transfer, which transfers only the attention patterns from a pre-trained teacher Vision Transformer (ViT) to a randomly initialized standard student ViT, is sufficient to recover the full benefit of the teacher's pre-trained weights. We revisit this finding on a comprehensive benchmark of 20 teachers from 11 well-known ViT families and reveal that Attention Transfer is not universally effective. While 7 families transfer successfully, 4 consistently fail, falling up to 5.1% below the from-scratch no-transfer baseline. Further results demonstrate that this failure is family-consistent across model sizes, and persists under extended training durations, different transfer datasets, and out-of-distribution evaluations. Controlled analyses then consistently localize the problem to the attention-routing channel, indicating that the key issue is not whether the student can match the teacher's attention patterns, but whether the matched patterns remain functional for the student. Crucially, we identify architectural mismatch between the pre-trained teacher and the standard student as the primary mechanism. By adding only the teacher's native architectural components to the student in a randomly initialized state, we completely reverse the failure for all 4 families. Notably, these components alone do not improve from-scratch training, confirming that they specifically unlock the usability of the teacher's attention. We further show that this failure is not explained by the inadequate choice of transfer loss, differences in pre-training recipes, or the intrinsic quality of attention maps. Our findings refine the prevailing understanding of attention in ViT representations: attention is sufficient only when the student architecture matches the teacher.

Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +4

Institute for Infocomm Research (I2R), A*STAR, Singapore · Sichuan University · Shanghai Jiao Tong University

Self-Attention Vision Transformer Transfer Learning Attention Mechanisms Teacher-Student Learning Knowledge Distillation

Ask PDF

May 28, 2026

A Mechanistic Analysis of Adversarial Fine-tuning of Vision Transformers

The widespread use of image classification models in high-risk, real-world situations necessitates making these models robust to slight disturbances or perturbations, such as blurring or sharpening, in the input images. While vision transformers (ViTs) play an integral role in many modern-day multi-modal models like Vision-Language-Models (VLMs) and Vision-Language-Action (VLA) models, they have received a lack of attention in the setting of robustness. In this work, we analyze the effects of adversarial fine-tuning, a popular method for improving model robustness to image perturbations, on a ViT's performance on perturbed and regular images through a mechanistic lens. We adversarially train a ViT on low-frequency and high-frequency image corruptions, and attempt to explain changes in downstream model performance through an examination of the model's attention mechanisms, internal representations, and knowledge evolution. Overall, our results suggest that, while fine-tuning on inputs with common corruptions improves model performance and certainty on new instances of corrupted data, these improvements do not transfer to other classes of corruptions not seen in the training. Additionally, despite observing changes in visual attention and knowledge evolution across layers, we found that adversarial training did not lead to fundamental changes in the sparse representations learned by ViTs.

Hannah Gao, Isha Agarwal, Dylan Hadfield-Menell +1

Massachusetts Institute of Technology · MIT Cambridge, MA

Adversarial Training Fine-Tuning Image Corruption Robustness Vision Transformer Adversarial Robustness

Ask PDF

Sep 28, 2026

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers

Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they require from-scratch pre-training and typically underperform the original Softmax version. How to initialize linear ViTs both efficiently and effectively still remains unclear. In this work, we explicitly ask: given that most foundation ViTs are built on the mainstream Softmax attention, can linear ViTs benefit from their pre-trained weights? Recent works on Attention Transfer show that attention is the effective transferable component between Softmax ViTs, suggesting attention alone suffices for such reuse. However, we find the opposite for Softmax-to-linear transfer. The attention weights are operator-specific: copying them barely helps, and is sometimes even worse than random initialization. Instead, the attention's token routing behavior can be recovered through distillation with a proper loss design, letting linear ViTs reduce the gap and even match Softmax ones. In contrast, the MLP weights, which carry the learned representation, are operator-agnostic: they can be transferred by simple direct copying, which already carries most of the benefit of the pre-trained weights. Thus, copying MLPs can serve as an effective foundation for Softmax-to-linear transfer: paired with the distilled attention, linear ViTs eventually close the remaining gap and even surpass Softmax ones. These findings hold consistently across various linear ViT variants, different model sizes, and diverse datasets. We hope this study deepens the understanding of reusing pre-trained weights across attention operators: copy what stays the same and distill what differs, to recover the benefit across the Softmax-to-linear boundary.

Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +9

Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Nanyang Technological University · ST Engineering Geo-Insights, Singapore +3

Neural Network Initialization Vision Transformer Efficient ViTs Transfer Learning Knowledge Distillation

Ask PDF