ViT-AdaLA: Adapting Vision Transformers with Linear Attention

Yifan Li; Seunghyun Yoon; Viet Dac Lai; Franck Dernoncourt; Jason Kuen; Yu Kong; Trung Bui

cs.CVMar 17, 2026

ViT-AdaLA: Adapting Vision Transformers with Linear Attention

Authors: Yifan Li, Seunghyun Yoon, Viet Dac Lai, Franck Dernoncourt, Jason Kuen, Yu Kong, Trung Bui

Organizations: Michigan State University, East Lansing, MI, USA. · Adobe Research.

Abstract

Vision Transformers (ViTs) based vision foundation models (VFMs) have achieved remarkable performance across diverse vision tasks, but suffer from quadratic complexity that limits scalability to long sequences. Existing linear attention approaches for ViTs are typically trained from scratch, requiring substantial computational resources, while linearization-based methods developed for large language model decoders do not transfer well to ViTs. To address these challenges, we propose ViT-AdaLA, a novel framework for effectively adapting and transferring prior knowledge from VFMs to linear attention ViTs. ViT-AdaLA consists of three stages: attention alignment, feature alignment, and supervised fine-tuning. In the attention alignment stage, we align vanilla linear attention with the original softmax-based attention in each block to approximate the behavior of softmax attention. However, residual approximation errors inevitably accumulate across layers. We mitigate this by fine-tuning the linearized ViT to align its final-layer features with a frozen softmax VFM teacher. Finally, the adapted prior knowledge is transferred to downstream tasks through supervised fine-tuning. Extensive experiments on classification and segmentation tasks demonstrate the effectiveness and generality of ViT-AdaLA over various state-of-the-art linear attention counterpart.

Figures & tables

Figure 1: Comparison between training-from-scratch and linearization paradigms for ViTs with linear attention. Training-from-scratch linear attention paradigms focus on designing accurate attention approximation methods, which typically require large-scale pretraining to acquire strong prior knowledge. In contrast, ViT linearization leverages an off-the-shelf pretrained ViT, substantially reducing the need for extensive pretraining.

Figure 2: Comparison of decoder and encoder–decoder architectures. In decoder-based LLMs, the LLM serves as both the feature extractor and the target generator. In contrast, in vision models, ViTs function solely as feature extractors, while a separate task-specific head is responsible for target generation.

Figure 3: Efficiency comparison of different attentions, including peak memory and GFLOPS varying with sequence length. Only attention module is benchmarked in these experiments. “Vanilla” indicates the vanilla linear attention ( Katharopoulos et al., 2020 ) .

Figure 4: ViT-AdaLA consists of three stages: attention alignment, feature alignment, and supervised fine-tuning. First, softmax attention is approximated by tuning only the linear attention modules. Second, to mitigate residual approximation errors that accumulate across layers, the feature alignment stage finetunes the entire linearized model by aligning its final-layer representations with those of the original softmax-based teacher. Finally, supervised fine-tuning is performed to transfer the adapted prior knowledge to downstream tasks.

Figure 5: Linear attention architecture and Stage 1 training loss comparison with LoLCATS ( $SM$ : softmax; $\oplus$ : concatenation). LoLCATS approximates the attention output based on Hedgehog ( Zhang et al., 2024 ) by tuning only two additional mapping modules applied to the queries and keys individually. In contrast, we tune all query, key, and value weights to approximate the attention output, which is both more efficient and more effective than the original LoLCATS approach.

Figure 6: Visualization of PCA-projected features from the final layer of DINOv2-L. We compare the original softmax-based features with those produced by ViT-AdaLA and Monarch attention ( Yaras et al., 2025 ) by projecting them to three channels using PCA. The results indicate that ViT-AdaLA better preserves the prior feature knowledge of the VFM. We provide more visualization results in the App. B .

	Backbone	Res.	Params	FLOPS	Peak Mem.	Throughput	Top-1 Acc.
	Backbone	Res.	(M)	(G)	(GB)	(imgs/s)	(%)
Softmax ( Oquab et al., 2024 )	DINOv2-L	$512^{2}$	304.20	310.60	1.3181	36.52	86.8
Hedgehog ( Zhang et al., 2024 )	DINOv2-L	$512^{2}$	305.77	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	58.8
LoLCATS ( Zhang et al., 2025 )	DINOv2-L	$512^{2}$	305.77	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	61.6
Linformer ( Wang et al., 2020 )	DINOv2-L	$512^{2}$	305.77	262.19 ↓15.6%	1.2221 ↓7.3%	45.41 ↑25.3%	68.9
Performer ( Choromanski et al., 2021 )	DINOv2-L	$512^{2}$	304.20	265.41 ↓14.6%	1.2164 ↓7.7%	36.48 ↑0.1%	75.2

Table 1: Top-1 fine-tuning accuracy comparison on ImageNet-1K under different vision foundation models with multiple linear attention baselines. We reproduce all baselines to ensure a fair comparison. The classification head is a single linear layer for all the methods.

	Backbone	Res.	Params	FLOPS	Peak Mem.	Throughput	mIoU
	Backbone	Res.	(M)	(G)	(GB)	(imgs/s)	(%)
Softmax ( Oquab et al., 2024 )	DINOv2-L	$512^{2}$	304.20	310.60	1.3181	36.52	56.73
Hedgehog ( Zhang et al., 2024 )	DINOv2-L	$512^{2}$	305.77	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	22.52
LoLCATS ( Zhang et al., 2025 )	DINOv2-L	$512^{2}$	305.77	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	17.42
Linformer ( Wang et al., 2020 )	DINOv2-L	$512^{2}$	305.77	262.19 ↓15.6%	1.2221 ↓7.3%	45.41 ↑25.3%	18.82
Performer ( Choromanski et al., 2021 )	DINOv2-L	$512^{2}$	304.20	265.41 ↓14.6%	1.2164 ↓7.7%	36.48 ↑0.1%	41.16

Table 2: mIoU fine-tuning comparison on ADE20K under different vision foundation models with multiple linear attention baselines. We reproduce all baselines to ensure a fair comparison. The segmentation head is Mask2former for all the methods.

	Backbone	Res.	Params	FLOPS	Peak Mem.	Throughput	mIoU
	Backbone	Res.	(M)	(G)	(GB)	(imgs/s)	(%)
Softmax ( Oquab et al., 2024 )	DINOv2-L	$512^{2}$	304.20	310.60	1.3181	36.52	74.86
Hedgehog ( Zhang et al., 2024 )	DINOv2-L	$512^{2}$	305.77	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	43.97
LoLCATS ( Zhang et al., 2025 )	DINOv2-L	$512^{2}$	305.77	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	34.71
Linformer ( Wang et al., 2020 )	DINOv2-L	$512^{2}$	305.77	262.19 ↓15.6%	1.2221 ↓7.3%	45.41 ↑25.3%	37.24
Performer ( Choromanski et al., 2021 )	DINOv2-L	$512^{2}$	304.20	265.41 ↓14.6%	1.2164 ↓7.7%	36.48 ↑0.1%	43.11

Table 4: mIoU fine-tuning comparison on Cityscapes under different input resolutions (512 vs. 1024) based on DINOv2-L. We reproduce all baselines to ensure a fair comparison. The segmentation head is Mask2former for all the methods.

Figure 8: Training loss comparison of different linear attention variants in Stage 1 on DINOv2-L. To ensure a fair comparison, the query, key, and value weights are tuned in every layer for each baseline. Vanilla linear attention exhibits superior approximation performance compared to the other variants.

Panel of Figure 9 — Figure 9: Resolution scalability analysis on the Cityscapes dataset in terms of (a) mIoU, (b) throughput, and (c) peak memory.

Figure 10: ADE20K performance across different Stage 2 training epochs for DINOv2-L.

Figure 11: ADE20K performance of adapting the task model or the VFM for DINOv2-L.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	Stage 1	Stage 2				Stage 3
Epochs	4	DINOv2	SigLIP	CLIP	IN-1K	Cls	seg
Epochs	4	30	10	15	20	10	20
Batch size	32	16				8
Learning rate	1e-2	1e-3				1e-4
Weight decay	0.05
Learning schedule	fixed	Polynomial (decay factor of 0.9)

Appendix

Table 5: Hyperparameters of different training stages and VFMs.

Figure 12: Stage 1 loss comparison across linear-attention activation functions. We compare four activations: softmax, softplus, ReLU, and ELU+1.

Stages	Stage 1 Time (per epoch)	Stage 2 Time (per epoch)
DINOv2-L	4min 37s	10h 48min 46s
SigLIP-L	4min 19s	11h 22min 15s
CLIP-L	5min 56s	11h 09min 21s
IN1K-L	4min 34s	10h 02min 54s
Mean	4min 51s	10h 55min 41s

Appendix

Table 6: Comparison of training efficiency across various Visual Foundation Models (VFMs) during Stages 1 and 2. Stage 1 is conducted using the COCO dataset, while Stage 2 utilizes ImageNet-22K for large-scale pretraining. All the experiments are conducted on 8 H100s.

Method	ADE20K (mIoU)	Cityscapes (mIoU)	IN1K (Acc.)
Softmax (DINOv2-B)	53.93	79.73	84.93
ViT-AdaLA (DINOv2-B)	51.46	74.60	84.37

Appendix

Table 7: Performance comparison using DINOv2-B on ADE20K, Cityscapes and ImageNet-1K. We pretrained our ViT-AdaLA (DINOv2-B) for 40 epochs.

Method	ADE20K (mIoU)	Cityscapes (mIoU)	IN1K (Acc.)
Softmax (IN1K-L)	50.83	74.86	85.66
ViT-AdaLA (IN1K-L)	49.75	72.40	84.23
Vanilla Linear Attention (train from scratch, IN1K-L)	25.80	45.09	65.72

Appendix

Table 8: Performance comparison of training-from-scratch and on ADE20K, Cityscapes, and IN1K. Our ViT-AdaLA is pretrained for 20 epochs (Stage 2) on ImageNet-22K, whereas the vanilla linear-attention baseline is pretrained from scratch on ImageNet-1K for 200 epochs to ensure a fair comparison.

Method	Resolution	Accuracy (IN1K)	mIoU (ADE20K)
CLIP-L (tune all)	512 $\times$ 512	85.5	42.85
CLIP-L (only tune QKV)	512 $\times$ 512	82.5	35.41
DINOv2-L (tune all)	512 $\times$ 512	86.0	55.55
DINOv2-L (only tune QKV)	512 $\times$ 512	76.4	48.82

Appendix

Table 9: Comparison of tuning strategies on CLIP-L and DINOv2-L at 512

\times

512 resolution.

	Backbone	Res.	Params	FLOPS	Peak Mem.	Throughput	Top-1 Acc.
	Backbone	Res.	(M)	(G)	(GB)	(imgs/s)	(%)
Softmax ( Oquab et al., 2024 )	CLIP-L	$512^{2}$	304.20	310.60	1.3181	36.52	86.8
Hedgehog ( Zhang et al., 2024 )	CLIP-L	$512^{2}$	305.77	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	67.4
LoLCATS ( Zhang et al., 2025 )	CLIP-L	$512^{2}$	305.77	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	67.21
Linformer ( Wang et al., 2020 )	CLIP-L	$512^{2}$	305.77	262.19 ↓15.6%	1.2221 ↓7.3%	45.41 ↑25.3%	56.8
Performer ( Choromanski et al., 2021 )	CLIP-L	$512^{2}$	304.20	265.41 ↓14.6%	1.2164 ↓7.7%	36.48 ↑0.1%	69.9

Appendix

Table 10: Top-1 fine-tuning accuracy comparison on ImageNet-1K under CLIP-L with multiple linear attention baselines. We reproduce all baselines to ensure a fair comparison. The classification head is a single linear layer for all the methods.

	Backbone	Res.	Params	FLOPS	Peak Mem.	Throughput	mIoU
	Backbone	Res.	(M)	(G)	(GB)	(imgs/s)	(%)
Softmax ( Oquab et al., 2024 )	IN1K ViT-L	$512^{2}$	304.15	310.60	1.3181	36.52	65.11
Hedgehog ( Zhang et al., 2024 )	IN1K ViT-L	$512^{2}$	305.72	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	37.69
LoLCATS ( Zhang et al., 2025 )	IN1K ViT-L	$512^{2}$	305.72	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	25.04
Linformer ( Wang et al., 2020 )	IN1K ViT-L	$512^{2}$	305.72	262.19 ↓15.6%	1.2221 ↓7.3%	45.41 ↑25.3%	17.30
Performer ( Choromanski et al., 2021 )	IN1K ViT-L	$512^{2}$	304.15	265.41 ↓14.6%	1.2164 ↓7.7%	36.48 ↑0.1%	39.44

Appendix

Table 11: mIoU fine-tuning comparison on Cityscapes under different input resolutions (512 vs. 1024) based on IN1K ViT-L and SigLIP-L. We reproduce all baselines to ensure a fair comparison. The segmentation head is Mask2former for all the methods.

	Backbone	Res.	Params	FLOPS	Peak Mem.	Throughput	mIoU
	Backbone	Res.	(M)	(G)	(GB)	(imgs/s)	(%)
Softmax ( Oquab et al., 2024 )	IN1K ViT-L	$512^{2}$	304.20	310.60	1.3181	36.52	50.83
Hedgehog ( Zhang et al., 2024 )	IN1K ViT-L	$512^{2}$	305.77	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	25.15
LoLCATS ( Zhang et al., 2025 )	IN1K ViT-L	$512^{2}$	305.77	265.41 ↓14.5%	1.2221 ↓7.3%	37.44 ↑2.5%	20.17
Linformer ( Wang et al., 2020 )	IN1K ViT-L	$512^{2}$	305.77	262.19 ↓15.6%	1.2221 ↓7.3%	45.41 ↑25.3%	11.49
Performer ( Choromanski et al., 2021 )	IN1K ViT-L	$512^{2}$	304.20	265.41 ↓14.6%	1.2164 ↓7.7%	36.48 ↑0.1%	31.01

Appendix

Table 12: mIoU fine-tuning comparison on ADE20K under ViT-IN1K and SigLIP with multiple linear attention baselines. We reproduce all baselines to ensure a fair comparison. The segmentation head is Mask2former for all the methods.

Figure 13: Visualization of PCA-projected features from the final layer of DINOv2-L. Original softmax features, ViT-AdaLA (ours) and Monarch Attention features are compared by projecting to three channels using PCA. These results indicate that ViT-AdaLA can learn more prior knowledge from the original VFM.

Figure 14: Visualization of PCA-projected features from the final layer of DINOv2-L. We ablate the Stages 1 and 2 training procedures and visualize the resulting PCA features for comparison. The results indicate that Stage 2 is crucial for extracting prior knowledge from VFMs, while the inclusion of Stage 1 further enhances this knowledge transfer.

Explore similar work

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
Sep 28, 2026Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +9Neural Network Initialization Vision Transformer
Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Nanyang Technological University · ST Engineering Geo-Insights, Singapore +3
Attentive multilayer fusion for vision transformers
Jan 14, 2026Laure Ciernik, Marco Morik, Lukas Thede +4Linear Probing Vision Transformer
Machine Learning Group, Technische Universität Berlin, Berlin, Germany · Hector Fellow Academy, Karlsruhe, Germany · Berlin Institute for the Foundations of Learning and Data (BIFOLD), Berlin, Germany +7
V-CoLA: Vision Token Compression with Linear Attention
Oct 8, 2026Hao Jiang, Yiru Mao, Tianpeng Bu +9Visual Token Compression Efficient VLM Inference
Alibaba Cloud Computing, Alibaba Group

Sep 28, 2026J/K move · Enter open · S save

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers

Huaiyuan Qin, Muli Yang, Gabriel James Goenawan, Shiqi Huang+8

Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Nanyang Technological University · ST Engineering Geo-Insights, Singapore+3

Neural Network Initialization Vision Transformer Efficient ViTs Transfer Learning Knowledge Distillation

Open paper AskPDF arXiv

Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they require from-scratch pre-training and typically underperform the original Softmax version. How to initialize linear ViTs both efficiently and effectively still remains unclear. In this work, we explicitly ask: given that most foundation ViTs are built on the mainstream Softmax attention, can linear ViTs benefit from their pre-trained weights? Recent works on Attention Transfer show that attention is the effective transferable component between Softmax ViTs, suggesting attention alone suffices for such reuse. However, we find the opposite for Softmax-to-linear transfer. The attention weights are operator-specific: copying them barely helps, and is sometimes even worse than random initialization. Instead, the attention's token routing behavior can be recovered through distillation with a proper loss design, letting linear ViTs reduce the gap and even match Softmax ones. In contrast, the MLP weights, which carry the learned representation, are operator-agnostic: they can be transferred by simple direct copying, which already carries most of the benefit of the pre-trained weights. Thus, copying MLPs can serve as an effective foundation for Softmax-to-linear transfer: paired with the distilled attention, linear ViTs eventually close the remaining gap and even surpass Softmax ones. These findings hold consistently across various linear ViT variants, different model sizes, and diverse datasets. We hope this study deepens the understanding of reusing pre-trained weights across attention operators: copy what stays the same and distill what differs, to recover the benefit across the Softmax-to-linear boundary.

Abstract

Explore similar work

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers

Attentive multilayer fusion for vision transformers

V-CoLA: Vision Token Compression with Linear Attention