HAViT: Historical Attention Vision Transformer

Swarnendu Banik; Manish Das; Shiv Ram Dubey; Satish Kumar Singh

cs.CVMar 19, 2026

HAViT: Historical Attention Vision Transformer

Authors: Swarnendu Banik, Manish Das, Shiv Ram Dubey, Satish Kumar Singh

Organizations: Computer Vision and Biometrics Lab, Indian Institute of Information Technology, Allahabad

Abstract

Vision Transformers have excelled in computer vision but their attention mechanisms operate independently across layers, limiting information flow and feature learning. We propose an effective cross-layer attention propagation method that preserves and integrates historical attention matrices across encoder layers, offering a principled refinement of inter-layer information flow in Vision Transformers. This approach enables progressive refinement of attention patterns throughout the transformer hierarchy, enhancing feature acquisition and optimization dynamics. The method requires minimal architectural changes, adding only attention matrix storage and blending operations. Comprehensive experiments on CIFAR-100 and TinyImageNet demonstrate consistent accuracy improvements, with ViT performance increasing from 75.74% to 77.07% on CIFAR-100 (+1.33%) and from 57.82% to 59.07% on TinyImageNet (+1.25%). Cross-architecture validation shows similar gains across transformer variants, with CaiT showing 1.01% enhancement. Systematic analysis identifies the blending hyperparameter of historical attention (alpha = 0.45) as optimal across all configurations, providing the ideal balance between current and historical attention information. Random initialization consistently outperforms zero initialization, indicating that diverse initial attention patterns accelerate convergence and improve final performance. Our code is publicly available at https://github.com/banik-s/HAViT.

Figures & tables

Fig. 1: Overview of the modified ViT with historical attention propagation.

$\alpha$	Accuracy	Accuracy(%)
	(Random)	(Zero)
0.10	76.14 ( $\uparrow$ 0.40)	75.83 ( $\uparrow$ 0.09)
0.25	76.29 ( $\uparrow$ 0.55)	76.02 ( $\uparrow$ 0.28)
0.40	76.85 ( $\uparrow$ 1.11)	76.25 ( $\uparrow$ 0.51)
0.45	77.07 ( $\uparrow$ 1.33)	76.50 ( $\uparrow$ 0.76)
0.50	76.43 ( $\uparrow$ 0.69)	76.41 ( $\uparrow$ 0.67)

TABLE I: CIFAR-100 Results with

\alpha

Variation (Baseline ViT accuracy = 75.74)

Panel of Figure 3 — Fig. 2: (a) Test accuracy comparison over epochs for CIFAR-100 dataset. (b) Training loss comparison over epochs for CIFAR-100 dataset.

$\alpha$	Accuracy	Accuracy (%)
	(Random)	(Zero)
0.10	57.79 (↓0.03)	57.89 (↑0.07)
0.25	58.21 (↑0.39)	57.90 (↑0.08)
0.40	58.93 (↑1.11)	58.32 (↑0.50)
0.45	59.07 (↑1.25)	58.73 (↑0.91)
0.50	58.74 (↑0.92)	58.47 (↑0.65)

TABLE II: TinyImageNet Results with

\alpha

Variation (Baseline ViT accuracy = 57.82)

Panel of Figure 5 — Fig. 3: (a) Test accuracy comparison over epochs for TinyImageNet dataset. (b) Training loss comparison over epochs for TinyImageNet dataset.

Panel of Figure 6 — Fig. 4: Initialization Strategy Impact Analysis: (a) Accuracy for $\alpha$ on CIFAR-100 (ViT baseline 75.74%), (b) Accuracy for $\alpha$ on TinyImageNet (ViT baseline 57.82%).

Fig. 5: Attention map comparison between Base ViT and HAViT across CIFAR-100 classes

Architecture	Dataset	Baseline (%)	Modified (%)
ViT	CIFAR-100	75.74	77.07
CaiT	CIFAR-100	73.85	74.86
ViT	TinyImageNet	57.82	59.07
CaiT	TinyImageNet	58.01	59.02

TABLE III: Cross-Architecture Comparison (optimal

\alpha=0.45

, random initialization.)

Method	Architecture Type	Accuracy (%)
ResNet-56 [ 25 ]	CNN	74.81
ResNet-110 [ 25 ]	CNN	76.63
ViT-Lite-7/4 [ 31 ]	Standard ViT	73.94
CVT-7/4 [ 31 ]	Compact ViT	76.49
CCT-2/3×2 [ 31 ]	Compact Convolutional ViT	66.93
CCT-7/3×2 [ 31 ]	Compact Convolutional ViT	77.72

TABLE IV: Accuracy Comparison with SOTA Models on CIFAR-100 Dataset

Method	Architecture Type	Accuracy (%)
ResNet-18 [ 25 ]	CNN	53.32
ResNet-56 [ 25 ]	CNN	58.77
ViT (baseline) [ 1 ]	Standard ViT	57.82
HSViT-C3A4 [ 29 ]	Modified ViT	56.73
SATA-ViT [ 30 ]	Modified ViT	58.77
HAViT (Ours)	Modified ViT	59.07

TABLE V: Accuracy Comparison with SOTA Models on TinyImageNet Dataset

Explore similar work

JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search
May 26, 2026Dongyun Zou, Zhuoyang Zhang, Junyu Chen +8Efficient Transformer Inference Vision Foundation Models
MIT · NVIDIA · University of Pennsylvania +1
Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores
May 12, 2026Alan Z. Song, Yinjie Chen, Mu Nan +3Visual Representation Learning Vision Transformer
Carnegie Mellon University · University of Hong Kong
HiLRP: Toward One Trustworthy Explanation for Vision Transformer: Conservation-Valid Attribution via Attention Primitives
Sep 1, 2026Sathiyamohan Nishankar, Pubudu Sanjeewani, Asanka Perera +1Transformer Interpretability Vision Transformer
Faculty of Engineering, University of Peradeniya, Sri Lanka · School of Computing Technologies, RMIT University, Melbourne, Australia · School of Engineering & Digital Technologies, University of Southern Queensland, Brisbane, Australia +1

May 26, 2026J/K move · Enter open · S save

JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search

Dongyun Zou, Zhuoyang Zhang, Junyu Chen, Wenkun He+7

MIT · NVIDIA · University of Pennsylvania+1

Efficient Transformer Inference Vision Foundation Models Transformer Attention Efficient ViTs Neural Architecture Search

Open paper AskPDF arXiv

We introduce JetViT, a novel family of hybrid-architecture Vision Transformer (ViT) models that match the accuracy of state-of-the-art full-attention vision foundation models while achieving substantially higher inference efficiency on high-resolution images. At the core of our approach is Post-Training Attention Search, a post-training acceleration framework that converts pre-trained full-attention ViTs into efficient hybrid-attention variants by identifying and replacing redundant full-attention blocks with linear or window-attention blocks. By inheriting the MLP and attention weights from the base model, Post-Training Attention Search efficiently explores the architectural design space through three key steps: (1) optimizing the linear-attention block design; (2) finding the best combination of linear-attention and window-attention blocks; and (3) identifying and preserving critical full-attention blocks. We evaluate JetViT on two representative high-resolution vision foundation models, DINOv3 and DepthAnythingV2. On the NVIDIA H100 GPU, JetViT achieves up to 1.79x higher throughput and up to 44.81% lower latency without sacrificing accuracy. We will release our code and accelerated ViT models soon.

Abstract

Explore similar work

JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search

Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores

HiLRP: Toward One Trustworthy Explanation for Vision Transformer: Conservation-Valid Attribution via Attention Primitives