Steering Sparse Autoencoder Latents to Control Dynamic Head Pruning in Vision Transformers (Student Abstract)

Yousung Lee; Dongsoo Har

cs.CVMar 23, 2026

Steering Sparse Autoencoder Latents to Control Dynamic Head Pruning in Vision Transformers (Student Abstract)

Authors: Yousung Lee, Dongsoo Har

Organizations: Korea Advanced Institute of Science and Technology, Daejeon 34051, Korea

Abstract

Dynamic head pruning in Vision Transformers (ViTs) improves efficiency by removing redundant attention heads, but existing pruning policies are often difficult to interpret and control. In this work, we propose a novel framework by integrating Sparse Autoencoders (SAEs) with dynamic pruning, leveraging their ability to disentangle dense embeddings into interpretable and controllable sparse latents. Specifically, we train an SAE on the final-layer residual embedding of the ViT and amplify the sparse latents with different strategies to alter pruning decisions. Among them, per-class steering reveals compact, class-specific head subsets that preserve accuracy. For example, bowl improves accuracy (76% to 82%) while reducing head usage (0.72 to 0.33) via heads h2 and h5. These results show that sparse latent features enable class-specific control of dynamic pruning, effectively bridging pruning efficiency and mechanistic interpretability in ViTs.

Figures & tables

Figure 1: Comparison of dynamic head pruning before (left) and after (right) SAE latent steering, where $x$ denotes the CLS token from the final-layer residual stream input, and $\tilde{x}$ denotes the steered embedding.

Figure 2: Accuracy (%) and head usage ratio in the final layer under different strategies as $\alpha$ increases from $0$ to $1.5$ .

Figure 3: Effect of per-class steering ( $\alpha=1.2$ ) on head activation patterns. Columns indicate the top- $5$ classes ranked by accuracy gain, and rows correspond to heads ( $h_{0}$ – $h_{5}$ ).

Figure 4: Negative steering ( $\alpha<0$ ) increases head usage by suppressing pruning signals.

Panel of Figure 5 — Figure 5: Semantic structure of SAE latents, showing class-wise overlap and distinct latent usage patterns.

Attention Head Pruning

Attention head pruning removes redundant Transformer attention heads to reduce model size, computation, memory use, or inference latency.

See all 22 papers

Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers
Jul 31, 2026Kamil Książek, Piotr Suszyński, Michał Jan Włodarczyk +2Transformer Interpretability Attention Head Analysis
Centre for Credible Artificial Intelligence Warsaw University of Technology Warsaw, Poland · Faculty of Mathematics and Computer Science Jagiellonian University Krakow, Poland · University of Warsaw Warsaw, Poland
HiAP: A Multi-Granular Stochastic Auto-Pruning Framework for Vision Transformers
Mar 12, 2026Andy Li, Aiden Durrant, Milan Markovic +1Efficient ViTs Attention Head Pruning
School of Natural and Computing Sciences, University of Aberdeen, UK · School of Computing Sciences, University of East Anglia, UK · Department of Physics and Technology, UiT The Arctic University of Norway, Norway
SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable
Aug 14, 2026Philip H. Lee, Parth PadalkarExplainable Image Classification Vision Transformer
Independent Researcher · Texas State University

Jul 31, 2026J/K move · Enter open · S save

Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers

Kamil Książek, Piotr Suszyński, Michał Jan Włodarczyk, Jacek Tabor+1

Centre for Credible Artificial Intelligence Warsaw University of Technology Warsaw, Poland · Faculty of Mathematics and Computer Science Jagiellonian University Krakow, Poland · University of Warsaw Warsaw, Poland

Transformer Interpretability Attention Head Analysis Vision Transformer Efficient ViTs Attention Head Pruning Attention Head Specialization

Open paper AskPDF arXiv

Vision foundation models, such as DINOv2, learn highly expressive representations but rely on massive, opaque architectures that demand substantial computational power and memory. To provide an interpretable-guided and efficient solution to this issue, we first propose a spectral analysis and new visualization technique for individual attention heads based on the Laplacian eigenvectors of their attention maps. Building upon recent observations regarding the block structure of Vision Transformers, we perform semantic clustering of attention heads and identify functional redundancies. Leveraging these insights, we introduce SAPER (Soft Attention PrunER), an end-to-end differentiable pruning framework based on the LapSum Soft Top-K approach. Extensive experiments on ImageNet-1K demonstrate that SAPER achieves a highly favorable accuracy-efficiency trade-off, outperforming the competitive RAPTOR baseline in FLOPs reduction while preserving strong classification performance.

Abstract

Attention Head Pruning

Explore similar work

Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers

HiAP: A Multi-Granular Stochastic Auto-Pruning Framework for Vision Transformers

SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable