cs.CVNov 9, 2026

Beyond Head Importance: Collective Computation in Video Transformers

Authors: Yi Yang, Siyuan Liu, Bingbing Nie

Organizations: School of Vehicle and Mobility, Tsinghua University, China · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University, China

Abstract

Attention heads in video Transformers vary greatly in functional importance. Understanding their contributions requires examining not only which heads are important, but also how they participate in collective computation within the network. We study this collective computation from two perspectives: the downstream use of head outputs and the composition of interactions across layers. Ablation analyses of four video Transformers show that high-importance heads identified on a single task also support other video understanding tasks and help preserve the representational structure that distinguishes different videos. Representational responses to head ablation correlate weakly with head importance after the attention module, but more strongly after the MLP in the same block. Adapting this downstream processing through MLP-only fine-tuning further improves the use of lower-importance head outputs. Meanwhile, through composition across layers, sets of heads with shorter spatiotemporal attention distances support stronger perturbation responses at distant tokens than sets with longer distances. Their joint ablation causes even greater performance loss than ablating equally sized sets of high-importance heads, revealing collective contributions not fully captured by individual importance rankings. These findings highlight collective computation as a key perspective for understanding attention head function: functional contributions depend on both how head outputs are used downstream and how local interactions compose across layers.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Brain-Aligned Multi-Stream Video Transformers with Sparse Self-Selection

    Jul 20, 2026Amir Hosein Fadaei, Mahyar Maleki, Mohammad-Reza A. DehaqaniTransformerEfficient ViTs

  2. Temporal-Attention Head Specialization During Video Diffusion Training

    Sep 14, 2026Taewoo Ha, Shafayat Mowla Anik, Dae Yeol Lee +2Video Diffusion ModelsAttention Head Analysis

  3. AB-Attention: Accurate Block Attention for Accelerating Video Diffusion Transformers

    Oct 29, 2026Bin Xu, Mo Zhang, Zhengfeiyang Zhang +6Diffusion Model Inference AccelerationBlock-Sparse Attention