Beyond Head Importance: Collective Computation in Video Transformers
Organizations: School of Vehicle and Mobility, Tsinghua University, China · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University, China
Abstract
Attention heads in video Transformers vary greatly in functional importance. Understanding their contributions requires examining not only which heads are important, but also how they participate in collective computation within the network. We study this collective computation from two perspectives: the downstream use of head outputs and the composition of interactions across layers. Ablation analyses of four video Transformers show that high-importance heads identified on a single task also support other video understanding tasks and help preserve the representational structure that distinguishes different videos. Representational responses to head ablation correlate weakly with head importance after the attention module, but more strongly after the MLP in the same block. Adapting this downstream processing through MLP-only fine-tuning further improves the use of lower-importance head outputs. Meanwhile, through composition across layers, sets of heads with shorter spatiotemporal attention distances support stronger perturbation responses at distant tokens than sets with longer distances. Their joint ablation causes even greater performance loss than ablating equally sized sets of high-importance heads, revealing collective contributions not fully captured by individual importance rankings. These findings highlight collective computation as a key perspective for understanding attention head function: functional contributions depend on both how head outputs are used downstream and how local interactions compose across layers.
Figures & tables
| Model | Ablated heads | Action | Object | Direction | Outcome | Top 25% | |
|---|---|---|---|---|---|---|---|
| V-JEPA | Top 25% | 57 | |||||
| Bottom 25% | |||||||
| VideoMAE | Top 25% | 46 | |||||
| Bottom 25% | |||||||
| V-JEPA2 | Top 25% | 51 | |||||
| Bottom 25% |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Symbol | Description | References |
|---|---|---|---|
| Spatiotemporal attention range | Attention-weighted mean spatial distance | [ 16 ] | |
| Attention-weighted mean temporal distance | [ 7 ] | ||
| Fraction of attention weight assigned to other temporal slices | [ 61 ] | ||
| Fraction of attention weight within the same-slice neighborhood | [ 37 ] | ||
| Attention weight structure | Deviation of the mean attention matrix from a shared kernel | [ 15 ] | |
| Consistency of relative attention patterns across query positions | [ 34 ] |
Attention Head Analysis
Attention head analysis studies how individual Transformer heads route information, support computations, and affect model behavior.
See all 89 papers