Beyond Head Importance: Collective Computation in Video Transformers
Authors: Yi Yang, Siyuan Liu, Bingbing Nie
Organizations: School of Vehicle and Mobility, Tsinghua University, China · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University, China
Attention heads in video Transformers vary greatly in functional importance. Understanding their contributions requires examining not only which heads are important, but also how they participate in collective computation within the network. We study this collective computation from two perspectives: the downstream use of head outputs and the composition of interactions across layers. Ablation analyses of four video Transformers show that high-importance heads identified on a single task also support other video understanding tasks and help preserve the representational structure that distinguishes different videos. Representational responses to head ablation correlate weakly with head importance after the attention module, but more strongly after the MLP in the same block. Adapting this downstream processing through MLP-only fine-tuning further improves the use of lower-importance head outputs. Meanwhile, through composition across layers, sets of heads with shorter spatiotemporal attention distances support stronger perturbation responses at distant tokens than sets with longer distances. Their joint ablation causes even greater performance loss than ablating equally sized sets of high-importance heads, revealing collective contributions not fully captured by individual importance rankings. These findings highlight collective computation as a key perspective for understanding attention head function: functional contributions depend on both how head outputs are used downstream and how local interactions compose across layers.
Figures & tables
Figure 1: Task performance under different attention head ablation orders. Solid lines indicate cumulative ablation from lowest to highest importance (Bottom-first), while dashed lines indicate ablation from highest to lowest importance (Top-first).
Model
Ablated heads
Action
Object
Direction
Outcome
ρObj,Act
Top 25% ∩
V-JEPA
Top 25%
−34.3
−30.6
−31.1
−19.3
0.61
57
Bottom 25%
+3.9
+0.8
+1.8
+2.5
VideoMAE
Top 25%
−25.7
−51.6
−10.0
−11.8
0.43
46
Bottom 25%
+6.1
−5.2
+2.5
+2.1
V-JEPA2
Top 25%
−27.9
−26.6
−20.7
−16.1
0.43
51
Bottom 25%
+12.9
0.0
+3.2
+2.5
Table 1: Cross-task ablation results for head sets of varying importance. Values indicate accuracy changes relative to Intact (in percentage points, pp) after ablating the top or bottom 25% of heads. The last two columns report the Spearman correlation between the Action and Object importance rankings and the overlap between their top 25% head sets (expected overlap by chance: 24).
Figure 2: Representational structure under head sets with different importance levels. (a) Cross-layer CKA matrices between intervened and Intact representations. (b) Shared UMAP projections before and after ablation, using an Intact-fitted mapping. (c) Layer-wise mean pairwise cosine similarity across videos. (d) Spearman correlation of inter-video distance rankings and 10-NN Jaccard similarity, both relative to Intact.
Figure 3: Head importance prediction and property contributions. (a) Spearman correlation between predicted and actual importance rankings for unseen heads, using different numbers of properties. Results are evaluated over 10 repetitions of six-fold cross-validation; intervals indicate variability across data splits. (b) Decrease in prediction correlation after randomly permuting each property individually when using seven properties.
Figure 4: Downstream MLP processing and the functional importance of attention heads. (a) Measurement points for ablation-induced representational changes Δattn and Δmlp . (b) Relationships between ablation-induced representational changes and head importance ranks. Each point represents a head. Spearman correlations ρ and p -values are annotated. (c) Absolute changes in within-block head importance ranks after ablating the MLP in the same block, shown for six blocks. Box plots show the distributions, and white diamonds indicate the means. (d) Layer-wise accuracy on Action and Object under different head retention and MLP update conditions.
Figure 5: Perturbation responses under different head sets. (a) Final-layer response distributions under three conditions. Red crosses mark the perturbation source, whose position is left blank. (b) Layer-wise ratios of Local half to Global half response magnitudes at the 52 boundary tokens in the temporal slice containing the perturbation source. Scatter points show per-token ratios, and the line shows their mean.
Figure 6: Performance loss from ablating head sets and recovery with fixed routing. (a) Cumulative ablation results with heads ranked by importance, spatial attention distance, or temporal attention distance. (b) Cross-tabulations of quartile groups for each pair of metrics. Q1 through Q4 correspond to importance from highest to lowest and attention distance from shortest to longest. Numbers indicate head counts in the corresponding groups. (c) Two example attention distributions, one for a spatially local head and one for a temporally local head. Crosses mark query positions; heatmaps and bar plots show the query token’s mean spatial and temporal attention distributions, respectively. (d) Action accuracy loss (pp) under mean-replacement ablation and replacement with fixed spatiotemporal kernels for three head sets: the top 25% by importance, the most spatially local 25%, and the most temporally local 25%. Arrows indicate the fraction of the ablation-induced loss recovered.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Symbol
Description
References
Spatiotemporal attention range
ds
Attention-weighted mean spatial distance
[ 16 ]
dt
Attention-weighted mean temporal distance
[ 7 ]
rcross
Fraction of attention weight assigned to other temporal slices
[ 61 ]
rlocal
Fraction of attention weight within the same-slice neighborhood
[ 37 ]
Attention weight structure
econv
Deviation of the mean attention matrix from a shared kernel
[ 15 ]
cquery
Consistency of relative attention patterns across query positions
[ 34 ]
Appendix
Table 2: Ten computational properties of attention heads. Symbols, descriptions, and related work for each property. Detailed definitions are provided in Appendix B .
Figure 7: Head importance distributions on Action and Object across four models. Importance is measured as the mean accuracy drop across random joint ablation trials containing the head. In each heatmap, heads within each block are sorted by decreasing importance. The curves below show the mean importance within each block.
Figure 8: Cross-task cumulative ablation results across models. We perform cumulative ablation using the importance ranking derived from Action, ablating heads from highest to lowest importance (Top-first) or lowest to highest importance (Bottom-first).
Figure 9: Video representations with the top or bottom half of heads retained. The first two columns show cross-layer CKA matrices comparing Intact with models retaining only the top or bottom 50% of heads by importance. The last two columns show shared UMAP projections of final-layer video representations, with lines connecting the positions of the same video before and after intervention.
Figure 10: Head importance prediction and property contributions across models. The top row shows Spearman correlations between predicted and actual importance ranks for unseen heads as a function of the number of properties. Lines indicate means, and shaded regions show the 2.5th–97.5th percentile intervals across data splits. The bottom row shows permutation importance using seven properties, measured as the decrease in predictive correlation caused by randomly permuting a single property.
Figure 11: Spatiotemporal attention range and head importance across models. The first two rows show the layerwise distributions of mean spatial attention distance ds and mean temporal attention distance dt , respectively. Points represent individual heads, and lines indicate the mean within each block. The third row shows the joint distribution of the two distances. Point colors indicate head importance on Action, measured as the mean accuracy drop across random joint ablation trials containing the head.
Figure 12: Attention patterns across different spatiotemporal attention ranges. The four columns show V-JEPA heads that are temporally global/spatially global, temporally local/spatially global, temporally global/spatially local, and temporally local/spatially local, respectively. The first row shows the full attention matrices, with queries and keys grouped by temporal slice in temporal order and spatial tokens within each slice flattened into a contiguous sequence. Grid lines mark boundaries between temporal slices. The remaining three rows show the attention distributions across temporal slices for three query positions, with green crosses marking the query positions.
Figure 13: Cumulative ablation of head sets across models and tasks. Heads are ranked by importance on Action or by mean spatial or temporal attention distance. The first and third columns ablate high-importance or local heads first, while the second and fourth columns ablate low-importance or global heads first.
Modern video transformers typically ignore principles from primate vision and are rarely evaluated against neural data, limiting their biological interpretability. We introduce a sparse winner-takes-all token selection module that replaces dense self-attention to improve efficiency and approximate competitive routing observed in biological visual circuits. We further propose a neuro-inspired split-and-fuse video transformer which uses two complementary pathways: a high-resolution, low-frame-rate "what" stream and a low-resolution, high-frame-rate "where" stream, fused before classification. On Kinetics-400 and Something-Something V2, our best variant operates on the Pareto frontier of accuracy versus inference time among models of comparable scale and pretraining, and showing improved robustness to spatial perturbations. Using representational similarity analysis between model embeddings and time-resolved EEG recordings for the same video stimuli, our model attains a peak brain-model correlation of 0.18 (about 78% of the noise ceiling) and consistently outperforms strong video transformer baselines, suggesting that pathway specialization and sparse competition are useful inductive biases for efficient, brain-aligned video understanding.
Amir Hosein Fadaei, Mahyar Maleki, Mohammad-Reza A. Dehaqani
University of Tehran, Faculty of Electrical and Computer Engineering, North Kargar st., Tehran, 1439957131, Tehran, Iran
Video diffusion transformers depend on temporal attention to coordinate information across frames, yet nearly everything known about this mechanism comes from analyzing trained models, so when and where temporal-attention structure forms during training remains poorly characterized. Population averages can also hide it, since a few specializing heads and a diffusing majority cancel in the mean. We therefore conduct a checkpoint-resolved census of every temporal-attention head across nine Open-Sora STDiT training runs spanning three model scales (306M to 1.03B parameters), scoring each head with an entropy-normalized measure of cross-frame attention concentration (CFAC) under a preregistered change-point and effect-size selection rule. The census reveals the sparse picture that averages obscure. Aggregate CFAC is flat or decreasing in every run, while a small minority of heads, roughly 4--13% in full-grid runs, develops pronounced concentration. Across seeds, the reproducible signal is positional but block-level. Selected heads repeatedly arise in the first temporal block, whereas individual head coordinates do not reproduce once block membership is accounted for. Among the analyzed 760M selected heads, attention maps converge to a small repertoire of local frame-routing motifs, self-frame diagonals and adjacent-frame bands, even when the responsible coordinates differ across runs. Correlation and ablation analyses do not establish a causal link to generated video quality, and we bound our claims accordingly. Beyond this STDiT family, the study contributes a transferable methodology. Checkpoint-resolved, per-head analysis under fixed selection rules can expose sparse temporal organization in other factorized video diffusion transformers and, with adapted routing metrics, in joint spatio-temporal architectures.
Taewoo Ha, Shafayat Mowla Anik, Dae Yeol Lee +2
Independent Researcher · University of Colorado - Colorado Springs · Dolby Laboratories +1
Video diffusion transformers generate high-quality videos, but the quadratic cost of 3D full attention dominates their inference. Dynamic block-sparse attention reduces this cost by computing exact attention only on key/value blocks selected from current activations. However, its fidelity degrades under aggressive sparsity: blocks are scored from coarse summaries or formed by costly global clustering, pruned blocks vanish from the softmax, and whole tiles are kept or dropped even when only a few of their keys matter. To address these limitations, we propose Accurate Block Attention (AB-Attention), a training-free method that accurately approximates dense attention at high sparsity. Its Intra-Block Clustering Selector (IBCS) estimates block importance faithfully yet cheaply by clustering queries only within each query block, without permuting tokens. Sampling with Inclusion-Probability Weighting (SIP) then samples blocks in proportion to these estimates and reweights them by inverse inclusion probability, so that one pass estimates dense attention rather than truncating it. Finally, its kernel gathers key/value blocks much smaller than a tile into full tensor-core tiles, refining selection at no extra tensor-core cost. On Wan2.1 and HunyuanVideo, AB-Attention skips 89--92% of attention, whereas the baselines compute 1.8--3.6× as much, yet it still improves PSNR by up to 4.4~dB over the strongest baseline and achieves up to 2.23× end-to-end speedup. Our code is available at https://github.com/InfRec-Cardinal/AB-Attention.
Bin Xu, Mo Zhang, Zhengfeiyang Zhang +6
InfRec · The Chinese University of Hong Kong, Shenzhen · Shandong University +5