Where Compute Matters: Heterogeneous Attention for Efficient Video Diffusion

Olga Zatsarynna; Denis Korzhenkov; Juergen Gall; Amir Habibian; Mohsen Ghafoorian

cs.CVSep 25, 2026

Where Compute Matters: Heterogeneous Attention for Efficient Video Diffusion

Authors: Olga Zatsarynna, Denis Korzhenkov, Juergen Gall, Amir Habibian, Mohsen Ghafoorian

Organizations: Qualcomm AI Research · University of Bonn

Abstract

Efficient video generation requires reducing the quadratic cost of self-attention over long spatio-temporal token sequences. Existing efficient-attention methods typically apply the same computation pattern to every token, even though denoising difficulty varies substantially across video regions and evolves throughout the generation process. We introduce HetA-DiT, a heterogeneous attention mechanism that adaptively allocates computation according to token difficulty. A lightweight uncertainty branch predicts a token-wise estimate of denoising difficulty, which is used to route uncertain tokens through dense global attention while processing more reliable tokens with efficient local attention. The resulting routing is content- and timestep-adaptive, retains global context where it matters most, and provides a single parameter for controlling the quality-efficiency trade-off. HetA-DiT is compatible with few-step distribution-matching distillation and introduces no additional Transformer evaluation at inference time by reusing uncertainty estimates from the preceding denoising step. We evaluate the method on DMD-distilled Wan2.2-5B and Wan2.1-1.3B models. Across VBench, VBench-2.0, and human preference evaluation, HetA-DiT maintains competitive generation quality while routing only approximately 20% of tokens through dense attention.

Figures & tables

Figure 1: Spend compute where it counts. Not all video tokens are equally hard to denoise; fine structures demand global context, while smooth regions are handled by cheap local attention. Our heterogeneous attention method incorporates this via an adaptive uncertainty-based token router (TR) that dispatches each token to local or dense attention.

Figure 2: HetA-DiT. Our approach assigns heterogeneous self-attention computation to tokens with different denoising complexities. Given the uncertainty map $u(\tau)$ predicted at step $\tau$ by the dedicated branch, we construct a routing mask $m(\tau)$ by thresholding the predicted uncertainties. Based on this mask $m(\tau)$ and the latent $x(\tau)$ , tokens with low uncertainty are processed using cheaper local self-attention in the following denoising step $\tau-\Delta$ , while high-uncertainty tokens are processed with full attention. The resulting outputs are then used to determine heterogeneous self-attention processing in the next denoising step.

Figure 3: Qualitative comparison of HetA-DiT to baseline DMD . Both models run 3-step denoising. Our method maintains the high quality of the full step-distilled model while being substantially more efficient in an already low-compute regime.

Models (Res. $480\times 832$ )	Tot. $\uparrow$	Hum.Fid. $\uparrow$	Creativ. $\uparrow$	Contr. $\uparrow$	Com. Sense $\uparrow$	Phys. $\uparrow$
Wan2.1 1.3B	56.0	80.7	48.7	34.0	63.4	53.8
Wan2.2 5B	58.1	79.4	57.7	36.8	64.1	51.3
HunyuanVideo	55.3	82.4	41.8	28.6	63.4	60.2
CogVideoX-1.5	53.4	72.1	43.7	29.6	63.2	48.2
DMD2s (2 steps) (Wan2.2)	57.9	76.9	54.2	37.9	59.9	60.9
DMD3s (3 steps) (Wan2.2)	60.8	80.7	58.7	36.2	64.5	63.9

Table 1: Comparison with state-of-the-art video diffusion models on VBench2.0

Table 6

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Radar plot comparing HetA-DiT on VBench [ 12 ] to the state-of-the-art efficient video generation models.

Figure 6: Radar plot comparing HetA-DiT on VBench-2.0 [ 13 ] to the state-of-the-art efficient video generation models.

Random %	Total $\uparrow$	Quality $\uparrow$	Semantic $\uparrow$
20% (all)	82.78	83.37	80.45
7%	83.27	83.71	81.52
3%	83.18	83.63	81.39
0%	83.00	83.48	81.25

Appendix

Table 8: Ablation study of random token percentage . Adding a random swapping component to uncertainty-based routing increases dense-attention coverage, leading to improved results.

Figure 7: Qualitative comparison of HetA-DiT to the state-of-the-art efficient video generation models. Prompt : A 3D model of a 1800s victorian house.

Figure 8: Qualitative comparison of HetA-DiT to the state-of-the-art efficient video generation models. Prompt : A beautiful coastal beach in spring, waves lapping on sand, Van Gogh style.

Figure 9: Qualitative comparison of HetA-DiT to the state-of-the-art efficient video generation models. Prompt : A person is playing flute.

Figure 10: Qualitative comparison of HetA-DiT to the state-of-the-art efficient video generation models. Prompt : a giraffe .

Figure 11: Qualitative comparison of HetA-DiT to the state-of-the-art efficient video generation models. Prompt : A jellyfish floating through the ocean, with bioluminescent tentacles .

Figure 12: Example of the predicted uncertainty map at diffusion timestep $\sigma=0.6$ .

Figure 13: Example of the predicted uncertainty map at diffusion timestep $\sigma=0.6$ .

Figure 14: Example of the predicted uncertainty map at diffusion timestep $\sigma=0.6$ .

Figure 15: Example of the predicted uncertainty map at diffusion timestep $\sigma=0.6$ .

Figure 16: Example of the predicted uncertainty map at diffusion timestep $\sigma=0.6$ .

Video Diffusion Models

Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.

See all 710 papers

Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation
May 7, 2026Ernie Chu, Vishal M. PatelVideo Diffusion Models Diffusion Transformer
Johns Hopkins University Baltimore, MD 21218
SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation
Aug 17, 2026Animesh Karnewar, Denis Korzhenkov, Amirhossein Habibian +1Video Diffusion Models Efficient Attention
Qualcomm AI Research
Token Radius Attention for Efficient Video Generation
Aug 3, 2026Jiayu Chen, Zhikun Jiang, Maoliang Li +6Video Diffusion Models Sparse Attention
School of Computer Science, Peking University · School of Electronics Engineering and Computer Science, Peking University · School of Computer Science and Engineering, Beihang University +2

May 7, 2026J/K move · Enter open · S save

Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation

Ernie Chu, Vishal M. Patel

Johns Hopkins University Baltimore, MD 21218

Video Diffusion Models Diffusion Transformer Diffusion Model Caching Diffusion Timestep Scheduling Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Diffusion Transformers (DiTs) have achieved state-of-the-art video generation quality, but they incur immense computational cost because standard inference applies the same number of denoising steps uniformly to every token in the sequence. It is well known that human vision ignores vast amounts of redundant motion. Why, then, do our densest models treat every spatiotemporal token with equal priority? In this paper, we introduce Heterogeneous Step Allocation (HSA), a training-free inference algorithm that assigns varying step budgets to different spatiotemporal tokens based on their velocity dynamics. To resolve the resulting sequence-length mismatch without sacrificing global context, HSA introduces a KV-cache synchronization mechanism that allows active tokens to attend to the full sequence while entirely bypassing inactive tokens. Furthermore, we derive a cached Euler update that advances the latent states of skipped tokens in a single operation without additional model evaluations. We evaluate HSA on the Wan-2 and LTX-2 models for both text-to-video (T2V) and image-to-video (I2V) generation. Our results demonstrate that HSA significantly outperforms previous state-of-the-art caching methods and the vanilla Flow Matching baseline, especially at aggressive acceleration regimes (e.g., 50% and 25% runtimes). Crucially, HSA achieves a superior quality-runtime Pareto frontier without the need for expensive offline profiling, robustly preserving structural integrity and generation quality even under tight computational budgets. Project page: https://ernestchu.github.io/hsa

Abstract

Video Diffusion Models

Explore similar work

Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation

SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation

Token Radius Attention for Efficient Video Generation