Organizations: State Key Laboratory for Novel Software Technology, Nanjing University · Kling Team, Kuaishou Technology · School of Intelligence Science and Technology, Nanjing University
The quadratic complexity of the attention mechanism severely limits the context scalability of Video Diffusion Transformers (DiTs). We find that the highly sparse spatio-temporal attention patterns exhibited in Video DiTs can be naturally represented by the Monarch matrix. It is a class of structured matrices with flexible sparsity, enabling sub-quadratic attention via an alternating minimization algorithm. Accordingly, we propose VMonarch, a novel attention mechanism for Video DiTs that enables efficient computation over the dynamic sparse patterns with structured Monarch matrices. First, we adapt spatio-temporal Monarch factorization to explicitly capture the intra-frame and inter-frame correlations of the video data. Second, we introduce a recomputation strategy to mitigate artifacts arising from instabilities during alternating minimization of Monarch matrices. Third, we propose a novel online entropy algorithm fused into FlashAttention, enabling fast Monarch matrix updates for long sequences. Extensive experiments demonstrate that VMonarch achieves comparable or superior generation quality to full attention on VBench after minimal tuning. It overcomes the attention bottleneck in Video DiTs, reduces attention FLOPs by a factor of 17.5, and achieves a speedup of over 5x in attention computation for long videos, surpassing state-of-the-art sparse attention methods at 90% sparsity. Code is available at https://github.com/EricLina/VMonarch.
Figures & tables
Figure 1: Video Monarch Attention (VMonarch) leverages dynamically updated spatial-temporal Monarch matrices [ 11 ] to model sparse attention patterns in video DiTs. When generating a 321-frame video at 448 × 832 resolution (approximately 118K tokens), VMonarch speeds up self-attention computation by 17.5 × compared to full attention (FA2) [ 8 ] and by 2 × compared to highly sparse alternatives like Video Sparse Attention (VSA) at 90% sparsity. After minimal fine-tuning, VMonarch attains comparable video generation quality to FA2.
Figure 2: Overview of our Video Monarch Attention. (a) We represent the N×N full attention matrix via two alternatingly optimized smaller Monarch factors L and R with spatial-temporal structure factorization. (b) We introduce a Recomputation Strategy to address quality degradation caused by excessively large temperature term cR for the first frame. (c) We propose an Online-Entropy algorithm integrated with FlashAttention to accelerate the iterative computation of Monarch matrices further.
Figure 3: Illustration of the attention sink phenomenon in video models. The first frame tends to accumulate excessive attention from subsequent frames, leading to a loss of fine-grained details when using Monarch Attention. Our First-Frame Recomputation Strategy effectively restores these details by recalculating the attention for the first frame using full attention.
Model
Resolution
Kernel
Quality
Efficiency
AQ ↑
BC ↑
DD ↑
IQ ↑
SC ↑
Sparsity ↑
TFLOPs ↓
Time (s) ↓
Wan2.1 1.3B
61x448x832
FullAttn ⊘
65.44%
95.08%
69.44%
64.86%
91.77%
-
159.7
63.4
VMoBA ⊘
42.00%
90.64%
88.89%
53.78%
82.00%
90.0%
75.8
71.7
VSA ⊘
42.86%
94.96%
63.89%
55.04%
89.17%
90.0%
69.5
49.9
VMonarch ⊘
63.65%
94.93%
54.17%
64.46%
92.43%
87.5%
75.4
47.7
FullAttn
66.07%
95.75%
68.06%
64.57%
94.15%
-
159.7
63.4
Table 1: Quality and efficiency comparison of our and other methods, including Full Attention [ 8 ] , VSA [ 69 ] , and VMoBA [ 50 ] . We verify the generalizability of our method across different architectures, parameter scales, and varying resolutions under the same tuning setting. We report Vbench [ 21 ] metrics to evaluate quality, including Aesthetic Quality (AQ), Background Consistency (BC), Dynamic Degree (DD), Imaging Quality (IQ), and Subject Consistency (SC). We also report the efficiency metrics, including attention sparsity, TFLOPs, and inference time. ⊘ denotes Training-Free setting. ⋆ denotes extrapolation on longer sequences using the Wan2.1-1.3B [ 47 ] model tuned at 61×448×832 resolution. We highlight the results of VMonarch if it is the best or second best among efficient attention methods.
Model
Similarity
Vbench Score (%)
PSNR ↑
SSIM ↑
AQ
BC
DD
IQ
SC
Softmax
-
-
65.44
95.08
69.44
64.86
91.77
VM-T1-F1
11.18
0.35
54.36
89.75
16.67
54.93
93.17
VM-T2-F1 †
11.65
0.40
63.39
92.25
29.17
67.76
88.46
VM-T2-F1
12.59
0.43
63.65
94.93
54.17
64.46
92.43
VM-T3-F1
12.21
0.41
63.89
95.01
50.00
65.12
93.55
Table 2: Ablation studies on training-free setting. In ’VM-T n -F k ’, T represents the number of iterations, and F denotes the size of the Monarch factor b corresponding to the number of frames. The † indicates the model variant without first frame computation.
Figure 4: Qualitative samples of our models. We compare the generation quality of the tuned models between full attention [ 8 ] , Video Sparse Attention (VSA) [ 69 ] , and our VMonarch Attention.
Figure 5: The kernel speedup ratio over Flash Attention 2 (FA2) [ 8 ] under varying sequence lengths. Our VMonarch demonstrates superior efficiency, achieving over 2 × speedup at 28k tokens and over 5 × speedup at 62k tokens compared to FA2, surpassing other sparse attention methods at 90% sparsity.
Model
Vbench Score (%)
Val
AQ
BC
DD
IQ
SC
Loss
Softmax
65.72
94.85
75.00
64.46
92.59
1.09
VM-T1-F1
66.53
95.82
59.72
65.28
92.69
1.27
VM-T2-F1
64.84
95.28
68.06
63.83
93.29
1.25
VM-T3-F1
66.22
96.12
56.94
65.72
94.06
1.24
VM-T5-F1
65.63
96.16
45.83
67.91
94.94
1.57
Table 3: Ablation studies on fine-tuned Video Monarch Attention models under different configurations. In ’VM-T n -F k ’, T represents iterations numbers, and F denotes the frame number corresponding to the Monarch Matrix b . † omits the recompute strategy for the first frame. ‡ means leveraging attention sink by averaging.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A : Various computational budgets compared with FA2 at a sequence length of 215 . In practice, b is usually larger than 256 .
Figure 8: Speed comparison of FlashAttention2(FA2) [ 8 ] , VMonarch-Naive, and our optimized VMonarch-Fast (Triton) implementation. After applying kernel optimization, VMonarch-Fast achieves an approximately 8× speedup over the naive implementation and surpasses the performance of FA2.
Figure 9: Memory benchmark of different attention mechanisms. We compare the relative memory usage against full attention.
Figure 10: Speed benchmark of different attention mechanisms. We compare the relative speed against full attention.
Method
Similarity
PSNR ↑
SSIM ↑
FlashAttention2 (FullAttn) [ 8 ]
-
-
Video Sparse Attention (VSA) [ 69 ]
9.71
0.32
Video MoBA (VMoBA) [ 50 ]
10.61
0.30
Video Monarch (VMonarch)
12.59
0.43
Appendix
Table 4: Similarity comparison of our and other baseline methods. All methods are evaluated under the training-free setting with a resolution of 61×448×832 .
Figure 11: Attention map comparison among Full Attention, Sparse Attention, and Monarch Attention.
Exp.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR1↑
SSIM1↑
LPIPS1↓
IQ ↑
SC ↑
OC ↑
Score ↑
1
M (1)
11.01
0.21
96.41%
10.21
0.12
90.44%
48.54%
87.00%
16.28%
2.19
2
M (1) + cR∗ + cL1
10.66
0.23
80.39%
11.21
0.30
51.54%
62.22%
87.53%
18.85%
3.11
3
M (1) + cR1 + cL1
10.66
0.23
80.39%
11.21
0.30
51.54%
62.22%
87.53%
18.85%
3.11
4
M (1) + cR1 + cL∗
11.01
0.18
99.78%
10.29
0.11
95.63%
41.32%
85.87%
15.69%
1.98
5
M (1) + cR∗ + cL∗
11.01
0.18
99.78%
10.29
0.11
95.63%
41.32%
85.87%
15.69%
1.98
6
M (1) + cR1 + cL0
11.01
0.21
96.41%
10.21
0.12
90.44%
48.54%
87.00%
16.28%
2.19
Appendix
Table 5: Ablation study on VMonarch Attention. We evaluate several variants by adjusting its iterative parameters, including the number of iterations and the manual control of intermediate terms. Metrics with a superscript ’1’ (e.g., PSNR1 ) are calculated on the first frame only. The score is calculated as Score=(PSNR+PSNR1)/200+(SSIM+SSIM1)/2+(1−LPIPS+1−LPIPS1)/2+IQ+SC+OC , where IQ, SC, and OC denote Imaging Quality, Subject Consistency, and Overall Consistency, respectively. In the Method column, M (t) denotes the number of iterations t . +Re represents our recomputation mechanism. For the control of cR , + cR∗ indicates fixing it to 1 for all frames, + cR1 for the first frame only, and + cR0 means no fixing. For the control of cL , + cL∗ indicates fixing it to the initial expected value log(M) for all frames, where M is the monarch factor size, + cL1 for the first frame only, and + cL0 means no fixing.
Figure 12: Comparison of different Monarch factorizations.
Figure 13: More visual results comparing Full Attention, VSA, and VMonarch on various prompts.
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention. Sparse attention mitigates this by skipping unnecessary computation in the attention matrix, but existing methods apply a fixed sparsity ratio throughout denoising and struggle to accurately determine which computations to skip. To address these limitations, we analyze attention dynamics through attention entropy and mean attention distance, and find that attention becomes increasingly concentrated across denoising steps. We further observe that blocks selected early continue to capture a large fraction of the total attention weight at later steps. These findings suggest two acceleration opportunities: the sparsity ratio should grow as denoising progresses, and earlier search results can be reused to reduce the search space. Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead. GSRS adopts Top-p selection for adaptive sparsity control, supported by a precise evaluation scheme that computes exact attention weights within the FlashAttention computation at negligible overhead. Experiments on HunyuanVideo and Wan 2.1 validate that GSRS achieves 1.55×--2.21× speedup while maintaining generation quality comparable to full attention.
The quadratic computational complexity of self-attention mechanisms imposes a critical constraint on transformer-based video generation, particularly for synthesizing ultra-long and high-definition sequences. While most sparse attention techniques employ fixed or heuristic patterns, they often struggle to accommodate the heterogeneous and layer-specific redundancies inherent in video diffusion transformers (VDiTs). We find that different attention heads consistently concentrate most of their attention mass in different coarse query-key regions, a behavior we call functional specialization. Although individual attention values vary with the generated content, these regions remain similar across prompts and random seeds. Leveraging these insights, we introduce Compact Attention, a hardware-aware sparse attention framework for efficient long-video synthesis. We quantify this stability and use it to construct searchable, reusable, head-specific masks. Based on the observation, we propose a deformable dual-window sparse pattern mechanism that flexibly approximates local and axial attention regions within one tile-aligned mask family. To avoid manually choosing a mask for each head, we develop a recall-driven offline auto-search algorithm that identifies high-quality sparse masks while leveraging temporal stability for mask reuse. Extensive evaluations on Wan2.1 (14B) and Hunyuan-Video demonstrate that Compact Attention achieves 2.25x/3.03x attention speedups over FlashAttention-2 and 1.20x/1.74x over FlashAttention-3, respectively, while maintaining visual quality and temporal consistency comparable to full-attention baselines. Project Page: https://yo-ava.github.io/Compact-Attention.github.io/
Qirui Li, Guangcong Zheng, Qi Zhao +6
College of Computer Science & Technology, Zhejiang University · Huawei Technologies
Scaling Diffusion Transformers to generate high-resolution, long videos is constrained by the quadratic cost of self-attention, and existing sparse attention methods degrade under high sparsity. We show empirically that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full attention. Based on this insight, we propose Veda, a distilled sparse attention framework that formulates tile selection as an explicit reconstruction problem from full attention. Veda integrates statistics-aware tile scoring with head-aware tiling to reduce estimation error and structural mismatch, enabling aggressive sparsity. A hardware-efficient tile-skipping kernel converts theoretical sparsity into practical wall-clock speedups. Experiments on large video diffusion models, including Waver and Wan2.1, demonstrate substantial acceleration with no noticeable degradation in generation quality. To generate 720P 10-second videos on Waver-T2V-12B, Veda achieves a 5.1× end-to-end speedup and a 10.5× self-attention speedup, reducing attention overhead from 92% to 50%. Notably, the gains increase with sequence length, indicating that Veda scales favorably with spatiotemporal resolution across models.
Shihao Han, Hao Yang, Xinting Hu +3
ByteDance Inc. · University of Science and Technology of China · The University of Hong Kong