Organizations: State Key Laboratory for Novel Software Technology, Nanjing University · Kling Team, Kuaishou Technology · School of Intelligence Science and Technology, Nanjing University
The quadratic complexity of the attention mechanism severely limits the context scalability of Video Diffusion Transformers (DiTs). We find that the highly sparse spatio-temporal attention patterns exhibited in Video DiTs can be naturally represented by the Monarch matrix. It is a class of structured matrices with flexible sparsity, enabling sub-quadratic attention via an alternating minimization algorithm. Accordingly, we propose VMonarch, a novel attention mechanism for Video DiTs that enables efficient computation over the dynamic sparse patterns with structured Monarch matrices. First, we adapt spatio-temporal Monarch factorization to explicitly capture the intra-frame and inter-frame correlations of the video data. Second, we introduce a recomputation strategy to mitigate artifacts arising from instabilities during alternating minimization of Monarch matrices. Third, we propose a novel online entropy algorithm fused into FlashAttention, enabling fast Monarch matrix updates for long sequences. Extensive experiments demonstrate that VMonarch achieves comparable or superior generation quality to full attention on VBench after minimal tuning. It overcomes the attention bottleneck in Video DiTs, reduces attention FLOPs by a factor of 17.5, and achieves a speedup of over 5x in attention computation for long videos, surpassing state-of-the-art sparse attention methods at 90% sparsity. Code is available at https://github.com/EricLina/VMonarch.
Figures & tables
Figure 1: Video Monarch Attention (VMonarch) leverages dynamically updated spatial-temporal Monarch matrices [ 11 ] to model sparse attention patterns in video DiTs. When generating a 321-frame video at 448 × 832 resolution (approximately 118K tokens), VMonarch speeds up self-attention computation by 17.5 × compared to full attention (FA2) [ 8 ] and by 2 × compared to highly sparse alternatives like Video Sparse Attention (VSA) at 90% sparsity. After minimal fine-tuning, VMonarch attains comparable video generation quality to FA2.
Figure 2: Overview of our Video Monarch Attention. (a) We represent the N×N full attention matrix via two alternatingly optimized smaller Monarch factors L and R with spatial-temporal structure factorization. (b) We introduce a Recomputation Strategy to address quality degradation caused by excessively large temperature term cR for the first frame. (c) We propose an Online-Entropy algorithm integrated with FlashAttention to accelerate the iterative computation of Monarch matrices further.
Figure 3: Illustration of the attention sink phenomenon in video models. The first frame tends to accumulate excessive attention from subsequent frames, leading to a loss of fine-grained details when using Monarch Attention. Our First-Frame Recomputation Strategy effectively restores these details by recalculating the attention for the first frame using full attention.
Model
Resolution
Kernel
Quality
Efficiency
AQ ↑
BC ↑
DD ↑
IQ ↑
SC ↑
Sparsity ↑
TFLOPs ↓
Time (s) ↓
Wan2.1 1.3B
61x448x832
FullAttn ⊘
65.44%
95.08%
69.44%
64.86%
91.77%
-
159.7
63.4
VMoBA ⊘
42.00%
90.64%
88.89%
53.78%
82.00%
90.0%
75.8
71.7
VSA ⊘
42.86%
94.96%
63.89%
55.04%
89.17%
90.0%
69.5
49.9
VMonarch ⊘
63.65%
94.93%
54.17%
64.46%
92.43%
87.5%
75.4
47.7
FullAttn
66.07%
95.75%
68.06%
64.57%
94.15%
-
159.7
63.4
Table 1: Quality and efficiency comparison of our and other methods, including Full Attention [ 8 ] , VSA [ 69 ] , and VMoBA [ 50 ] . We verify the generalizability of our method across different architectures, parameter scales, and varying resolutions under the same tuning setting. We report Vbench [ 21 ] metrics to evaluate quality, including Aesthetic Quality (AQ), Background Consistency (BC), Dynamic Degree (DD), Imaging Quality (IQ), and Subject Consistency (SC). We also report the efficiency metrics, including attention sparsity, TFLOPs, and inference time. ⊘ denotes Training-Free setting. ⋆ denotes extrapolation on longer sequences using the Wan2.1-1.3B [ 47 ] model tuned at 61×448×832 resolution. We highlight the results of VMonarch if it is the best or second best among efficient attention methods.
Model
Similarity
Vbench Score (%)
PSNR ↑
SSIM ↑
AQ
BC
DD
IQ
SC
Softmax
-
-
65.44
95.08
69.44
64.86
91.77
VM-T1-F1
11.18
0.35
54.36
89.75
16.67
54.93
93.17
VM-T2-F1 †
11.65
0.40
63.39
92.25
29.17
67.76
88.46
VM-T2-F1
12.59
0.43
63.65
94.93
54.17
64.46
92.43
VM-T3-F1
12.21
0.41
63.89
95.01
50.00
65.12
93.55
Table 2: Ablation studies on training-free setting. In ’VM-T n -F k ’, T represents the number of iterations, and F denotes the size of the Monarch factor b corresponding to the number of frames. The † indicates the model variant without first frame computation.
Figure 4: Qualitative samples of our models. We compare the generation quality of the tuned models between full attention [ 8 ] , Video Sparse Attention (VSA) [ 69 ] , and our VMonarch Attention.
Figure 5: The kernel speedup ratio over Flash Attention 2 (FA2) [ 8 ] under varying sequence lengths. Our VMonarch demonstrates superior efficiency, achieving over 2 × speedup at 28k tokens and over 5 × speedup at 62k tokens compared to FA2, surpassing other sparse attention methods at 90% sparsity.
Model
Vbench Score (%)
Val
AQ
BC
DD
IQ
SC
Loss
Softmax
65.72
94.85
75.00
64.46
92.59
1.09
VM-T1-F1
66.53
95.82
59.72
65.28
92.69
1.27
VM-T2-F1
64.84
95.28
68.06
63.83
93.29
1.25
VM-T3-F1
66.22
96.12
56.94
65.72
94.06
1.24
VM-T5-F1
65.63
96.16
45.83
67.91
94.94
1.57
Table 3: Ablation studies on fine-tuned Video Monarch Attention models under different configurations. In ’VM-T n -F k ’, T represents iterations numbers, and F denotes the frame number corresponding to the Monarch Matrix b . † omits the recompute strategy for the first frame. ‡ means leveraging attention sink by averaging.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A : Various computational budgets compared with FA2 at a sequence length of 215 . In practice, b is usually larger than 256 .
Figure 8: Speed comparison of FlashAttention2(FA2) [ 8 ] , VMonarch-Naive, and our optimized VMonarch-Fast (Triton) implementation. After applying kernel optimization, VMonarch-Fast achieves an approximately 8× speedup over the naive implementation and surpasses the performance of FA2.
Figure 9: Memory benchmark of different attention mechanisms. We compare the relative memory usage against full attention.
Figure 10: Speed benchmark of different attention mechanisms. We compare the relative speed against full attention.
Method
Similarity
PSNR ↑
SSIM ↑
FlashAttention2 (FullAttn) [ 8 ]
-
-
Video Sparse Attention (VSA) [ 69 ]
9.71
0.32
Video MoBA (VMoBA) [ 50 ]
10.61
0.30
Video Monarch (VMonarch)
12.59
0.43
Appendix
Table 4: Similarity comparison of our and other baseline methods. All methods are evaluated under the training-free setting with a resolution of 61×448×832 .
Figure 11: Attention map comparison among Full Attention, Sparse Attention, and Monarch Attention.
Exp.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR1↑
SSIM1↑
LPIPS1↓
IQ ↑
SC ↑
OC ↑
Score ↑
1
M (1)
11.01
0.21
96.41%
10.21
0.12
90.44%
48.54%
87.00%
16.28%
2.19
2
M (1) + cR∗ + cL1
10.66
0.23
80.39%
11.21
0.30
51.54%
62.22%
87.53%
18.85%
3.11
3
M (1) + cR1 + cL1
10.66
0.23
80.39%
11.21
0.30
51.54%
62.22%
87.53%
18.85%
3.11
4
M (1) + cR1 + cL∗
11.01
0.18
99.78%
10.29
0.11
95.63%
41.32%
85.87%
15.69%
1.98
5
M (1) + cR∗ + cL∗
11.01
0.18
99.78%
10.29
0.11
95.63%
41.32%
85.87%
15.69%
1.98
6
M (1) + cR1 + cL0
11.01
0.21
96.41%
10.21
0.12
90.44%
48.54%
87.00%
16.28%
2.19
Appendix
Table 5: Ablation study on VMonarch Attention. We evaluate several variants by adjusting its iterative parameters, including the number of iterations and the manual control of intermediate terms. Metrics with a superscript ’1’ (e.g., PSNR1 ) are calculated on the first frame only. The score is calculated as Score=(PSNR+PSNR1)/200+(SSIM+SSIM1)/2+(1−LPIPS+1−LPIPS1)/2+IQ+SC+OC , where IQ, SC, and OC denote Imaging Quality, Subject Consistency, and Overall Consistency, respectively. In the Method column, M (t) denotes the number of iterations t . +Re represents our recomputation mechanism. For the control of cR , + cR∗ indicates fixing it to 1 for all frames, + cR1 for the first frame only, and + cR0 means no fixing. For the control of cL , + cL∗ indicates fixing it to the initial expected value log(M) for all frames, where M is the monarch factor size, + cL1 for the first frame only, and + cL0 means no fixing.
Figure 12: Comparison of different Monarch factorizations.
Figure 13: More visual results comparing Full Attention, VSA, and VMonarch on various prompts.
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.