High-fidelity video generation remains challenging for diffusion models due to the difficulty of modeling complex spatio-temporal dynamics efficiently. Recent video diffusion methods typically represent a video as a sequence of spatio-temporal tokens which can be modeled using Diffusion Transformers (DiTs). However, this approach faces a trade-off between the strong but expensive Full 3D Attention and the efficient but temporally limited Local Factorized Attention. To resolve this trade-off, we propose Matrix Attention, a frame-level temporal attention mechanism that processes an entire frame as a matrix and generates query, key, and value matrices via matrix-native operations. By attending across frames rather than tokens, Matrix Attention effectively preserves global spatio-temporal structure and adapts to significant motion. We build FrameDiT-G, a DiT architecture based on MatrixAttention, and further introduce FrameDiT-H, which integrates Matrix Attention with Local Factorized Attention to capture both large and small motion. Extensive experiments show that FrameDiT-H achieves state-of-the-art results across multiple video generation benchmarks, offering improved temporal coherence and video quality while maintaining efficiency comparable to Local Factorized Attention.
Figures & tables
Property
Full 3D
Spatio-Temporal Factorized
Local
Global
Hybrid
Handling Large Motion
✓
✗
✓
✓
Computationally Efficient
✗
✓
✓
✓
Fusing Global and Local
✗
✗
✗
✓
Table 1: Comparison of our proposed Global and Hybrid (Global–Local) factorized attention mechanisms (bold) with existing attention designs for DiT. The symbols ✓ , ✗ indicate whether each method possesses a given property.
Figure 1: Overview of the proposed FrameDiT . Built on the Diffusion Transformer with interleaved Spatial and Temporal blocks. (a) Local: conventional local factorized attention; (b) Global (ours): replaces temporal attention with Matrix Attention for frame-level temporal attention; (c) Global–Local Hybrid (ours): combines local and global temporal attention for unified spatio-temporal modeling.
Figure 2: Text-to-video generation comparison between Latte and our FrameDiT-H . We show 4 of 16 generated frames.
Figure 3: Scaling with video length. We compare Local Factorized, Full 3D attention, and our FrameDiT variants as video length increases from 16 to 128 frames on the 128×128 Taichi dataset. From left to right: FVD, FLOPs, inference latency, and peak memory. While Full 3D achieves competitive quality, it exhibits steep growth in computational and memory costs. In contrast, our models maintain comparable or better FVD while scaling more efficiently, with latency and memory close to Local Factorized attention.
Model
UCF101
Sky
Taichi-HD
Face
MoCoGAN [ 49 ]
2886.9
206.6
-
124.7
DIGAN [ 58 ]
1630.2
83.1
128.1
62.5
StyleGAN-V [ 43 ]
1431.0
79.5
143.5
47.4
VideoGPT [ 54 ]
2880.6
222.7
-
185.9
PVDM [ 57 ]
343.6
55.4
540.2
355.9
LVDM [ 14 ]
372.9
95.2
99.0
-
Table 2: Comparison of our FrameDiT-G/H with baselines for generating 16-frame videos at 256×256 resolution . Baseline results are reported from their original papers. * indicates results obtained using the official AR-Diffusion checkpoints.
Figure 4: FVD comparison of different models as increasing model size . Each bubble shows a model variant, where y-axis reports FVD, and bubble diameter is proportional to GFLOPs.
Model
#Params
Attention Type
Total Score
Quality Score
Semantic Score
Subject Consistency
Background Consistency
Temporal Flickering
Motion Smoothness
Dynamic Degree
Temporal Style
Latte [ 31 ]
1B
Factorized
77.29
79.72
67.58
88.88
95.40
98.89
94.63
68.89
24.76
Lavie [ 52 ]
3B
Factorized
77.08
78.78
70.31
91.41
97.47
98.30
96.38
49.72
25.93
OpenSoraPlan 1.1 [ 27 ]
1B
Factorized
78.00
80.91
66.38
95.73
96.73
99.03
98.28
47.72
23.87
OpenSoraPlan 1.3 [ 27 ]
2.7B
Full 3D
77.23
80.14
65.62
97.79
97.24
99.20
99.05
30.28
22.47
LTX-Video [ 12 ]
1.9B
Full 3D
80.00
82.30
70.79
96.56
97.20
99.34
98.86
54.35
22.62
Wan 2.1 [ 51 ]
2B
Full 3D
84.26
85.30
80.09
96.34
97.29
99.49
97.44
85.56
25.31
Table 3: Performance comparison of various text-to-video generation models on the VBench benchmark.
U normalization
FVD ↓
FVMD ↓
FID ↓
No
70.31
990.00
14.44
Softmax
66.15
943.32
13.45
ℓ1-normalization
66.79
987.43
13.83
ℓ2-normalization
67.13
984.72
13.44
Table 4: Comparison of the normalization for row-weight matrix U . Result shows that softmax achieves the best overall performance across FVD, FVMD, and FID metrics.
Nqk
GFLOPs
FVD ↓
FVMD ↓
FID ↓
1
341.60
72.16
1042.76
14.47
2
342.03
71.71
1044.55
14.81
4
342.89
70.50
1035.34
14.74
8
344.62
69.41
1000.23
14.45
16
348.07
68.21
990.02
14.30
32
354.96
67.40
959.91
14.01
Table 5: Effect of the row-weight dimension Nqk . Smaller Nqk provides stronger compression and lower GFLOPs, while larger Nqk consistently improves FVD, FVMD.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
FVD ↓
FVMD ↓
FID ↓
16
32
64
128
16
32
64
128
16
32
64
128
FrameDiT-H -Concat
66.15
126.90
225.39
256.40
943.32
478.14
125.81
70.10
13.45
15.48
17.54
22.29
FrameDiT-H -Gated
67.55
130.88
233.27
265.25
964.13
528.42
153.65
89.23
13.14
15.29
17.50
21.42
Appendix
Table 6: Comparison between different Fusion layers for FrameDiT-H . FrameDiT-H -Concat is the model using Concat+MLP fusion, and FrameDiT-H -Gated is the model using sigmoid fusion described in B.3 .
Figure 5: Qualitative comparison on 128-frame Taichi-HD 128×128 . Local Factorized Attention exhibits severe temporal drift and collapsing human structure. In contrast, Full 3D model and FrameDiT-G , FrameDiT-H remain stable even at 128 frames, generating smooth and coherent motion. The slight blurring of small regions (hands, face) arises from the low-resolution encoding of the Stable Diffusion 2.0 autoencoder.
Figure 6: Qualitative comparison on UCF101 between prior video generative models and our approach.