Diffusion Transformers (DiTs) have become a common backbone for high-quality text-to-image and text-to-video generation, but repeated dense self-attention remains a major inference bottleneck as token sequences grow and the same operator is invoked across many denoising steps. Our study suggests that dense attention in pretrained DiTs contains reusable layer/head-specific high-mass structure that can be identified from a small offline calibration set and reused across future generations. This leads to Sink-Sparse Hybrid Attention, a train-free attention replacement that reuses static sink masks and supplements them with a lightweight pooled-KV branch, while leaving model weights and sampling schedules unchanged. On Wan2.1-T2V-1.3B, the method attains 1.54× end-to-end speedup while maintaining generation quality. On Z-Image-Turbo, it attains 1.57× end-to-end speedup with competitive GenEval2 performance. These results indicate that reusable importance-aware sparse support can provide an effective train-free acceleration path for both T2V and T2I DiTs.
Figures & tables
Figure 1. Profiling sampling latency in the evaluated T2V setting. We decompose the sampling time Tsample (excluding model initialization and video writing) into self-attention time Tattn and the remaining components Tother across different denoising schedules. Self-attention consistently occupies a large fraction of the total runtime. Bar chart profiling sampling latency across denoising schedules, separating self-attention time from the remaining runtime and showing that attention occupies a large fraction of inference cost.
Figure 2. Attention pattern similarity across different prompts. We visualize representative dense self-attention maps from Wan2.1-T2V-1.3B and Z-Image-Turbo , each under three different prompts. Despite changes in prompt content, the dominant stripe-like structures remain visually similar.
Figure 3. Coverage of dense attention mass captured by the static sink mask across denoising steps. We evaluate a representative shared-step static sink mask with mean retained block density 38.8% (i.e., 61.2% sparsity). For each denoising step, the sink-mask curve shows the mean coverage across layer–head pairs on held-out cases, and the shaded band shows the 25th–75th percentile range. A budget-matched random mask is shown as a reference. Coverage-versus-step plot for a representative shared-step static sink mask, showing the fraction of dense attention probability mass preserved throughout the default 50-step denoising schedule, together with a budget-matched random baseline and shaded 25th–75th percentile ranges across layer–head pairs.
Figure 4. Workflow of Sink-Sparse Hybrid Attention. Offline calibration collects dense self-attention observations from a small calibration set, converts token-level attention into exact block-mass scores, averages them across observations, and applies a row-wise coverage threshold τ to obtain one static sink mask for each layer and head. Online inference reuses these masks at every denoising step: a static sparse sink branch preserves recurrently important interactions, a pooled-KV compensation branch supplies coarse omitted context using full-resolution queries and pooled keys/values, and the two branches are fused into the final attention output. Two-stage workflow of Sink-Sparse Hybrid Attention. The offline stage measures dense token attention, aggregates it into exact block-level mass, averages it across calibration observations, and constructs one static sink mask per layer and head through row-wise coverage selection. The online stage reuses the same static sink masks at every denoising step, combines a static sparse sink branch with a pooled-KV compensation branch, and fuses their outputs into the final attention result.
Method
Time(s) ↓
Speedup ↑
SubCons. ↑
MotSm. ↑
DynDeg. ↑
BgCons. ↑
Scene ↑
Aesth. ↑
ImgQual ↑
OverCons. ↑
VBench Avg (8 dims) ↑
Dense
164.80
1.00 ×
0.894
0.972
0.861
0.941
0.145
0.588
0.614
0.235
0.656
SVG1 ( Xi et al., 2025 )
115.49
1.43 ×
0.941
0.970
0.819
0.950
0.126
0.565
0.652
0.224
0.656
SpargeAttn ( Zhang et al., 2025d )
139.68
1.18 ×
0.930
0.954
0.833
0.961
0.154
0.608
0.676
0.208
0.665
Ours-Hybrid ( τ=0.70 )
94.57
1.74 ×
0.864
0.967
0.858
0.927
0.137
0.568
0.619
0.233
0.647
Ours-Hybrid ( τ=0.85 )
106.73
1.54 ×
0.871
0.976
0.931
0.933
0.148
0.566
0.615
0.233
0.659
Table 1. Main T2V comparison on Wan2.1-T2V-1.3B under a unified evaluation setup. Dense is the end-to-end reference under the same operator stack. Speedup is reported relative to this dense reference ( 164.80s ). We report two representative operating points of Ours-Hybrid : a speed-oriented setting at τ=0.70 and a quality-oriented setting at τ=0.85 . VBench Avg (8 dims) is computed from the underlying full-precision per-dimension scores before rounding, so it may not exactly equal the mean of the displayed three-decimal values.
Method
Soft-TIFA AM ↑
Soft-TIFA GM ↑
Time(s) ↓
Speedup ↑
Dense
67.80
16.38
23.39
1.00 ×
SpargeAttn ( Zhang et al., 2025d )
66.88
15.07
15.92
1.47 ×
Ours-Hybrid ( τ=0.85 )
66.48
15.21
14.88
1.57 ×
Table 2. Main T2I comparison on Z-Image-Turbo. Evaluated at 2048×2048 resolution with 8 denoising steps. We report Soft-TIFA AM , Soft-TIFA GM , end-to-end Time(s) , and Speedup relative to the dense baseline.
Figure 5. Representative qualitative comparison on Wan2.1-T2V-1.3B and Z-Image-Turbo. For Wan2.1-T2V-1.3B, we show three synchronized frames (frames 16, 40, and 64 out of 81) for the prompt “Campfire at night in a snowy forest with starry sky in the background.” using the same noise seed across methods. For Z-Image-Turbo, we show image results for the prompt “Winter station at golden hour.” in the order Dense, Ours-Hybrid ( τ=0.85 ), and SpargeAttn. Single-column qualitative comparison on both backbones. The upper part shows Wan2.1-T2V-1.3B outputs for the prompt 'Campfire at night in a snowy forest with starry sky in the background.' with four rows: Dense, Ours-Hybrid with tau equals 0.85, SpargeAttn, and SVG1; each row contains three synchronized frames sampled at frames 16, 40, and 64 out of 81. The lower part shows Z-Image-Turbo outputs for the prompt 'Winter station at golden hour.' in the order Dense, Ours-Hybrid with tau equals 0.85, and SpargeAttn.
Figure 6. Efficiency profile and quality–efficiency frontier in the T2V setting. Left: End-to-end runtime versus main-branch effective sparsity for Ours-Sink and Ours-Hybrid , with Dense , SVG1 , and SpargeAttn shown as reference points. Right: Quality–efficiency frontier measured by VBench Avg (8 dims) versus end-to-end runtime. Lower coverage thresholds yield stronger acceleration, while higher thresholds improve quality; τ=0.70 and τ=0.85 are highlighted as representative speed-oriented and quality-oriented operating points, respectively. A two-panel figure summarizing the efficiency and trade-off behavior of the proposed method in the T2V setting. The left panel plots end-to-end runtime against main-branch effective sparsity for Ours-Sink and Ours-Hybrid, together with Dense, SVG1, and SpargeAttn as reference points. The right panel plots VBench Avg against end-to-end runtime and highlights tau equals 0.70 and tau equals 0.85 as representative operating points.
Calibration Budget
Time(s) ↓
Offline Calib. Cost ↓
VBench Avg (8 dims) ↑
2×2
92.84
11.0 min
0.645
4×2
94.57
22.0 min
0.647
4×4
93.71
43.9 min
0.650
8×4
94.18
87.9 min
0.649
Table 3. Effect of calibration set size for Ours-Hybrid at τ=0.70 on Wan2.1-T2V-1.3B. Calibration budget is specified by the number of prompts and seeds used for offline mask acquisition, where each prompt–seed pair contributes one full denoising trajectory. Offline calibration cost is a rough one-time estimate computed from the shared-stack dense latency ( 164.80s per trajectory), reported in minutes. Larger calibration budgets improve quality modestly, and the gains largely saturate by 4 × 4 .
τ
Sink Time ↓
Sink Avg ↑
Hybrid Time ↓
Hybrid Avg ↑
Δ Avg ↑
0.60
81.82
0.553
88.88
0.624
+0.071
0.65
84.60
0.563
91.69
0.626
+0.063
0.70
87.22
0.581
94.57
0.647
+0.066
0.75
89.98
0.607
97.12
0.649
+0.042
0.80
94.04
0.623
101.37
0.652
+0.029
0.85
99.64
0.653
106.73
0.659
+0.006
Table 4. Effect of the pooled-KV compensation branch across coverage thresholds on Wan2.1-T2V-1.3B. Ours-Sink uses only the static sink mask, while Ours-Hybrid additionally enables the pooled-KV compensation branch with adaptive fusion. Δ Avg denotes the gain of Ours-Hybrid over Ours-Sink .
While Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, their reliance on 3D full attention creates a quadratic computational bottleneck. Existing sparse methods face a dilemma: dynamic pruning suffers from prohibitive runtime overhead and memory fragmentation, while static heuristics fail to capture fine-grained dependencies. In this work, we propose ScalingAttention, a training-free framework grounded in a key inductive bias: while individual activations are input-dependent, the high-mass attention regions for each head rapidly converge to a stable, prompt-agnostic Intrinsic Sparse Topology. This topology is weight-encoded, scale-invariant, and efficient to extract. ScalingAttention decouples topology discovery from sparsity control via: (1) WEST (Weight-Encoded Sparse Topology), which extracts a robust block-sparse prior mask offline to eliminate runtime search; (2) FAST (Fidelity-Aware Sensitivity Tuning), which adaptively tunes head-wise sparsity based on diffusion fidelity requirements. To ensure practical acceleration, we co-design a hardware-aligned bit-wise block-sparse kernel. Experiments on Wan2.1 show up to 1.90X end-to-end speedup with superior fidelity, establishing a new Pareto frontier over state-of-the-art baselines.
Ruiliang Zhou, Xuecheng Wu, Kang He +6
KlingAI Research · Beijing Institute of Technology · NVIDIA +1
Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention. Sparse attention mitigates this by skipping unnecessary computation in the attention matrix, but existing methods apply a fixed sparsity ratio throughout denoising and struggle to accurately determine which computations to skip. To address these limitations, we analyze attention dynamics through attention entropy and mean attention distance, and find that attention becomes increasingly concentrated across denoising steps. We further observe that blocks selected early continue to capture a large fraction of the total attention weight at later steps. These findings suggest two acceleration opportunities: the sparsity ratio should grow as denoising progresses, and earlier search results can be reused to reduce the search space. Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead. GSRS adopts Top-p selection for adaptive sparsity control, supported by a precise evaluation scheme that computes exact attention weights within the FlashAttention computation at negligible overhead. Experiments on HunyuanVideo and Wan 2.1 validate that GSRS achieves 1.55×--2.21× speedup while maintaining generation quality comparable to full attention.
Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1× end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.