Rapid advances in image and video generation have enabled high-fidelity synthesis at higher resolutions and longer durations, making the quadratic cost of dense self-attention a major bottleneck as visual token sequences grow. Sparse attention seeks to preserve generation quality at lower computational cost by retaining informative query-key interactions and omitting less consequential ones, with end-to-end gains depending on the cost of constructing sparse support and the efficiency with which the system maps the resulting workload to hardware. We survey sparse attention for visual generation through a 4+1+3 analytical framework that links algorithmic design to system implementation. The framework examines how static and dynamic methods use source information and the evidence derived from it to decide which interactions to retain and at what granularity, while accounting for the roles of reuse in reducing repeated computation and correction in compensating for omitted contributions. We relate these algorithmic choices to their implementation on hardware through the framework's three system dimensions of data organization, execution, and dataflow. We summarize common evaluation metrics and discuss open challenges and research directions, providing researchers and practitioners with a reference for the joint design of sparse attention algorithms and systems.
Figures & tables
Fig. 1: Development timeline of representative sparse-attention methods and systems, showing a shift from algorithmic sparsity design toward tighter algorithm–system co-design for more structured and efficient sparse execution.
Survey
Topic Focus
Method Coverage
Alg.–Sys. Analysis
Sparse Attention
Visual Generation
Algorithmic Methods
System Methods
Unified Framework
Efficient Transformers [ 18 ]
×
×
✓
×
×
Sparsity in Transformers [ 19 ]
×
×
✓
✓
×
Attention in Diffusion Models [ 21 ]
×
✓
✓
×
×
Efficient Attention Methods [ 20 ]
×
×
✓
✓
×
Efficient Video Diffusion Models [ 22 ]
×
✓
✓
×
×
TABLE I: Comparison with related surveys that discuss sparse attention. A check mark indicates a primary topic under Topic Focus, systematic coverage of the corresponding methods under Method Coverage, or a framework linking algorithmic and system designs.
Fig. 2: Five approaches to accelerating visual generation. Distillation trains a smaller student model, caching reuses intermediate results across timesteps, quantization lowers numerical precision, pruning removes selected model components, and sparse attention computes attention over selected query–key pairs.
Fig. 3: The 4+1+3 framework for sparse attention. The source, evidence construction, granularity, and support specification determine which query–key pairs are retained. Reuse avoids repeating work across denoising steps, and correction compensates for contributions from omitted pairs. Data organization, execution, and dataflow determine how the sparse attention workload runs on hardware. The tile example shows how the shape of the support affects computation. Both patterns retain 16 query–key pairs, but with 2×2 tiles, the scattered pattern requires computing 56 entries and the tile-aligned pattern requires only 16.
TABLE II: The 4+1+3 design space for sparse attention in visual generation. The four support construction dimensions describe information sources, how evidence is derived, the unit of selection, and the rules specifying which connections are retained. The auxiliary dimension covers reuse and correction. The three system dimensions describe how sparse data is organized, how workloads are executed, and how data moves through the system. The framework provides a basis for analyzing sparse attention designs. The table draws on representative methods from the literature.
Fig. 4: Structure of the framework-guided review with representative papers for each topic.
Fig. 5: Representative sparse-attention support patterns. Rows index queries and columns index keys; light cells are skipped query–key pairs. Blue cells mark the primary selected support. Dark gray cells mark mandatory or structural connections, such as global anchor rows and columns, self-blocks, and the second axis of a factorized pattern. Orange cells mark auxiliary connections, such as random additions, secondary routes, and long-range or multiscale links.
Fig. 6: Representative methods for text-to-image generation using their official configurations, with FLUX.1-dev as the shared backbone.
Fig. 7: Representative methods for text-to-video generation using their official configurations, with Wan2.1 T2V 14B as the shared backbone.
TABLE III: Representative methods described by the 4+1+3 framework: algorithmic dimensions. Underlined text denotes a framework component, followed by its corresponding framework choices. A dash indicates no separate entry.
TABLE IV: Representative system methods in the 4+1+3 framework. Underlined text denotes a component, followed by its corresponding framework choices. A dash indicates no separate entry.
Evaluation metrics
Metric
Task
Target
FID [ 151 ]
Image and frame-level video
Distance between real and generated feature distributions.
IS [ 152 ]
Image and class-conditioned video
Classifier confidence and semantic diversity across generated samples.
PSNR
Image, video, editing, world model, 3D and 4D
Pixel-level fidelity to a paired reference or dense output.
SSIM [ 153 ]
Image, video, editing, world model, 3D and 4D
Local luminance, contrast, and structural similarity.
LPIPS [ 154 ]
Image, video, editing, 3D and 4D
Learned perceptual distance between paired visual outputs.
TABLE V: Common Metrics and Benchmarks for Visual Generation
Metric
Target
Attention sparsity
Fraction of attention edges or blocks skipped.
FLOPs
Arithmetic cost of attention or the complete model.
Attention latency
Execution time of the sparse attention operator.
Attention speedup
Dense-to-sparse attention latency ratio under the same tensor shape.
End-to-end latency
Complete generation time including routing and non-attention modules.
End-to-end speedup
Dense-to-sparse complete-generation latency ratio under matched settings.
TABLE VI: Metrics for Sparse-Attention Acceleration
The quadratic computational complexity of self-attention mechanisms imposes a critical constraint on transformer-based video generation, particularly for synthesizing ultra-long and high-definition sequences. While most sparse attention techniques employ fixed or heuristic patterns, they often struggle to accommodate the heterogeneous and layer-specific redundancies inherent in video diffusion transformers (VDiTs). We find that different attention heads consistently concentrate most of their attention mass in different coarse query-key regions, a behavior we call functional specialization. Although individual attention values vary with the generated content, these regions remain similar across prompts and random seeds. Leveraging these insights, we introduce Compact Attention, a hardware-aware sparse attention framework for efficient long-video synthesis. We quantify this stability and use it to construct searchable, reusable, head-specific masks. Based on the observation, we propose a deformable dual-window sparse pattern mechanism that flexibly approximates local and axial attention regions within one tile-aligned mask family. To avoid manually choosing a mask for each head, we develop a recall-driven offline auto-search algorithm that identifies high-quality sparse masks while leveraging temporal stability for mask reuse. Extensive evaluations on Wan2.1 (14B) and Hunyuan-Video demonstrate that Compact Attention achieves 2.25x/3.03x attention speedups over FlashAttention-2 and 1.20x/1.74x over FlashAttention-3, respectively, while maintaining visual quality and temporal consistency comparable to full-attention baselines. Project Page: https://yo-ava.github.io/Compact-Attention.github.io/
Qirui Li, Guangcong Zheng, Qi Zhao +6
College of Computer Science & Technology, Zhejiang University · Huawei Technologies
Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention. Sparse attention mitigates this by skipping unnecessary computation in the attention matrix, but existing methods apply a fixed sparsity ratio throughout denoising and struggle to accurately determine which computations to skip. To address these limitations, we analyze attention dynamics through attention entropy and mean attention distance, and find that attention becomes increasingly concentrated across denoising steps. We further observe that blocks selected early continue to capture a large fraction of the total attention weight at later steps. These findings suggest two acceleration opportunities: the sparsity ratio should grow as denoising progresses, and earlier search results can be reused to reduce the search space. Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead. GSRS adopts Top-p selection for adaptive sparsity control, supported by a precise evaluation scheme that computes exact attention weights within the FlashAttention computation at negligible overhead. Experiments on HunyuanVideo and Wan 2.1 validate that GSRS achieves 1.55×--2.21× speedup while maintaining generation quality comparable to full attention.
Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1× end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.