Rapid advances in image and video generation have enabled high-fidelity synthesis at higher resolutions and longer durations, making the quadratic cost of dense self-attention a major bottleneck as visual token sequences grow. Sparse attention seeks to preserve generation quality at lower computational cost by retaining informative query-key interactions and omitting less consequential ones, with end-to-end gains depending on the cost of constructing sparse support and the efficiency with which the system maps the resulting workload to hardware. We survey sparse attention for visual generation through a 4+1+3 analytical framework that links algorithmic design to system implementation. The framework examines how static and dynamic methods use source information and the evidence derived from it to decide which interactions to retain and at what granularity, while accounting for the roles of reuse in reducing repeated computation and correction in compensating for omitted contributions. We relate these algorithmic choices to their implementation on hardware through the framework's three system dimensions of data organization, execution, and dataflow. We summarize common evaluation metrics and discuss open challenges and research directions, providing researchers and practitioners with a reference for the joint design of sparse attention algorithms and systems.
Figures & tables
Fig. 1: Development timeline of representative sparse-attention methods and systems, showing a shift from algorithmic sparsity design toward tighter algorithm–system co-design for more structured and efficient sparse execution.
Survey
Topic Focus
Method Coverage
Alg.–Sys. Analysis
Sparse Attention
Visual Generation
Algorithmic Methods
System Methods
Unified Framework
Efficient Transformers [ 18 ]
×
×
✓
×
×
Sparsity in Transformers [ 19 ]
×
×
✓
✓
×
Attention in Diffusion Models [ 21 ]
×
✓
✓
×
×
Efficient Attention Methods [ 20 ]
×
×
✓
✓
×
Efficient Video Diffusion Models [ 22 ]
×
✓
✓
×
×
TABLE I: Comparison with related surveys that discuss sparse attention. A check mark indicates a primary topic under Topic Focus, systematic coverage of the corresponding methods under Method Coverage, or a framework linking algorithmic and system designs.
Fig. 2: Five approaches to accelerating visual generation. Distillation trains a smaller student model, caching reuses intermediate results across timesteps, quantization lowers numerical precision, pruning removes selected model components, and sparse attention computes attention over selected query–key pairs.
Fig. 3: The 4+1+3 framework for sparse attention. The source, evidence construction, granularity, and support specification determine which query–key pairs are retained. Reuse avoids repeating work across denoising steps, and correction compensates for contributions from omitted pairs. Data organization, execution, and dataflow determine how the sparse attention workload runs on hardware. The tile example shows how the shape of the support affects computation. Both patterns retain 16 query–key pairs, but with 2×2 tiles, the scattered pattern requires computing 56 entries and the tile-aligned pattern requires only 16.
TABLE II: The 4+1+3 design space for sparse attention in visual generation. The four support construction dimensions describe information sources, how evidence is derived, the unit of selection, and the rules specifying which connections are retained. The auxiliary dimension covers reuse and correction. The three system dimensions describe how sparse data is organized, how workloads are executed, and how data moves through the system. The framework provides a basis for analyzing sparse attention designs. The table draws on representative methods from the literature.
Fig. 4: Structure of the framework-guided review with representative papers for each topic.
Fig. 5: Representative sparse-attention support patterns. Rows index queries and columns index keys; light cells are skipped query–key pairs. Blue cells mark the primary selected support. Dark gray cells mark mandatory or structural connections, such as global anchor rows and columns, self-blocks, and the second axis of a factorized pattern. Orange cells mark auxiliary connections, such as random additions, secondary routes, and long-range or multiscale links.
Fig. 6: Representative methods for text-to-image generation using their official configurations, with FLUX.1-dev as the shared backbone.
Fig. 7: Representative methods for text-to-video generation using their official configurations, with Wan2.1 T2V 14B as the shared backbone.
TABLE III: Representative methods described by the 4+1+3 framework: algorithmic dimensions. Underlined text denotes a framework component, followed by its corresponding framework choices. A dash indicates no separate entry.
TABLE IV: Representative system methods in the 4+1+3 framework. Underlined text denotes a component, followed by its corresponding framework choices. A dash indicates no separate entry.
Evaluation metrics
Metric
Task
Target
FID [ 151 ]
Image and frame-level video
Distance between real and generated feature distributions.
IS [ 152 ]
Image and class-conditioned video
Classifier confidence and semantic diversity across generated samples.
PSNR
Image, video, editing, world model, 3D and 4D
Pixel-level fidelity to a paired reference or dense output.
SSIM [ 153 ]
Image, video, editing, world model, 3D and 4D
Local luminance, contrast, and structural similarity.
LPIPS [ 154 ]
Image, video, editing, 3D and 4D
Learned perceptual distance between paired visual outputs.
TABLE V: Common Metrics and Benchmarks for Visual Generation
Metric
Target
Attention sparsity
Fraction of attention edges or blocks skipped.
FLOPs
Arithmetic cost of attention or the complete model.
Attention latency
Execution time of the sparse attention operator.
Attention speedup
Dense-to-sparse attention latency ratio under the same tensor shape.
End-to-end latency
Complete generation time including routing and non-attention modules.
End-to-end speedup
Dense-to-sparse complete-generation latency ratio under matched settings.
TABLE VI: Metrics for Sparse-Attention Acceleration