Diffusion Transformers have become the dominant architecture for video generation. Their substantial computational cost motivates scaling inference across multi-GPU servers, yet efficient scaling remains challenging on commodity GPUs connected via PCIe, whose bandwidth is limited. Although sparse attention substantially reduces computation, its implications for communication remain underexplored. This paper argues that attention sparsity should be treated as a communication primitive. We present SparSP, an efficient sparse sequence parallel communication system that co-designs token distribution, communication routing, and asynchronous execution for sparse video diffusion models. First, Dependency-Aware Placement distributes sequence blocks according to diffusion models' sparse attention patterns. Second, Demand-Directed KV Routing transfers KV blocks directly to requesting GPUs without intermediate relays. Third, a Decoupled Transfer Runtime separates communication from GPU computation to reduce resource contention and maximize effective bandwidth. Our evaluation shows that SparSP improves attention performance by 1.38- 1.5×, achieves an average 1.17× (up to 1.69×) end-to-end speedup across three representative servers and three video diffusion models, and reduces communication volume by 12.54-23.05%. Moreover, we achieve an average 1.53-1.76× bandwidth improvement over NCCL.
Figures & tables
Figure 1. System overview. Dependency-aware placement and direct routing reduce KV movement; the Decoupled Runtime and fine-grained chunking coordinate transfer progress and attention execution.
Figure 2. Workflow of two sequence-parallel techniques.
Figure 3. Compute sparsity outpaces communication sparsity. (a) Computation and communication savings enabled by sparsity. (b) Time breakdown of dense and sparse attention.
Figure 4. Remote K is inefficient with naive partition.
Figure 5. Ring forwarding amplifies sparse communication across GPU numbers.
Figure 6
Figure 8. Dependency-aware placement reduces rank-level union amplification. (a) Contiguous sequence partitioning transfers more KV blocks because co-located queries have dissimilar dependencies. (b) Grouping queries with overlapping dependencies reduces the transfer.
Figure 9. (a) Conventional Ring Attention carries unnecessary tokens due to the demand of the next hop. (b) Demand-Directed KV Routing only transfers necessary tokens.
Figure 10. Illustration of Decoupled Transfer Runtime.
Figure 11. (a) Conventional Ring Attention with uniform chunk sizes. (b) Adaptive chunk size tuned by phases to achieve better overlap.
Figure 12. End-to-end execution latency. 2 2 2 Note: SGLang does not support LTX-Video.
Figure 13. Attention execution latency in one step. 2
Figure 14. Optimization breakdown.
Figure 15. Benefits from Demand-Directed KV Routing.
Figure 16. Bandwidth of NCCL and SparSP .
Figure 17. End-to-end latency over different GPU numbers. 2
Figure 18. Attention latency over different GPU numbers. 2
Figure 19. Communication reduction over GPU numbers
Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-p routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem. We present \method{}, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. \method{} uses Top-p routing, a Top-k safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, \method{} reduces average load imbalance from 1.34 to 1.08 and delivers a 4.41× attention speedup over FlashAttention, while achieving a 2.02--2.11× DiT inference speedup with competitive video quality.
Hao Liu, Chenghuan Huang, Ye Huang +7
Sun Yat-sen University · WeChat HPC, Tencent Inc. · Peking University +1
While Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, their reliance on 3D full attention creates a quadratic computational bottleneck. Existing sparse methods face a dilemma: dynamic pruning suffers from prohibitive runtime overhead and memory fragmentation, while static heuristics fail to capture fine-grained dependencies. In this work, we propose ScalingAttention, a training-free framework grounded in a key inductive bias: while individual activations are input-dependent, the high-mass attention regions for each head rapidly converge to a stable, prompt-agnostic Intrinsic Sparse Topology. This topology is weight-encoded, scale-invariant, and efficient to extract. ScalingAttention decouples topology discovery from sparsity control via: (1) WEST (Weight-Encoded Sparse Topology), which extracts a robust block-sparse prior mask offline to eliminate runtime search; (2) FAST (Fidelity-Aware Sensitivity Tuning), which adaptively tunes head-wise sparsity based on diffusion fidelity requirements. To ensure practical acceleration, we co-design a hardware-aligned bit-wise block-sparse kernel. Experiments on Wan2.1 show up to 1.90X end-to-end speedup with superior fidelity, establishing a new Pareto frontier over state-of-the-art baselines.
Ruiliang Zhou, Xuecheng Wu, Kang He +6
KlingAI Research · Beijing Institute of Technology · NVIDIA +1
Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1× end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.