Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers

Bingxin Xing; Yu Zhang; Jiatong Ding; Lei Lu; Ping Wang

cs.CVOct 16, 2026

Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers

Authors: Bingxin Xing, Yu Zhang, Jiatong Ding, Lei Lu, Ping Wang

Organizations: Xi’an Jiaotong University · Xiamen University · Shanghai Jiao Tong University

Abstract

Diffusion Transformers (DiTs) have become a common backbone for high-quality text-to-image and text-to-video generation, but repeated dense self-attention remains a major inference bottleneck as token sequences grow and the same operator is invoked across many denoising steps. Our study suggests that dense attention in pretrained DiTs contains reusable layer/head-specific high-mass structure that can be identified from a small offline calibration set and reused across future generations. This leads to Sink-Sparse Hybrid Attention, a train-free attention replacement that reuses static sink masks and supplements them with a lightweight pooled-KV branch, while leaving model weights and sampling schedules unchanged. On Wan2.1-T2V-1.3B, the method attains 1.54 $\times$ end-to-end speedup while maintaining generation quality. On Z-Image-Turbo, it attains 1.57 $\times$ end-to-end speedup with competitive GenEval2 performance. These results indicate that reusable importance-aware sparse support can provide an effective train-free acceleration path for both T2V and T2I DiTs.

Figures & tables

Figure 1. Profiling sampling latency in the evaluated T2V setting. We decompose the sampling time $T_{\mathrm{sample}}$ (excluding model initialization and video writing) into self-attention time $T_{\mathrm{attn}}$ and the remaining components $T_{\mathrm{other}}$ across different denoising schedules. Self-attention consistently occupies a large fraction of the total runtime. Bar chart profiling sampling latency across denoising schedules, separating self-attention time from the remaining runtime and showing that attention occupies a large fraction of inference cost.

Panel of Figure 2 — Figure 2. Attention pattern similarity across different prompts. We visualize representative dense self-attention maps from Wan2.1-T2V-1.3B and Z-Image-Turbo , each under three different prompts. Despite changes in prompt content, the dominant stripe-like structures remain visually similar.

Figure 3. Coverage of dense attention mass captured by the static sink mask across denoising steps. We evaluate a representative shared-step static sink mask with mean retained block density

38.8\%

(i.e.,

61.2\%

sparsity). For each denoising step, the sink-mask curve shows the mean coverage across layer–head pairs on held-out cases, and the shaded band shows the 25th–75th percentile range. A budget-matched random mask is shown as a reference. Coverage-versus-step plot for a representative shared-step static sink mask, showing the fraction of dense attention probability mass preserved throughout the default 50-step denoising schedule, together with a budget-matched random baseline and shaded 25th–75th percentile ranges across layer–head pairs.

Figure 4. Workflow of Sink-Sparse Hybrid Attention. Offline calibration collects dense self-attention observations from a small calibration set, converts token-level attention into exact block-mass scores, averages them across observations, and applies a row-wise coverage threshold $\tau$ to obtain one static sink mask for each layer and head. Online inference reuses these masks at every denoising step: a static sparse sink branch preserves recurrently important interactions, a pooled-KV compensation branch supplies coarse omitted context using full-resolution queries and pooled keys/values, and the two branches are fused into the final attention output. Two-stage workflow of Sink-Sparse Hybrid Attention. The offline stage measures dense token attention, aggregates it into exact block-level mass, averages it across calibration observations, and constructs one static sink mask per layer and head through row-wise coverage selection. The online stage reuses the same static sink masks at every denoising step, combines a static sparse sink branch with a pooled-KV compensation branch, and fuses their outputs into the final attention result.

Method	Time(s) $\downarrow$	Speedup $\uparrow$	SubCons. $\uparrow$	MotSm. $\uparrow$	DynDeg. $\uparrow$	BgCons. $\uparrow$	Scene $\uparrow$	Aesth. $\uparrow$	ImgQual $\uparrow$	OverCons. $\uparrow$	VBench Avg (8 dims) $\uparrow$
Dense	164.80	1.00 $\times$	0.894	0.972	0.861	0.941	0.145	0.588	0.614	0.235	0.656
SVG1 ( Xi et al., 2025 )	115.49	1.43 $\times$	0.941	0.970	0.819	0.950	0.126	0.565	0.652	0.224	0.656
SpargeAttn ( Zhang et al., 2025d )	139.68	1.18 $\times$	0.930	0.954	0.833	0.961	0.154	0.608	0.676	0.208	0.665
Ours-Hybrid ( $\tau=0.70$ )	94.57	1.74 $\times$	0.864	0.967	0.858	0.927	0.137	0.568	0.619	0.233	0.647
Ours-Hybrid ( $\tau=0.85$ )	106.73	1.54 $\times$	0.871	0.976	0.931	0.933	0.148	0.566	0.615	0.233	0.659

Table 1. Main T2V comparison on Wan2.1-T2V-1.3B under a unified evaluation setup. Dense is the end-to-end reference under the same operator stack. Speedup is reported relative to this dense reference ( 164.80s ). We report two representative operating points of Ours-Hybrid : a speed-oriented setting at

\tau=0.70

and a quality-oriented setting at

\tau=0.85

. VBench Avg (8 dims) is computed from the underlying full-precision per-dimension scores before rounding, so it may not exactly equal the mean of the displayed three-decimal values.

Method	Soft-TIFA AM $\uparrow$	Soft-TIFA GM $\uparrow$	Time(s) $\downarrow$	Speedup $\uparrow$
Dense	67.80	16.38	23.39	1.00 $\times$
SpargeAttn ( Zhang et al., 2025d )	66.88	15.07	15.92	1.47 $\times$
Ours-Hybrid ( $\tau=0.85$ )	66.48	15.21	14.88	1.57 $\times$

Table 2. Main T2I comparison on Z-Image-Turbo. Evaluated at

2048\times 2048

resolution with 8 denoising steps. We report Soft-TIFA AM , Soft-TIFA GM , end-to-end Time(s) , and Speedup relative to the dense baseline.

Panel of Figure 5 — Figure 5. Representative qualitative comparison on Wan2.1-T2V-1.3B and Z-Image-Turbo. For Wan2.1-T2V-1.3B, we show three synchronized frames (frames 16, 40, and 64 out of 81) for the prompt “Campfire at night in a snowy forest with starry sky in the background.” using the same noise seed across methods. For Z-Image-Turbo, we show image results for the prompt “Winter station at golden hour.” in the order Dense, Ours-Hybrid ( $\tau=0.85$ ), and SpargeAttn. Single-column qualitative comparison on both backbones. The upper part shows Wan2.1-T2V-1.3B outputs for the prompt 'Campfire at night in a snowy forest with starry sky in the background.' with four rows: Dense, Ours-Hybrid with tau equals 0.85, SpargeAttn, and SVG1; each row contains three synchronized frames sampled at frames 16, 40, and 64 out of 81. The lower part shows Z-Image-Turbo outputs for the prompt 'Winter station at golden hour.' in the order Dense, Ours-Hybrid with tau equals 0.85, and SpargeAttn.

Figure 6. Efficiency profile and quality–efficiency frontier in the T2V setting. Left: End-to-end runtime versus main-branch effective sparsity for Ours-Sink and Ours-Hybrid , with Dense , SVG1 , and SpargeAttn shown as reference points. Right: Quality–efficiency frontier measured by VBench Avg (8 dims) versus end-to-end runtime. Lower coverage thresholds yield stronger acceleration, while higher thresholds improve quality;

\tau=0.70

and

\tau=0.85

are highlighted as representative speed-oriented and quality-oriented operating points, respectively. A two-panel figure summarizing the efficiency and trade-off behavior of the proposed method in the T2V setting. The left panel plots end-to-end runtime against main-branch effective sparsity for Ours-Sink and Ours-Hybrid, together with Dense, SVG1, and SpargeAttn as reference points. The right panel plots VBench Avg against end-to-end runtime and highlights tau equals 0.70 and tau equals 0.85 as representative operating points.

Calibration Budget	Time(s) $\downarrow$	Offline Calib. Cost $\downarrow$	VBench Avg (8 dims) $\uparrow$
$2\times 2$	92.84	11.0 min	0.645
$4\times 2$	94.57	22.0 min	0.647
$4\times 4$	93.71	43.9 min	0.650
$8\times 4$	94.18	87.9 min	0.649

Table 3. Effect of calibration set size for Ours-Hybrid at

\tau=0.70

on Wan2.1-T2V-1.3B. Calibration budget is specified by the number of prompts and seeds used for offline mask acquisition, where each prompt–seed pair contributes one full denoising trajectory. Offline calibration cost is a rough one-time estimate computed from the shared-stack dense latency ( 164.80s per trajectory), reported in minutes. Larger calibration budgets improve quality modestly, and the gains largely saturate by 4

\times

4 .

$\tau$	Sink Time $\downarrow$	Sink Avg $\uparrow$	Hybrid Time $\downarrow$	Hybrid Avg $\uparrow$	$\Delta$ Avg $\uparrow$
0.60	81.82	0.553	88.88	0.624	+0.071
0.65	84.60	0.563	91.69	0.626	+0.063
0.70	87.22	0.581	94.57	0.647	+0.066
0.75	89.98	0.607	97.12	0.649	+0.042
0.80	94.04	0.623	101.37	0.652	+0.029
0.85	99.64	0.653	106.73	0.659	+0.006

Table 4. Effect of the pooled-KV compensation branch across coverage thresholds on Wan2.1-T2V-1.3B. Ours-Sink uses only the static sink mask, while Ours-Hybrid additionally enables the pooled-KV compensation branch with adaptive fusion.

\Delta

Avg denotes the gain of Ours-Hybrid over Ours-Sink .

Diffusion Model Inference Acceleration

Diffusion model inference acceleration reduces the time and compute needed to generate or reconstruct outputs while preserving quality.

See all 506 papers

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
Jun 22, 2026Ruiliang Zhou, Xuecheng Wu, Kang He +6Video Diffusion Models Diffusion Transformer
KlingAI Research · Beijing Institute of Technology · NVIDIA +1
Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search
Oct 15, 2026Yun Dai, Yuyun Chen, Jiarui Wen +3Sparse Attention Diffusion Model Inference Acceleration
South China University of Technology
DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.

Jun 22, 2026J/K move · Enter open · S save

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers

Ruiliang Zhou, Xuecheng Wu, Kang He, Guangyun Han+5

KlingAI Research · Beijing Institute of Technology · NVIDIA+1

Video Diffusion Models Diffusion Transformer Sparse Attention Diffusion Model Inference Acceleration Block-Sparse Attention

Open paper AskPDF arXiv

While Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, their reliance on 3D full attention creates a quadratic computational bottleneck. Existing sparse methods face a dilemma: dynamic pruning suffers from prohibitive runtime overhead and memory fragmentation, while static heuristics fail to capture fine-grained dependencies. In this work, we propose ScalingAttention, a training-free framework grounded in a key inductive bias: while individual activations are input-dependent, the high-mass attention regions for each head rapidly converge to a stable, prompt-agnostic Intrinsic Sparse Topology. This topology is weight-encoded, scale-invariant, and efficient to extract. ScalingAttention decouples topology discovery from sparsity control via: (1) WEST (Weight-Encoded Sparse Topology), which extracts a robust block-sparse prior mask offline to eliminate runtime search; (2) FAST (Fidelity-Aware Sensitivity Tuning), which adaptively tunes head-wise sparsity based on diffusion fidelity requirements. To ensure practical acceleration, we co-design a hardware-aligned bit-wise block-sparse kernel. Experiments on Wan2.1 show up to 1.90X end-to-end speedup with superior fidelity, establishing a new Pareto frontier over state-of-the-art baselines.

Abstract

Diffusion Model Inference Acceleration

Explore similar work

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers

Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation