Organizations: Peking University, School of Computer Science, Beijing, China · Peking University, School of Electronics Engineering and Computer Science, Beijing, China · Beijing Institute of Technology, School of Computer Science & Technology, Beijing, China · Peking University, Beijing, China
Diffusion Transformers (DiTs) enable on-device image generation and editing but incur high attention costs at high resolutions. Existing sparse attention reduces computation, yet dynamic sparsity introduces preprocessing and irregular data accesses that poorly fit mobile NPUs. We present DiT.XPU, a sparse DiT inference framework for heterogeneous mobile SoCs. DiT.XPU combines tile-aligned sparse attention with anchor-based error compensation by exploiting slowly varying dense-sparse output residuals across denoising steps. Heterogeneous XPU collaboration overlaps NPU dense attention with GPU sparse calibration, while mask-guided dataflow reduces data transfers and layout conversion by overlapping data movement with computation. Across five DiT models on two Snapdragon smartphones, DiT.XPU achieves up to 1.56x end-to-end denoising speedup over the Hexagon-NPU backend and 11.5x attention-kernel speedup over the fastest alternative, while preserving generation quality close to dense attention. Code is available at https://github.com/IF-LAB-PKU/dit.xpu.
Figures & tables
图 1. Attention dominates high-resolution DiT latency. It accounts for 32%, 60%, and 86% of the SD3.5 Medium DiT runtime at 512, 1024, and 2048 resolution, respectively, and 64% for Z-Image Turbo at 2048.
图 2. Attention patterns across DiT models. The first column shows the full Q–K attention map of a representative layer, while the remaining columns visualize the spatial attention distributions of selected query tokens.
图 3. (a) Dynamic sparse attention workflow. (b) Static sparse attention workflow. The image K/V blocks are requested from the dynamically selected blocks in (a), and from the fixed mask in (b).
图 4. Architecture of a mobile SoC. Left: the CPU, GPU, and Hexagon NPU with their compute and memory resources. Right: the tensor stored in tile-first order, and the storage order within a 32×32 tile.
图 5. Dynamic and static sparse attention kernel breakdown for SD3.5 Medium at 1024*1024 on 8Gen4 NPU.
图 6. Calibration coefficient ρ and residual MSE across denoising steps for SD3.5 Medium at 1024×1024 resolution.
GenEval (%) ↑
HPSv2.1 ↑
ImageReward ↑
Model
Dense
Static
Dense
Static
Dense
Static
SD3.5 Medium
68.37
26.83
29.18
20.46
0.8774
−0.7486
SD3.5 Large
71.31
23.66
29.91
19.35
0.8781
−1.3234
Z-Image
75.35
65.88
27.74
25.50
0.6936
0.2371
Z-Image-Turbo
75.49
63.14
29.83
28.59
0.6960
0.4741
FLUX.2 [klein]
82.66
70.60
30.09
28.39
0.9108
0.5522
表 1. Static block-sparse attention substantially degrades generation quality across five DiT models.
Measurement
Per-query loading
Cross-query reuse
dit.xpu
DRAM read volume (MB)
456.3
216.4
51.36
Mean DMA request size (KB)
15.33
5.32
7.42
Data-wait time (ms)
4.52
2.11
0.02
K/V repacking time (ms)
5.45
3.40
0
Kernel latency (ms)
24.96
21.35
13.30
表 2. K/V data movement and kernel latency for SD3.5 Medium static sparse attention on the 8Gen4 NPU.
图 7. Overview of dit.xpu .
图 8. An illustration of the heterogeneous XPU collaboration.(1) sequential execution with CPU fallback, (2) NPU–GPU co-execution with explicit data copies, and (3) NPU–GPU co-execution through shared memory mappings.
图 9. (a) Naive Dataflow with repeated K/V loading and layout conversion; (b) Mask-Guided Dataflow with K/V reuse, prefetching, and fused QKV packing.
图 10. End-to-end denoising latency of dit.xpu and baseline backends across DiT models and mobile SoCs.
图 11. Attention kernel latency of dit.xpu and the baseline backends across DiT models and mobile SoCs.
Model
Architecture
Params
Heads
Head Dim
Layers
Steps
SD3.5 Medium
MMDiT-X
2.5B
24
64
24
20
SD3.5 Large
MMDiT
8.1B
38
64
38
20
Z-Image
S3-DiT
6B
30
128
32
15
Z-Image Turbo
S3-DiT
6B
30
128
32
8
FLUX.2 [klein]
RF Transformer
4B
24
128
25
4
表 3. Architecture and sampling configuration of the evaluated DiT models.
Latency (s) ↓
Dense
PISA
Sparge
Static
Ours
Speedup
SD3.5 Medium
579
1446
876
325
371
1.56×
SD3.5 Large
1579
3183
2185
1385
1206
1.31×
Z-Image
919
3030
1144
536
631
1.46×
Z-Image-Turbo
483
1239
616
292
326
1.48×
FLUX.2-klein
186
433
295
147
154
1.21×
Mean Speedup
1.00×
0.40×
0.72×
1.51×
1.40×
表 4. End-to-end denoising latency (s) of dense attention and three sparse-attention strategies at 1024×1024 resolution on the Snapdragon 8 Gen 4.
Model
CPU Dense
GPU Dense
GPU Sparse
NPU Dense
GPU/NPU Anchor
NPU Sparse
SD3.5 Medium
18.32
2.35
0.48
0.45
0.76
0.05
SD3.5 Large
12.88
3.77
0.77
0.79
1.18
0.08
FLUX.2-klein
12.17
4.20
0.23
0.53
0.72
0.07
Z-Image
15.16
5.37
0.28
0.73
0.90
0.09
Z-Image-Turbo
14.98
5.32
0.27
0.71
0.85
0.08
表 5. Single-kernel energy consumption across DiT models and execution configurations.
图 12. Qualitative comparison of dit.xpu and the baselines on three prompts, one per row: SD3.5-Medium, SD3.5-Large and Z-Image.
GenEval ↑
Dense
PISA
Sparge
Static
Ours
Δ
SD3.5 Medium
68.37
64.55
68.19
26.83
68.19
-0.18
SD3.5 Large
71.31
70.06
70.54
23.66
71.88
+0.57
Z-Image
75.35
73.30
75.13
65.88
76.26
+0.91
Z-Image-Turbo
75.49
72.49
75.47
63.14
76.54
+1.05
FLUX.2-klein
82.66
84.87
82.40
70.60
83.01
+0.35
Avg. Δ
–
-1.58
-0.29
-24.61
+0.54
表 6. Generation quality comparison across attention strategies. FID and CLIP are averaged over Landscape, People, and Food. Δ denotes the absolute change relative to Dense. For ↑ metrics, positive values indicate improvement; for FID ↓ , negative values indicate improvement.
图 13. Component ablation. Each build adds one technique to the previous one.
p
GenEval ↑
HPSv2.1 ↑
ImageReward ↑
Latency (s) ↓
2
68.36
29.63
0.8813
412.32
3
68.31
29.60
0.8725
394.88
4
68.24
29.55
0.8604
386.40
5
68.19
29.53
0.8575
371.11
6
66.07
29.42
0.8219
367.09
表 7. Generation quality and latency of dit.xpu under different anchor step intervals p on SD3.5 Medium at 1024×1024 resolution and 20 steps.
图 14. Heterogeneous-strategy ablation for a single denoising step on the Snapdragon 8 Gen 4 device.
图 15. KV load ablation of the attention kernel on the Snapdragon 8 Gen 4 device.
1920×1080
1080×1920
896×1152
1152×896
Dense
1565.88
1556.36
493.32
495.24
dit.xpu
786.14
803.97
324.48
317.02
表 8. Latency of SD3.5 Medium at different resolutions.
Method
GenEval ↑
Latency (s) ↓
Speedup ↑
Dense
0.6837
579
1.00 ×
ours
0.6819
371.11
1.56 ×
w/ TeaCache
0.6451
361.57
1.60 ×
w/ DBCache
0.6399
333.89
1.73 ×
w/ TaylorSeer
0.6423
333.14
1.74 ×
表 9. GenEval scores, inference latency and speedup of orthogonal caching on the DiT.xPU backend for the SD3.5-medium model at 1024×1024 resolution.
Diffusion Transformers (DiT) have established a new state-of-the-art in high-fidelity image synthesis; however, their massive computational complexity and memory requirements hinder local deployment on resource-constrained edge devices. In this paper, we introduce EdgeDiT, a family of hardware-efficient generative transformers specifically engineered for mobile Neural Processing Units (NPUs), such as the Qualcomm Hexagon and Apple Neural Engine (ANE). By leveraging a hardware-aware optimization framework, we systematically identify and prune structural redundancies within the DiT backbone that are particularly taxing for mobile data-flows. Our approach yields a series of lightweight models that achieve a 20-30% reduction in parameters, a 36-46% decrease in FLOPs, and a 1.65-fold reduction in on-device latency without sacrificing the scaling advantages or the expressive capacity of the original transformer architecture. Extensive benchmarking demonstrates that EdgeDiT offers a superior Pareto-optimal trade-off between Frechet Inception Distance (FID) and inference latency compared to both optimized mobile U-Nets and vanilla DiT variants. By enabling responsive, private, and offline generative AI directly on-device, EdgeDiT provides a scalable blueprint for transitioning large-scale foundation models from high-end GPUs to the palm of the user.
Recent advances in diffusion transformers (DiTs) have set new standards in image generation, yet remain impractical for on-device deployment due to their high computational and memory costs. In this work, we present an efficient DiT framework tailored for mobile and edge devices that achieves transformer-level generation quality under strict resource constraints. Our design combines three key components. First, we propose a compact DiT architecture with an adaptive global-local sparse attention mechanism that balances global context modeling and local detail preservation. Second, we propose an elastic training framework that jointly optimizes sub-DiTs of varying capacities within a unified supernetwork, allowing a single model to dynamically adjust for efficient inference across different hardware. Finally, we develop Knowledge-Guided Distribution Matching Distillation, a step-distillation pipeline that integrates the DMD objective with knowledge transfer from few-step teacher models, producing high-fidelity and low-latency generation (e.g., 4-step) suitable for real-time on-device use. Together, these contributions enable scalable, efficient, and high-quality diffusion models for deployment on diverse hardware.
Diffusion Transformers (DiTs) have become a common backbone for high-quality text-to-image and text-to-video generation, but repeated dense self-attention remains a major inference bottleneck as token sequences grow and the same operator is invoked across many denoising steps. Our study suggests that dense attention in pretrained DiTs contains reusable layer/head-specific high-mass structure that can be identified from a small offline calibration set and reused across future generations. This leads to Sink-Sparse Hybrid Attention, a train-free attention replacement that reuses static sink masks and supplements them with a lightweight pooled-KV branch, while leaving model weights and sampling schedules unchanged. On Wan2.1-T2V-1.3B, the method attains 1.54× end-to-end speedup while maintaining generation quality. On Z-Image-Turbo, it attains 1.57× end-to-end speedup with competitive GenEval2 performance. These results indicate that reusable importance-aware sparse support can provide an effective train-free acceleration path for both T2V and T2I DiTs.
Bingxin Xing, Yu Zhang, Jiatong Ding +2
Xi’an Jiaotong University · Xiamen University · Shanghai Jiao Tong University