Diffusion Transformers (DiT) have established a new state-of-the-art in high-fidelity image synthesis; however, their massive computational complexity and memory requirements hinder local deployment on resource-constrained edge devices. In this paper, we introduce EdgeDiT, a family of hardware-efficient generative transformers specifically engineered for mobile Neural Processing Units (NPUs), such as the Qualcomm Hexagon and Apple Neural Engine (ANE). By leveraging a hardware-aware optimization framework, we systematically identify and prune structural redundancies within the DiT backbone that are particularly taxing for mobile data-flows. Our approach yields a series of lightweight models that achieve a 20-30% reduction in parameters, a 36-46% decrease in FLOPs, and a 1.65-fold reduction in on-device latency without sacrificing the scaling advantages or the expressive capacity of the original transformer architecture. Extensive benchmarking demonstrates that EdgeDiT offers a superior Pareto-optimal trade-off between Frechet Inception Distance (FID) and inference latency compared to both optimized mobile U-Nets and vanilla DiT variants. By enabling responsive, private, and offline generative AI directly on-device, EdgeDiT provides a scalable blueprint for transitioning large-scale foundation models from high-end GPUs to the palm of the user.
Figures & tables
Figure 1: Flowsheet depicting the proposed method
Figure 2: hardware-aware surrogates search space design
Figure 3: image quality when Block Removal Surrogates are replacing 2 DiT Blocks vs 3 DiT Blocks
Figure 4: image quality vs MLP ratio surrogates
Figure 5: image quality vs Hidden dimension surrogates
Figure 6: Results of visual image quality comparison
Figure 7: Ablation: image quality of EdgeDiT with and without FwKD
Class Conditioned ImageNet 256×256
Model
Params (M)
CFG
FID - 50K ↓
SFID ↓
IS ↑
Precision ↑
Recall ↑
DiT - XL/2 [ 9 ]
675
4
16.23
11.06
80.91
0.93
0.26
EdgeDiT 1
471
4
12.3
13.97
75.72
0.92
0.24
EdgeDiT 6
530
4
12.4
14.96
78
0.91
0.25
Table 1: Benchmarking class-conditional image generation on ImageNet 256×256
Model
Image Size: 1×4×256×256
Image Size: 1×4×512×512
Params (M)
GMACs
GFlops
Params (M)
GMACs
GFlops
DiT S/2
32.96
6.06
12.13
33.26
31.44
62.93
DiT B/2
130.51
23.01
46.04
131.10
106.38
212.88
DiT L/2
458.10
80.70
161.47
458.89
360.98
722.27
DiT XL /2
675.13
118.62
237.34
676.01
524.54
1050.00
EdgeDiT 1
470.65
71.94
143.96
470.76
311.29
560.01
Table 2: Complexity Analysis: Parameters, GMACs, and GFLOPS for varying image resolutions.
Model
Image Size: 1×4×256×256
Image Size: 1×4×512×512
iPhone Latency
Samsung Latency
iPhone Latency
Samsung Latency
DiT S/2
6.63
8.35
93.31
52
DiT B/2
19.55
24.45
286.58
115.95
DiT L/2
68.99
85.00
790.26
376.08
DiT XL /2
118.56
129.00
1098.92
553.13
EdgeDiT 1
70.86
86.13
804.81
379.22
Table 3: Inference Latency (ms) comparison on mobile devices across different resolutions.
Diffusion Transformers use attention-based networks to generate and transform images, video, audio, and other structured data through iterative denoising.
Recent advances in diffusion transformers (DiTs) have set new standards in image generation, yet remain impractical for on-device deployment due to their high computational and memory costs. In this work, we present an efficient DiT framework tailored for mobile and edge devices that achieves transformer-level generation quality under strict resource constraints. Our design combines three key components. First, we propose a compact DiT architecture with an adaptive global-local sparse attention mechanism that balances global context modeling and local detail preservation. Second, we propose an elastic training framework that jointly optimizes sub-DiTs of varying capacities within a unified supernetwork, allowing a single model to dynamically adjust for efficient inference across different hardware. Finally, we develop Knowledge-Guided Distribution Matching Distillation, a step-distillation pipeline that integrates the DMD objective with knowledge transfer from few-step teacher models, producing high-fidelity and low-latency generation (e.g., 4-step) suitable for real-time on-device use. Together, these contributions enable scalable, efficient, and high-quality diffusion models for deployment on diverse hardware.
The Diffusion Transformer (DiT) architecture is the state-of-the-art paradigm for high-fidelity image generation, underpinning models like Stable Diffusion-3 and FLUX.1. However, deploying these models on resource-constrained mobile devices entails prohibitive computational and memory overhead. While efficiency-driven approaches like Linear-DiT and static pruning alleviate bottlenecks, they often incur quality degradation. Unlike cloud environments, mobile constraints require a single-model paradigm that dynamically balances fidelity and latency. We introduce ElasticDiT, which achieves this dynamic trade-off by adjusting spatial compression ratios and DiT block depths. By integrating Shift Sparse Block Attention (SSBA) and a Tiny DWT-Distilled VAE (T-DVAE), ElasticDiT reduces inference latency and memory footprint while maintaining image quality. Experiments confirm that ElasticDiT effectively covers a wide range of fidelity-latency trade-offs within a single set of parameters. By jointly adjusting compression and depth, a single ElasticDiT model can be reconfigured on-the-fly to outperform task-specific baselines. Specifically, our flex lite variant achieves an HPS of 32.87, surpassing the Flux model, while maintaining competitive quality at 84.16 percent average sparsity through SSBA. Furthermore, the plug-and-play T-DVAE provides SD3-level reconstruction with only 1/8x the computational cost of standard VAEs, and Flow-GRPO boosts semantic alignment (GenEval: 66.93 to 73.62). These results demonstrate that ElasticDiT offers a versatile, hardware-adaptive solution that eliminates the need for multiple specialized models, providing a promising path for future high-resolution image generation on mobile devices.
The remarkable generation quality of modern diffusion models often comes at the cost of massive parameter counts, which necessitate server-side inference with significant computational costs and potential privacy risks. Consequently, there is growing momentum toward developing efficient on-device alternatives. While recent efforts have optimized text-to-image models for mobile hardware, they remain relatively bulky, typically ranging from 0.5B to 1B parameters. We present BlazeEdit, a highly efficient, generalist image-to-image diffusion model tailored for on-device deployment. By identifying that many practical image editing tasks do not require text-based guidance, we eliminate the text-conditioning components and develop a multi-task architecture that consolidates object removal, outpainting, tone correction, relighting, and sticker generation into a single, compact model of only 195M parameters. BlazeEdit achieves a substantial reduction in download size and memory overhead while maintaining competitive generation quality. It completes a full inference pass in just 290ms on a Pixel 10, delivering a seamless, privacy-preserving, and lightning-fast experience for generalist image editing on the edge.