SODA: Sensitivity-Oriented Dynamic Acceleration for Diffusion Transformer

Tong Shao; Yusen Fu; Guoying Sun; Jingde Kong; Zhuotao Tian; Jingyong Su

cs.CVMar 7, 2026

SODA: Sensitivity-Oriented Dynamic Acceleration for Diffusion Transformer

Authors: Tong Shao, Yusen Fu, Guoying Sun, Jingde Kong, Zhuotao Tian, Jingyong Su

Organizations: Harbin Institute of Technology, Shenzhen

Abstract

Diffusion Transformers have become a dominant paradigm in visual generation, yet their low inference efficiency remains a key bottleneck hindering further advancement. Among common training-free techniques, caching offers high acceleration efficiency but often compromises fidelity, whereas pruning shows the opposite trade-off. Integrating caching with pruning achieves a balance between acceleration and generation quality. However, existing methods typically employ fixed and heuristic schemes to configure caching and pruning strategies. While they roughly follow the overall sensitivity trend of generation models to acceleration, they fail to capture fine-grained and complex variations, inevitably skipping highly sensitive computations and leading to quality degradation. Furthermore, such manually designed strategies exhibit poor generalization. To address these issues, we propose SODA, a Sensitivity-Oriented Dynamic Acceleration method that adaptively performs caching and pruning based on fine-grained sensitivity. SODA builds an offline sensitivity error modeling framework across timesteps, layers, and modules to capture the sensitivity to different acceleration operations. The cache intervals are optimized via dynamic programming with sensitivity error as the cost function, minimizing the impact of caching on model sensitivity. During pruning and cache reuse, SODA adaptively determines the pruning timing and rate to preserve computations of highly sensitive tokens, significantly enhancing generation fidelity. Extensive experiments on DiT-XL/2, PixArt- $α$ , and OpenSora demonstrate that SODA achieves state-of-the-art generation fidelity under controllable acceleration ratios. Our code is released publicly at: https://github.com/leaves162/SODA.

Figures & tables

Figure 1: Our SODA enables sensitivity awareness and adaptive acceleration strategy decision. (a). Examples of model-internal sensitivity to acceleration. The sensitivity of different timesteps, layers, and modules to acceleration is highly complex and dynamic. (b). Correlation between acceleration strategies and model sensitivity. Heuristic methods such as DuCa [ 69 ] fail to be consistent with the complex sensitivity. (c). Acceleration versus generation quality. Our SODA significantly outperforms the baselines by alleviating quality degradation under acceleration.

Figure 2: Model architecture of SODA. We propose SODA, a Sensitivity-Oriented Dynamic Acceleration method. (1). Offline Fine-grained Sensitivity Modeling (OFS): Defining error to measure the fine-grained sensitivity of timesteps, layers and modules before inference. (2). Dynamic Caching Scheduling Optimization (DCS): Employing dynamic programming to identify the optimal combination of cache intervals that yields the minimal sensitivity impact. (3). Unified Adaptive Strategy Formulation (UAS): Achieving adaptive scheduling for pruning timing and rate guided by sensitivity errors when pruning.

Figure 3: Comparison of cumulative sensitivity error and generation fidelity. Compared with ToCa and DuCa, our DCS module in SODA effectively reduces sensitivity error under the same acceleration efficiency. When combined with the UAS module, generation fidelity is further improved.

DiT-XL/2	Sampling	Latency(s)↓	FLOPs(T)↓	Speed↑	FID↓	sFID↓	IS↑	Precision↑	Recall↑
250 steps	DDPM	2.508	118.68	1.00×	2.23	4.57	275.65	0.82	0.58
SODA ( $N_{s}$ =125)	DDPM	1.778	76.70	1.55×	2.21	4.72	272.01	0.82	0.58
FORA [ 46 ]	DDPM	-	43.36	2.74×	2.80	6.13	/	0.80	0.59
ToCa [ 68 ]	DDPM	1.211	43.22	2.75×	2.58	5.74	256.26	0.80	0.59
DuCa [ 69 ]	DDPM	1.170	43.42	2.73×	2.59	5.68	256.36	0.80	0.59
SODA ( $N_{s}$ =72)	DDPM	1.070	43.42	2.73×	2.47	5.09	262.30	0.81	0.59

Table 1: Quantitative comparison of DiT-XL/2 on class-conditional image generation. Gray background indicates our SODA, and bold font highlights the best result. The arrow denotes whether lower or higher values indicate superior performance.

N_{s}

represents the caching times and is used to adjust the acceleration budget. Results on more baselines and generation models are provided in Appendix B.2.

Figure 4: Qualitative comparison of acceleration. (a) DiT-XL/2 (DDIM). (b) PixArt- $\alpha$ . (c) OpenSora. We highlight the areas with red dashed boxes to emphasize the comparison. Our SODA achieves higher generation fidelity under the same or higher acceleration efficiency compared with baselines. More visualizations are provided in Appendix C.

PixArt- $\alpha$	Latency (s)↓	FLOPs (T)↓	Spe.↑	MS-COCO2017 w/ difference(‰)
PixArt- $\alpha$	Latency (s)↓	FLOPs (T)↓	Spe.↑	FID-30k↓	CLIP↑
20 steps	0.857	11.18	1.00×	28.10	16.29
10 steps	0.491	5.54	2.00×	37.57	15.83
FORA(ArXiv24)	-	5.66	1.98×	29.67 55.9↓	16.40 6.7↑
ToCa(ICLR25)	0.492	6.05	1.85×	28.38 9.9↓	16.43 8.6↑
DuCa(ArXiv24)	0.479	5.99	1.87×	28.05 1.8↑	16.42 8.0↑

Table 2: Quantitative comparison of PixArt-

\boldsymbol{\alpha}

on text-to-image generation. Subscripts indicate the relative difference compared to the original results. Values in red denote the best performance.

OpenSora	Latency (s)↓	FLOPs (T)↓	Spe.↑	VBench(%↑) w/ degrad(‰)
30 steps	102.287	3283.20	1.00×	79.13
SODA ( $N_{s}$ =21)	86.684	2319.94	1.42×	79.13 0.00↓
PAB(ICLR25)	-	2558.25	1.28×	76.95 27.55↓
FORA(ArXiv24)	-	1751.32	1.87×	76.91 28.06↓
$\Delta$ -DiT(ArXiv24)	-	3166.47	1.04×	78.21 11.62↓
ToCa(ICLR25)	54.698	1394.03	2.36×	78.34 9.98↓

Table 3: Quantitative comparison of OpenSora on text-to-video generation. Subscripts indicate the relative degradation compared to the original results.

Modules	FID↓	sFID↓	IS↑	Precision↑	Recall↑
Vanilla	3.83	5.24	213.12	0.77	0.57
OFS w/ DCS	2.78	4.63	234.78	0.80	0.58
OFS w/ UAS	2.89	4.75	235.01	0.79	0.58
SODA (Ours)	2.75	4.56	235.65	0.80	0.58
L1	2.92	4.91	233.62	0.79	0.58
L2	2.88	4.88	233.91	0.79	0.58

Table 4: Ablation study for different modules under 2.5× acceleration on DiT-XL/2 with DDIM. “Vanilla” is the naive caching with fixed cache intervals. “L1”, “L2”, and “COS” denote the metrics used to define sensitivity error in OFS module.

Figure 5: VBench detailed metric comparison. We normalize metrics by setting the highest value to 100%, and visualize 9 selected metrics along with acceleration efficiency.

Figure 6: Analysis of sensitivity error distribution. (a) Mean–variance of offline sensitivity errors under different cache intervals. The variance remains stable and low across multiple random generations, indicating that sensitivity is primarily model‑related and marginally affected by content changes. (b) Consistency of sensitivity error distributions between offline modeling and actual inference under different cache intervals. It reveals that the sensitivity error distributions obtained from offline modeling are consistent with those during actual inference.

Phase	Cost	DiT-XL/2	PixArt- $\alpha$	OpenSora
	# of samples	100	100	10
Offline modeling	Time of generation	69.71s	90.13s	24.23m
	Time of OFS	91.23s	98.71s	31.03m
	Memory of generation	4.09GB	22.31GB	52.40GB
	Memory of OFS	4.65GB	22.65GB	52.40GB
Online use	Memory of $\mathcal{E}$	0.16MB	0.13MB	0.27MB

Table 5: Offline cost of different models. # denotes the number. The cost of OFS includes the cost of random content generation (the cost of generation). This modeling process needs to be performed only once per generation model and can be permanently reused, independent of variations in CFG, seed, or prompt. At runtime, the sensitivity errors

\mathcal{E}

are simply loaded into memory, introducing no time overhead and negligible memory cost.

Figure 7: The impact of parameters

\lambda

and

\beta

on DiT-XL/2 with DDIM.

\Delta

FID

\leq

0.02, and

\Delta

\leq

2.2.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Sensitivity error visualization of DiT-XL/2. We visualize sensitivity errors of each module (attention is abbreviated as ATT) at every timestep and layer during the denoising process, under different cache intervals and pruning rates.

Figure 9: Sensitivity error visualization of PixArt- $\alpha$ . We visualize the sensitivity error of each module (attention is abbreviated as ATT, cross-attention as CAT) at every timestep and layer during the denoising process, under different cache intervals and pruning rates.

Figure 10: Sensitivity error visualization of MLP in OpenSora. We visualize sensitivity errors of each module at every timestep and layer during the denoising process, under different cache intervals and pruning rates.

Figure 11: Sensitivity error visualization of different models. There exist distinct phases where the sensitivity error reduction is particularly pronounced in every model.

Model	Sampling	Latency(s)↓	FLOPs(G)↓	Speed↑	FID↓	sFID↓	IS↑	Precision↑	Recall↑
DiT	DDPM	2.508	118.68	1.00×	2.23	4.57	275.65	0.82	0.58
SODA ( $N_{s}$ =125)	DDPM	1.778	76.70	1.55×	2.21	4.72	272.01	0.82	0.58
SODA ( $N_{s}$ =72)	DDPM	1.070	43.42	2.73×	2.47	5.09	262.30	0.81	0.59
SODA ( $N_{s}$ =69)	DDPM	0.839	35.25	3.37×	2.88	6.11	252.03	0.80	0.59
SODA ( $N_{s}$ =63)	DDPM	0.704	29.66	4.00×	3.33	6.76	237.69	0.78	0.60
DiT	DDIM	0.533	23.74	1.00×	2.25	4.33	239.97	0.80	0.59

Appendix

Table 6: More results of DiT-XL/2 on class-conditional image generation. The arrow denotes whether lower or higher values indicate superior performance.

N_{s}

represents the number of cache intervals and is used to adjust the acceleration budget. All speeds are measured on RTX 4090 GPU.

Model	Latency(s)↓	FLOPs(G)↓	VBench(%)↑
Model	Latency(s)↓	FLOPs(G)↓	Total	Imaging quality	Subject consistency	Scene
OpenSora	102.287	3283.20	79.13	61.14	95.09	52.46
SODA ( $N_{s}$ =21)	86.684	2319.94 1.42×	79.13	60.63	95.26	53.35
$\Delta$ -DiT (ArXiv24)	-	3166.47	78.21	-	-	-
PAB (ICLR25)	-	2558.25	76.95	-	-	-
FORA (ArXiv24)	-	1751.32	76.91	-	-	-

Appendix

Table 7: More VBench results of OpenSora on text-to-video generation. All speeds are measured on 80G A100 GPU.

Qwen-Img	Lat.(s)↓ /Spe.↑	CLIP↑	IR↑	Wan2.1	Lat.(s)↓ /Spe.↑	Vbench (%)↑
Origin	73.431 1.00×	35.69	1.0242	Origin	187.46 1.00×	78.97
Base	41.135 1.79×	35.14	0.8677	Base	118.82 1.58×	74.24
Ours	40.749 1.80×	35.72	1.0352	Ours	119.46 1.57×	75.61

Appendix

Table 8: Results on more generation models.

DiT-XL/2	Lat.(s)↓	Spe.↑	IS↑	FID↓	sFID↓	P↑	R↑
DDPM/250	2.508	1.00×	275.65	2.23	4.57	0.82	0.58
$\Delta$ -DiT [ 6 ]	-	1.52×	271.03	2.31	-	-	-
Ours	1.778	1.41×	272.01	2.21	4.72	0.82	0.58
Ours	1.651	1.52×	271.77	2.23	4.76	0.82	0.59
DDIM/20	0.243	1.00×	223.23	3.51	4.92	0.79	0.57
GOC(FORA) [ 40 ]	0.167	1.46×	191.44	6.78	8.74	0.74	0.53

Appendix

Table 9: Comparison with more baselines.

Training caching	Lat.(s)↓ /Spe.↑	IS↑	FID↓	sFID↓	Distillation one-step	Spe.↑	Fidelity↑ ( $\frac{\text{Method}}{\text{Origin}}$ )(%)	Costs↓ GPU/time
DiT-XL/2	0.533 1.00×	239.97	2.25	4.33	DiT-XL/2	1×	100	-
L2C [ 35 ]	None 1.25×	233.26	2.62	4.50	ShortCut [ 13 ]	$\sim$ 50×	$\sim$ 40	TPUV3s/2days
HarmoniCa [ 20 ]	None 1.30×	238.74	2.36	4.24	$\pi$ -flow [ 3 ]	$\sim$ 50×	$\sim$ 78	Not mentioned
Ours	0.403 1.32×	241.71	2.37	4.58	Ours	$\sim$ 4×	$\sim$ 98	Free

Appendix

Table 10: Comparison with training-based caching and few-step distillation.

Model	VBench(%)↑
Model	Total	Imaging quality	Subject consistency	Scene
Vanilla	76.41	54.23	93.54	50.51
OFS w/ DCS	78.37	56.54	94.12	53.41
OFS w/ UAS	78.41	57.55	94.07	53.03
SODA (Ours)	78.49	58.14	94.60	53.33

Appendix

Table 11: Ablation study of OpenSora on text-to-video generation.

FID↓		$\beta$
		0.4	0.5	0.6	0.7	0.8
$\lambda$	0.4	27.46	27.44	27.42	27.38	27.33
	0.5	27.44	27.42	27.37	27.33	27.32
	0.6	27.36	27.35	27.34	27.33	27.32

Appendix

Table 12: The impact of parameters

\lambda

and

\beta

on PixArt-

\alpha

Figure 12: Visualization results of different acceleration methods on DiT-XL/2. Comparisons are highlighted with red dashed boxes.

Figure 13: Visualization results of different acceleration methods on PixArt- $\mathrm{\alpha}$ .

DiT-XL/2	FlashAtt	Lat.(s)↓	Spe.↑	IS↑	FID↓	sFID↓	P↑	R↑
Mean	✓	0.201	2.65×	233.22	2.80	4.61	0.80	0.58
Variance	✓	0.211	2.53×	233.20	2.82	2.67	0.80	0.58
Norm	✓	0.197	2.70×	233.18	2.83	4.66	0.80	0.58
Attn weights	✗	0.300	1.78×	233.25	2.81	4.63	0.80	0.58

Appendix

Table 13: Ablation study for pruning location.

Figure 14: Visualization results of different acceleration methods on OpenSora.

Diffusion Model Inference Acceleration

Diffusion model inference acceleration reduces the time and compute needed to generate or reconstruct outputs while preserving quality.

See all 506 papers

Apr 4, 2026

DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity

Diffusion models demonstrate outstanding performance in image generation, but their multi-step inference mechanism requires immense computational cost. Previous works accelerate inference by leveraging layer or token cache techniques to reduce computational cost. However, these methods fail to achieve superior acceleration performance in few-step diffusion transformer models due to inefficient feature caching strategies, manually designed sparsity allocation, and the practice of retaining complete forward computations in several steps in these token cache methods. To tackle these challenges, we propose a differentiable layer-wise sparsity optimization framework for diffusion transformer models, leveraging token caching to reduce token computation costs and enhance acceleration. Our method optimizes layer-wise sparsity allocation in an end-to-end manner through a learnable network combined with a dynamic programming solver. Additionally, our proposed two-stage training strategy eliminates the need for full-step processing in existing methods, further improving efficiency. We conducted extensive experiments on a range of diffusion-transformer models, including DiT-XL/2, PixArt-

α

, FLUX, and Wan2.1. Across these architectures, our method consistently improves efficiency without degrading sample quality. For example, on PixArt-

α

with 20 sampling steps, we reduce computational cost by

54\%

while achieving generation metrics that surpass those of the original model, substantially outperforming prior approaches. These results demonstrate that our method delivers large efficiency gains while often improving generation quality.

Haowei Zhu, Ji Liu, Ziqiong Liu +4

Advanced Micro Devices, Inc. · Tsinghua University · BNRist

Diffusion Model Inference Acceleration Diffusion Models Diffusion Transformer Diffusion Model Caching

Ask PDF

Feb 28, 2026

Jano: Adaptive Diffusion Generation with Early-stage Convergence Awareness

Diffusion models have achieved remarkable success in generative AI, yet their computational efficiency remains a significant challenge, particularly for Diffusion Transformers (DiTs) requiring intensive full-attention computation. While existing acceleration approaches focus on content-agnostic uniform optimization strategies, we observe that different regions in generated content exhibit heterogeneous convergence patterns during the denoising process. We present Jano, a training-free framework that leverages this insight for efficient region-aware generation. Jano introduces an early-stage complexity recognition algorithm that accurately identifies regional convergence requirements within initial denoising steps, coupled with an adaptive token scheduling runtime that optimizes computational resource allocation. Through comprehensive evaluation on state-of-the-art models, Jano achieves substantial acceleration (average 2.0 times speedup, up to 2.4 times) while preserving generation quality. Our work challenges conventional uniform processing assumptions and provides a practical solution for accelerating large-scale content generation. The source code of our implementation is available at https://github.com/chen-yy20/Jano.

Yuyang Chen, Linqian Zeng, Yijin ZHou +2

Shanghai Jiaotong University · Tsinghua University

Diffusion Timestep Scheduling Diffusion Model Inference Acceleration Diffusion Models Diffusion Transformer

Ask PDF

Aug 24, 2026

Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache

Diffusion Transformers achieve high-fidelity image and video generation, but their iterative sampling remains expensive, for each denoising step requires large matrix operations. Existing cache-based acceleration reduces redundant computation yet increases the VRAM footprint by storing intermediate states, which can directly constrain inference batch size. In this work, we propose a training-free acceleration method that performs stepwise forecasting for DiT sampling using a Barycentric Extrapolator. By leveraging barycentric extrapolation, our predictor is numerically stable and alleviates oscillatory artifacts analogous to the Runge phenomenon during forward forecasting. Across extensive experiments on both image and video generation, our approach provides a favorable trade-off between memory usage and perceptual quality, while delivering up to 3.30x end-to-end sampling speedup compared with baseline DiT inference.

Chengjie Lu, Tianchi Deng, Zhengqi He +3

College of Electronics and Information Engineering, Shenzhen University · School of Artificial Intelligence, Shenzhen University · College of Computer Science and Software Engineering, Shenzhen University

Diffusion Model Inference Acceleration Diffusion-Based Image Generation Diffusion Transformer Video Diffusion Models

Ask PDF

Abstract

Diffusion Model Inference Acceleration

Explore similar work

DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity

Jano: Adaptive Diffusion Generation with Early-stage Convergence Awareness

Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache