Rethinking What to Cache in Few-Step Diffusion Transformers: Solver-Aware Target Selection

Shuo Yang; Lihao Fang; Yi Zhang; Haixiang Wang; Xincheng Ye; Shufan Chen; Jipeng Guo; Youqing Wang

cs.CVOct 2, 2026

Rethinking What to Cache in Few-Step Diffusion Transformers: Solver-Aware Target Selection

Authors: Shuo Yang, Lihao Fang, Yi Zhang, Haixiang Wang, Xincheng Ye, Shufan Chen, Jipeng Guo, Youqing Wang

Organizations: Beijing University of Chemical Technology · ROBOT-XZC · Tsinghua University

Abstract

Diffusion Transformers (DiTs) can generate high-quality images and videos, but generating each sample requires multiple costly DiT forward passes. Two common ways to accelerate DiT sampling are step distillation, which reduces the number of sampling steps, and caching, which skips some DiT evaluations by reusing a tensor computed at an earlier step. Most caching methods decide in advance which tensor to reuse. After distillation, adjacent sampling steps are farther apart. Reusing a tensor across this larger gap introduces more error, so choosing what to cache becomes especially important. We therefore introduce AutoTarget, a method that chooses the cached tensor for a given model, solver, and reuse schedule. AutoTarget uses a small set of runs without cache reuse to measure the error caused by reusing each candidate tensor, then selects the candidate with the lowest error. We also analyze how an error at one reuse step affects the final sample. For Euler sampling, we identify cache targets that produce the same trajectory and show why a stored solver update may not. Experiments on distilled image and video DiTs show that the best cache target changes with the model, image resolution, and solver. AutoTarget reduces DiT evaluations and retained cache storage. Generation quality remains close to the corresponding uncached run. On the tested PixArt-LCM and FLUX.1-schnell settings, its calibration ranking matches the ranking from held-out cached runs. To help others reproduce the method, we provide its core implementation on GitHub at https://github.com/wali1024-offical/AutoTarget.

Figures & tables

Figure 1: Examples generated by AutoTarget with FLUX.1-schnell.

Figure 2: A DiT sampling step. AutoTarget compares the outputs passed to the solver, rather than internal features.

Figure 3: Cache-target selection across configurations. Risks are normalized by the row minimum; held-out columns report top-1 wins and PSNR gains over the runner-up.

Figure 4: AutoTarget calibrates cache targets on uncached runs and reuses the accepted target on new prompts.

FLUX.1-dev step-budget reference
Method	Efficiency			Image Quality		Paired-output Fidelity		Retained cache
	Latency (s) $\downarrow$	Overall $\uparrow$	Cache-only $\uparrow$	ImageReward $\uparrow$	CLIP $\uparrow$	PSNR (dB) $\uparrow$	SSIM $\uparrow$	MiB $\downarrow$
FLUX.1-dev (50 steps)	17.92	1.00 $\times$	N/A	0.985	32.187	$\infty$	1.0000	0.00
FLUX.1-dev (25 steps)	9.31	1.93 $\times$	N/A	0.967	32.041	29.431	0.7315	0.00
FLUX.1-dev (20 steps)	8.14	2.20 $\times$	N/A	0.946	30.912	29.037	0.6991	0.00
FLUX.1-dev (17 steps)	6.71	2.67 $\times$	N/A	0.912	29.308	28.795	0.6723	0.00

Table 1: Text-to-image results. Overall speedup is relative to 50-step FLUX.1-dev; cache-only speedup is relative to each uncached few-step model. Fidelity uses paired 50-step FLUX.1-dev outputs for FLUX and paired uncached four-step outputs for PixArt-LCM.

Method	Latency (s) $\downarrow$	Overall $\uparrow$	Cache-only $\uparrow$	VBench Semantic $\uparrow$	VBench Quality $\uparrow$	VBench Total $\uparrow$
HunyuanVideo (50 steps)	1136.2	1.00 $\times$	N/A	72.4	80.5	78.9
HunyuanVideo (CFG-distilled, 20 steps)	231.5	4.91 $\times$	1.000 $\times$	63.2	78.6	75.5
TeaCache ( $l=0.15$ )	234.8	4.84 $\times$	0.986 $\times$	64.5	79.3	76.3
DiCache	302.8	3.75 $\times$	0.765 $\times$	58.8	75.6	72.2
TaylorSeer ( $N=3$ , $O=1$ )	265.0	4.29 $\times$	0.874 $\times$	64.3	79.4	76.4
HiCache	260.5	4.36 $\times$	0.889 $\times$	64.9	79.6	76.7

Table 2: HunyuanVideo results. Overall and cache-only speedups are relative to the 50-step and uncached CFG-distilled 20-step models, respectively.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Calibration and held-out cached-run rankings agree in the verified settings. The fixed rule

\gamma\geq 1.10

abstains on shifted LTX-Video.

Figure 6: For PixArt-LCM and FLUX.1-schnell, calibration selects denoised/clean and velocity, respectively. Each wins on all 20 held-out prompts; bars and bands show paired 95% bootstrap intervals.

Rank	Candidate target	Calibration risk	Risk / selected
1	velocity	0.005963	1.000
2	denoised residual	0.008601	1.442
3	denoised/clean	0.010180	1.707
4	$\epsilon$	0.017475	2.931

Appendix

Table 3: PixArt-LCM

512^{2}

calibration errors for the controlled setting in Figure 3 . Each value is the mean squared decoded RGB error over four prompts and one reuse step. Velocity has the lowest error. The ratio of the second-lowest error to the lowest is

\gamma=1.442

Configuration	Prompts	Calls	Cached PSNR	Anchor-only PSNR	Cached $-$ anchor-only	Default three-step PSNR
FLUX.1-schnell, Euler	50	3	32.575	32.577	$-0.001$ [-0.004, 0.001]	N/A
PixArt-LCM	50	3	23.930	23.914	$+0.016$ [0.013, 0.019]	17.502

Appendix

Table 4: Reduced-step controls with the same number of DiT evaluations. “Anchor-only” runs the solver only at the steps that refresh the cache. “Default three-step” uses the standard three-step schedule from the LCM pipeline. Brackets show paired 95% bootstrap confidence intervals over prompts.

Method	Threshold	FCE	Aggregate PSNR	SSIM	Latency (s)
AutoTarget	N/A	3.000	22.952	0.777	0.123
TeaCache	0.10	4.000	exact	1.000	0.157
TeaCache	0.30	4.000	exact	1.000	0.158
TeaCache	0.55	3.000	6.988	0.053	0.121
TeaCache	0.80	3.000	5.766	0.037	0.122
DiCache	0.10	4.000	exact	1.000	0.157

Appendix

Table 5: PixArt-LCM baseline sweep on 50 paired prompts. Aggregate PSNR uses mean RGB MSE and remains finite when some outputs are exact. FCE is the mean number of full DiT evaluations.

Solver	Selected	Ranking	$\epsilon$ PSNR	clean PSNR	velocity PSNR	Finite clean
DDIM	$\epsilon$	$\epsilon>v>\mathrm{clean}$	14.154	non-finite	13.998	0/20
DPM-Solver++ (order 2)	$\epsilon$	$\epsilon>\mathrm{clean}>v$	18.145	17.936	17.760	20/20

Appendix

Table 6: Solver comparison on the same PixArt-XL model at

1024^{2}

. Both solvers use 20 steps, the same 20 prompts and paired seeds, and ten full DiT evaluations at the same even-numbered steps. The table reports candidates even when they produce non-finite values.

Figure 7: HunyuanVideo speed, fidelity, and retained-cache size on 20 matched prompts. The selected native target retains 288 KiB; cache size excludes temporary runtime memory.

Figure 8: Velocity gives higher fidelity to the exact output than reusing the next latent. The paired CLIPScore differences are inconclusive and do not follow the PSNR gains.

Figure 9: PixArt-LCM target fidelity on 200 DrawBench prompts (left) and selection stability from 32 calibration runs with 1,000 sampled subsets per set size (right).

Explore similar work

Mar 5, 2026

Frequency-Aware Error-Bounded Caching for Accelerating Diffusion Transformers

Diffusion Transformers (DiTs) have emerged as the dominant architecture for high-quality image and video generation, yet their iterative denoising process incurs substantial computational cost during inference. Existing caching methods accelerate DiTs by reusing intermediate computations across timesteps, but they share a common limitation: treating the denoising process as uniform across time,depth, and feature dimensions. In this work, we identify three orthogonal axes of non-uniformity in DiT denoising: (1) temporal -- sensitivity to caching errors varies dramatically across the denoising trajectory; (2) depth -- consecutive caching decisions lead to cascading approximation errors; and (3) feature -- different components of the hidden state exhibit heterogeneous temporal dynamics. Based on these observations, we propose SpectralCache, a unified caching framework comprising Timestep-Aware Dynamic Scheduling (TADS), Cumulative Error Budgets (CEB), and Frequency-Decomposed Caching (FDC). On FLUX.1-schnell at 512x512 resolution, SpectralCache achieves 2.46x speedup with LPIPS 0.217 and SSIM 0.727, outperforming TeaCache (2.12x, LPIPS 0.215, SSIM 0.734) by 16% in speed while maintaining comparable quality (LPIPS difference < 1%). Our approach is training-free, plug-and-play, and compatible with existing DiT architectures.

Guandong Li

iFLYTEK

Diffusion Model Caching Diffusion Model Inference Acceleration Diffusion Transformer Diffusion Timestep Scheduling

Ask PDF

Aug 24, 2026

Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache

Diffusion Transformers achieve high-fidelity image and video generation, but their iterative sampling remains expensive, for each denoising step requires large matrix operations. Existing cache-based acceleration reduces redundant computation yet increases the VRAM footprint by storing intermediate states, which can directly constrain inference batch size. In this work, we propose a training-free acceleration method that performs stepwise forecasting for DiT sampling using a Barycentric Extrapolator. By leveraging barycentric extrapolation, our predictor is numerically stable and alleviates oscillatory artifacts analogous to the Runge phenomenon during forward forecasting. Across extensive experiments on both image and video generation, our approach provides a favorable trade-off between memory usage and perceptual quality, while delivering up to 3.30x end-to-end sampling speedup compared with baseline DiT inference.

Chengjie Lu, Tianchi Deng, Zhengqi He +3

College of Electronics and Information Engineering, Shenzhen University · School of Artificial Intelligence, Shenzhen University · College of Computer Science and Software Engineering, Shenzhen University

Diffusion Model Inference Acceleration Diffusion-Based Image Generation Diffusion Transformer Video Diffusion Models

Ask PDF

May 24, 2026

Trajectory-Consistent Calibration for Cache-Accelerated Diffusion Models

Diffusion Transformers require repeated denoiser evaluations during iterative sampling, making inference computationally expensive. Cache-based acceleration reduces this cost by reusing intermediate representations across denoising steps, but can introduce representation deviations and degrade generation quality. In this paper, we analyze these deviations and show that effective calibration should consider both the direct mismatch caused by reuse and the subsequent trajectory shift induced by earlier corrections. To address this challenge, we propose Trajectory-Consistent Calibration (TCC), a training-free method that calibrates cached representations toward their full-computation counterparts. Specifically, rather than estimating all calibration priors from a single uncorrected cache trajectory, TCC uses an offline iterative procedure so that each prior accounts for the trajectory shift induced by preceding calibrations. Experiments on PixArt-alpha and DiT-XL/2 show that TCC consistently improves FID across representative cache-based acceleration methods while preserving their underlying reuse policies. Notably, in a representative PixArt-alpha cache-acceleration setting based on FORA, TCC reduces FID from 29.83 to 27.35, slightly surpassing the full-computation baseline.

Mingyu Liang, Dingkun Xu, Jingwei Xu

Laboratory for Novel Software Technology, Nanjing University, China

Diffusion Transformer Diffusion-Based Image Generation Diffusion Model Caching Diffusion Model Inference Acceleration

Ask PDF