Frequency-Aware Error-Bounded Caching for Accelerating Diffusion Transformers

Guandong Li

cs.CVMar 5, 2026

Frequency-Aware Error-Bounded Caching for Accelerating Diffusion Transformers

Authors: Guandong Li

Organizations: iFLYTEK

Abstract

Diffusion Transformers (DiTs) have emerged as the dominant architecture for high-quality image and video generation, yet their iterative denoising process incurs substantial computational cost during inference. Existing caching methods accelerate DiTs by reusing intermediate computations across timesteps, but they share a common limitation: treating the denoising process as uniform across time,depth, and feature dimensions. In this work, we identify three orthogonal axes of non-uniformity in DiT denoising: (1) temporal -- sensitivity to caching errors varies dramatically across the denoising trajectory; (2) depth -- consecutive caching decisions lead to cascading approximation errors; and (3) feature -- different components of the hidden state exhibit heterogeneous temporal dynamics. Based on these observations, we propose SpectralCache, a unified caching framework comprising Timestep-Aware Dynamic Scheduling (TADS), Cumulative Error Budgets (CEB), and Frequency-Decomposed Caching (FDC). On FLUX.1-schnell at 512x512 resolution, SpectralCache achieves 2.46x speedup with LPIPS 0.217 and SSIM 0.727, outperforming TeaCache (2.12x, LPIPS 0.215, SSIM 0.734) by 16% in speed while maintaining comparable quality (LPIPS difference < 1%). Our approach is training-free, plug-and-play, and compatible with existing DiT architectures.

Figures & tables

Figure 1: Per-timestep caching sensitivity on FLUX.1-schnell (20 steps). The L2 error

\mathcal{E}(t)

exhibits an asymmetric U-shaped profile: early and late timesteps are highly sensitive, while the middle regime (

t_{6}

–

t_{14}

) is remarkably tolerant.

Figure 2: Error comparison between consecutive and randomly distributed block caching at matched cache rates on FLUX.1-schnell. Consecutive caching produces substantially higher L2 errors, confirming super-linear error accumulation through the residual stream.

Figure 3: Relative L2 change across 8 DCT frequency bands for a middle transformer block (

\ell=9

) on FLUX.1-schnell. Low-frequency components (bands 1–2) exhibit

{\sim}30\%

higher temporal volatility than high-frequency components (bands 7–8), revealing spectral heterogeneity in hidden state dynamics.

Figure 4: SpectralCache framework. Input

\mathbf{H}_{t,0}

is normalized to

\mathbf{M}_{t}

. TADS computes adaptive threshold

\tau^{\text{eff}}

. Three checks (CEB, FDC, distance) gate caching. If all pass, cached residual is reused; otherwise, full computation is performed.

Figure 5: TADS cosine bell schedule. The scaling factor

s(t)

is small at the endpoints (conservative caching) and peaks at the midpoint (aggressive caching), aligning with the U-shaped sensitivity profile from Fig. 1 .

Method	Time (s)	Speedup
No Cache	4.24	1.00 $\times$
FBCache	2.26	1.87 $\times$
TeaCache	2.00	2.12 $\times$
FastCache	0.94	4.51 $\times$
SpectralCache	1.72	2.46 $\times$

Table 1: Inference latency comparison on FLUX.1-schnell (

512{\times}512

, 20 steps, seed 42). Speedup is relative to the uncached baseline.

Method	Speedup	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
No Cache	1.00 $\times$	—	—	—
FBCache	1.87 $\times$	0.145	0.792	22.45
TeaCache	2.12 $\times$	0.215	0.734	20.51
FastCache	4.51 $\times$	0.559	0.360	14.53
SpectralCache	2.46 $\times$	0.217	0.727	20.41

Table 2: Generation quality on FLUX.1-schnell (

512{\times}512

, 20 steps, 10-image pairwise comparison against uncached baseline).

TADS	CEB	FDC	Speedup	LPIPS $\downarrow$	SSIM $\uparrow$
✗	✗	✗	2.29 $\times$	0.207	0.723
✓	✗	✗	2.04 $\times$	0.213	0.717
✗	✓	✗	2.08 $\times$	0.207	0.723
✗	✗	✓	2.12 $\times$	0.207	0.723
✓	✓	✗	1.79 $\times$	0.205	0.726
✓	✗	✓	1.74 $\times$	0.213	0.717

Table 3: Ablation study on FLUX.1-schnell (

512{\times}512

, 20 steps). Each row enables the indicated components on top of the base caching infrastructure (polynomial rescaling + accumulated distance).

Panel of Figure 6 — Figure 6: Qualitative comparison on FLUX.1-schnell ( $512{\times}512$ , 20 steps). SpectralCache and TeaCache both preserve visual quality close to the uncached baseline. SpectralCache achieves 16% higher speedup ( $2.46\times$ vs. TeaCache’s $2.12\times$ ) while maintaining comparable perceptual fidelity (LPIPS $0.217$ vs. $0.215$ ).

$\tau$	Speedup	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
0.3	1.53 $\times$	0.139	0.808	23.08
0.4	1.89 $\times$	0.187	0.756	21.17
0.5	2.24 $\times$	0.206	0.740	20.67
0.6	2.21 $\times$	0.212	0.737	20.58
0.8	2.46 $\times$	0.217	0.727	20.41

Table 4: Threshold sensitivity of SpectralCache on FLUX.1-schnell (

512{\times}512

, 20 steps, 10-image pairwise comparison).

Explore similar work

Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache
Aug 24, 2026Chengjie Lu, Tianchi Deng, Zhengqi He +3Diffusion Model Inference Acceleration Diffusion-Based Image Generation
College of Electronics and Information Engineering, Shenzhen University · School of Artificial Intelligence, Shenzhen University · College of Computer Science and Software Engineering, Shenzhen University
Rethinking What to Cache in Few-Step Diffusion Transformers: Solver-Aware Target Selection
Oct 2, 2026Shuo Yang, Lihao Fang, Yi Zhang +5Diffusion Model Caching Diffusion Model Inference Acceleration
Beijing University of Chemical Technology · ROBOT-XZC · Tsinghua University
Beyond Fixed Formulas: Data-Driven Linear Predictor for Efficient Diffusion Models
Apr 29, 2026Zhirong Shen, Rui Huang, Jiacheng Liu +6Diffusion Transformer Diffusion Model Caching
Shanghai Jiao Tong University · University of Electronic Science and Technology of China · Shandong University +2

Aug 24, 2026J/K move · Enter open · S save

Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache

Chengjie Lu, Tianchi Deng, Zhengqi He, Zhijian Gao+2

College of Electronics and Information Engineering, Shenzhen University · School of Artificial Intelligence, Shenzhen University · College of Computer Science and Software Engineering, Shenzhen University

Diffusion Model Inference Acceleration Diffusion-Based Image Generation Diffusion Transformer Video Diffusion Models

Open paper AskPDF arXiv

Diffusion Transformers achieve high-fidelity image and video generation, but their iterative sampling remains expensive, for each denoising step requires large matrix operations. Existing cache-based acceleration reduces redundant computation yet increases the VRAM footprint by storing intermediate states, which can directly constrain inference batch size. In this work, we propose a training-free acceleration method that performs stepwise forecasting for DiT sampling using a Barycentric Extrapolator. By leveraging barycentric extrapolation, our predictor is numerically stable and alleviates oscillatory artifacts analogous to the Runge phenomenon during forward forecasting. Across extensive experiments on both image and video generation, our approach provides a favorable trade-off between memory usage and perceptual quality, while delivering up to 3.30x end-to-end sampling speedup compared with baseline DiT inference.

Abstract

Explore similar work

Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache

Rethinking What to Cache in Few-Step Diffusion Transformers: Solver-Aware Target Selection

Beyond Fixed Formulas: Data-Driven Linear Predictor for Efficient Diffusion Models