Freeze What Is Settled, Cache What Is Smooth: Spatial Freezing Keeps Step Caching in Its Low-Error Regime

Baiyan Li; Ganxuan Yang

cs.CVOct 25, 2026

Freeze What Is Settled, Cache What Is Smooth: Spatial Freezing Keeps Step Caching in Its Low-Error Regime

Authors: Baiyan Li, Ganxuan Yang

Organizations: Lancaster University · Beijing Jiaotong University

Abstract

Instruction-based image editors built on diffusion transformers run tens of denoising steps. Training-free acceleration either reuses features across steps (step caching) or stops computing the regions an edit leaves untouched (region freezing), and the two have been combined before. We ask how a high speed-up target should be split between them. On Qwen-Image-2.1, freezing does more than remove tokens: it lowers the cost of every step, so the step cache can reach the same overall speed-up with a conservative threshold that keeps its error small. A per-step cost model fitted on development runs predicts held-out latency with a median error of 4.0% and reproduces the speed ceilings of the region methods we re-implemented (2.73 $\times$ predicted vs. 2.70 $\times$ measured for RegionE). Driving frozen tokens to the model's one-step $x_0$ estimate beats returning them to the reference image on every development sample, and in 90% of measured cases the composed error stays below the sum of the two individual errors. The resulting method, FreezePlan, keeps five full steps and gives the rest of the budget to TaylorSeer. On a pre-registered test split it runs 5.05 $\times$ faster end to end at 29.8 dB PSNR to the original output, with no significant drop in a VLM judge score; TaylorSeer at 5.02 $\times$ reaches 23.5 dB. The curve extends to 5.99 $\times$ at 28.0 dB. Swapping in the region selectors of prior methods under our schedule lowers median PSNR by only 0.5--0.6 dB, which places the gain in the schedule and budget split rather than in the selector.

Figures & tables

Fig. 1: FreezePlan (Sec. III-E ). (a) Schedule: $\tau$ full steps, one region decision, compact steps over active tokens with step caching, one optional refresh. (b) In a compact step, frozen tokens move straight to their anchor $c=\hat{x}_{0}(\sigma_{\tau})$ and their K/V is appended once to the prefix cache that active tokens attend to. The cached-step pattern in (a) is illustrative.

Fig. 2: Frozen-token anchor (development set, unchanged tokens). (a) Latent error reduction of the

x_{0}

anchor over the reference anchor vs. the step of the estimate (median and IQR over 44 samples). (b) Decoded PSNR of the unchanged region to the dense output and to the input.

Schedule	Full steps	Ceiling	Measured max.
RegionE [ 16 ]	13	2.73 $\times$	2.70 $\times$ (+TS)
SpotEdit [ 15 ]	7	4.45 $\times$	3.90 $\times$ (+TS)
FreezePlan	5	5.62 $\times$	5.48 $\times$
FreezePlan (no refresh)	4	6.48 $\times$	5.99 $\times$

TABLE I: Speed ceilings implied by Eq. ( 1 ) (all non-full steps free) and the highest measured speed-up on the test split.

Fig. 3: Composition on the development set. (a) Measured composed PSNR vs. the PSNR implied by summing the individual MSEs; 90% of points lie above the diagonal. (b) Speed-up vs. cache threshold

\delta

; labels give PSNR. With freezing, 5

\times

is reached at

\delta=0.3

Method	Speed $\uparrow$	PSNR $\uparrow$	Edit $\uparrow$	SSIM $\uparrow$	$\Delta$ VLM
TaylorSeer $\delta{=}0.8$	5.02 $\times$	23.45	19.71	0.707	$-$ 1.59 [ $-$ 1.93, $-$ 1.25]
TaylorSeer $\delta{=}1.0$	5.66 $\times$	19.39	17.11	0.522	$-$ 4.11 [ $-$ 4.67, $-$ 3.53]
DBCache $\delta{=}0.8$	5.03 $\times$	23.33	19.65	0.700	$-$ 1.67 [ $-$ 2.01, $-$ 1.33]
6 steps	5.17 $\times$	22.22	17.51	0.869	$-$ 0.23 [ $-$ 0.38, $-$ 0.10]
SpotEdit+TS †	3.90 $\times$	26.14	20.00	0.884	$-$ 0.44 [ $-$ 0.78, $-$ 0.17]
RegionE+TS †	2.70 $\times$	32.93	27.60	0.960	$+$ 0.12 [ $-$ 0.10, $+$ 0.47]

TABLE II: Test split B (44 samples, pre-registered). Medians;

\Delta

VLM: mean paired difference to the dense output with 95% CI. † Fastest setting the method reaches.

Fig. 4: Speed–fidelity on test split B. (a) PSNR to the dense output vs. real speed-up; (b)

\Delta

VLM. Shaded: the

\geq

\times

regime.

Variant	Speed	PSNR	$\rho$	$\Delta$ VLM
FreezePlan selector	5.03 $\times$	29.77	0.41	$-$ 0.06
RegionE selector [ 16 ]	5.23 $\times$	29.29	0.31	$+$ 0.05
SpotEdit selector [ 15 ]	5.01 $\times$	29.14	0.43	$-$ 0.16
reference anchor (SpotEdit-style)	5.06 $\times$	27.84	0.41	$-$ 0.40
no frozen K/V	5.15 $\times$	29.02	0.41	$-$ 0.21
no global fallback	5.11 $\times$	28.84	0.37	$-$ 0.15

TABLE III: Top: region selectors under the schedule and cache of FreezePlan (post hoc). Bottom: one component removed at a time. Test split B;

\rho

: mean fraction of tokens computed per step.

Fig. 5: “Replace the cup in her hands with a slice of strawberry cake” on an original anime illustration (outside the benchmark), at about 5 $\times$ real speed-up. Labels: speed-up $\cdot$ PSNR to the 40-step output; bottom row: crop of the edited region. Step caching alone (TaylorSeer, DBCache) breaks down; fewer steps give a plausible but different cake; the re-implemented region methods (SpotEdit, RegionE, both with TaylorSeer) cannot reach this speed with their schedules. FreezePlan (5.11 $\times$ ) matches the dense output.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Share of cases below the bound	90% (95% CI 86–94%)
Median ratio composed / bound	0.76 (95% CI 0.68–0.81)
Exceptions	18 (all at $\delta=0.3$ )
with refresh at step 16 / 21	15 / 3
distinct samples	15 of 30
Excess over the bound (median / max.)	1.5 / 4.0 dB
Cache share of the bound (exceptions / others)	0.56 / 0.40

Appendix

TABLE IV: Composed MSE relative to the additive bound (180 cases). CIs bootstrap over samples.

Configuration	Measured	Predicted	Time error
TaylorSeer $\delta=0.3$ / 0.8 / 1.0	4.09 / 5.02 / 5.66 $\times$	4.15 / 5.14 / 5.84 $\times$	1.3–2.3%
8 / 7 / 6 steps*	4.15 / 4.61 / 5.17 $\times$	4.03 / 4.45 / 4.97 $\times$	3.4–4.5%
RegionE+TS*	2.66–2.70 $\times$	2.61–2.64 $\times$	2.5%
SpotEdit+TS*	3.81–3.90 $\times$	4.09–4.16 $\times$	6.4%
FreezePlan+TS (all variants)	5.05–5.99 $\times$	4.78–5.59 $\times$	4.9–6.0%
All 1056 runs			median 4.0%, P90 7.9%

Appendix

TABLE V: Out-of-sample prediction on split B (1056 runs). Configurations marked * do not appear in the development set.

Subset	Samples	Fallback	$\rho$ (no fallback)
Global categories	20	9 (style 4/4, background 4/4, tone 1/4)	0.37
Local categories	24	2 (colour 1, text 1)	0.40
All	44	11 (25%)	0.38

Appendix

TABLE VI: Fallback rate and active fraction on split B.

Fig. 6: Split B, about 5 $\times$ ; rows chosen by the rule above (crops of the edited region below each row). Labels: speed-up $\cdot$ PSNR to the 40-step output.

Fig. 7: All five original anime illustrations, about 5 $\times$ , same configurations as split B.

Diffusion Model Inference Acceleration

Diffusion model inference acceleration reduces the time and compute needed to generate or reconstruct outputs while preserving quality.

See all 506 papers

Frequency-Aware Error-Bounded Caching for Accelerating Diffusion Transformers
Mar 5, 2026Guandong LiDiffusion Model Caching Diffusion Model Inference Acceleration
iFLYTEK
Budget-Constrained Step-Level Diffusion Caching
Jun 11, 2026Mingkun Lei, Tong Zhao, Liangyu Yuan +1Diffusion Model Caching Diffusion Model Inference Acceleration
AGI Lab, Westlake University
SpecEdit: Training-Free Acceleration for Diffusion based Image Editing via Semantic Locking
May 4, 2026Zhengan Yan, Shikang Zheng, Haoran Qin +9Diffusion Sampling Diffusion-Based Image Editing
EPIC Lab , Shanghai Jiao Tong University · Shandong University · Alibaba Group +2

Mar 5, 2026J/K move · Enter open · S save

Frequency-Aware Error-Bounded Caching for Accelerating Diffusion Transformers

Guandong Li

iFLYTEK

Diffusion Model Caching Diffusion Model Inference Acceleration Diffusion Transformer Diffusion Timestep Scheduling

Open paper AskPDF arXiv

Diffusion Transformers (DiTs) have emerged as the dominant architecture for high-quality image and video generation, yet their iterative denoising process incurs substantial computational cost during inference. Existing caching methods accelerate DiTs by reusing intermediate computations across timesteps, but they share a common limitation: treating the denoising process as uniform across time,depth, and feature dimensions. In this work, we identify three orthogonal axes of non-uniformity in DiT denoising: (1) temporal -- sensitivity to caching errors varies dramatically across the denoising trajectory; (2) depth -- consecutive caching decisions lead to cascading approximation errors; and (3) feature -- different components of the hidden state exhibit heterogeneous temporal dynamics. Based on these observations, we propose SpectralCache, a unified caching framework comprising Timestep-Aware Dynamic Scheduling (TADS), Cumulative Error Budgets (CEB), and Frequency-Decomposed Caching (FDC). On FLUX.1-schnell at 512x512 resolution, SpectralCache achieves 2.46x speedup with LPIPS 0.217 and SSIM 0.727, outperforming TeaCache (2.12x, LPIPS 0.215, SSIM 0.734) by 16% in speed while maintaining comparable quality (LPIPS difference < 1%). Our approach is training-free, plug-and-play, and compatible with existing DiT architectures.

Abstract

Diffusion Model Inference Acceleration

Explore similar work

Frequency-Aware Error-Bounded Caching for Accelerating Diffusion Transformers

Budget-Constrained Step-Level Diffusion Caching

SpecEdit: Training-Free Acceleration for Diffusion based Image Editing via Semantic Locking