Heuristics are Not Enough: Calibrated and Corrected Caching for Video Diffusion Acceleration

Jiyue Jiang; Zhiyu Huo; Yuhan Chen; Yanruisheng Shao; Gong Zhang; Lei Deng

cs.CVOct 12, 2026

Heuristics are Not Enough: Calibrated and Corrected Caching for Video Diffusion Acceleration

Authors: Jiyue Jiang, Zhiyu Huo, Yuhan Chen, Yanruisheng Shao, Gong Zhang, Lei Deng

Organizations: Department of Computer Science and Engineering, The Chinese University of Hong Kong · Theory Lab, 2012 Labs, Huawei Technology Co. Ltd

Abstract

Feature caching has become a mainstream training-free strategy for accelerating video diffusion models by reusing intermediate outputs across denoising steps. However, existing adaptive methods rely on heuristic difference measures misaligned with actual generation quality, employ a single global threshold that ignores heterogeneous phase dynamics, and never validate their budget estimates against true approximation error, allowing systematic bias to propagate uncorrected. We propose AlignCache, a closed-loop caching framework that directly aligns scheduling decisions with their true influence on the final generated video through three tightly coupled components: (1) phase-adaptive budget routing that partitions the denoising trajectory into structurally distinct phases, each governed by an independent threshold and a staleness-aware guard; (2) closed-loop threshold calibration that exploits counterfactual error measurement at the classifier-free guided output level to adjust decision boundaries online via an EMA feedback loop; and (3) momentum-augmented draft synthesis combining direction-magnitude gated residual extrapolation with anchor-guided unconditional correction. By transforming scheduling from a static risk accumulator into a feedback-driven, self-correcting controller, AlignCache consistently achieves superior acceleration-quality trade-offs across state-of-the-art video generation models. Our code is available at https://github.com/jiangjyjy/AlignCache.

Figures & tables

Figure 1: Overview and motivation of AlignCache. (a) Existing schedulers rely on predefined rules and prior knowledge to allocate exact and cached steps, while AlignCache leverages real-time feedback and self-correction. (b) AlignCache dominates other methods on PSNR, LPIPS, and speedup. (c) Comparison on “The bund Shanghai, pan left.” : AlignCache preserves fine details where competing methods distort (red boxes). Videos in (c) are generated by HunyuanVideo.

Figure 2: Empirical analysis of residual dynamics across denoising steps on Wan2.1-1.3B. (a) Residual magnitude exhibits distinct early, middle, and late phases. (b) Residual correction yields large error reduction over direct reuse in mid-to-late phases. (c) Approximate unconditional reuse provides consistent benefit across the trajectory. (d) Direct residual reuse error grows along the trajectory and differs across branches.

Figure 3: Overview of AlignCache. (a) Pipeline of phase-adaptive budget routing. The system decides whether to skip or perform exact computation based on the cumulative risk budget $B_{t}$ , age cap $A_{p_{t}}$ , and dynamic threshold $\theta_{p,t}$ . $F_{t}$ is the input at timestep $t$ , $\alpha_{t}$ the anchor age, and $\mathcal{W}$ denotes warm-up and tail timesteps unconditionally computed exactly. (b) Workflow of closed-loop threshold calibration. The process computes exact outputs and counterfactual skip drafts, then derives the skip distortion. The calibration ratio is updated from observed distortion and combined with the dynamic threshold. The resulting scale factor and phase partition $\theta_{p}$ determine $\theta_{p,t}$ in part (a). (c) Process of momentum-augmented draft synthesis. The current residual is predicted from the two most recent exact anchors $a(t)$ and $a^{-}(t)$ , applying momentum correction when direction and magnitude align, or a first-order estimate otherwise. The final drafts $(\hat{\mathbf{c}}_{t},\hat{\mathbf{u}}_{t})$ feed into part (b).

Methods	Efficiency		Visual Retention				VBench
Methods	Latency (s) $\downarrow$	Speedup $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	GMSD $\downarrow$	Total (%) $\uparrow$
Wan2.1-1.3B Wan et al. (2025) (81 frames, 832 $\times$ 480)
Wan2.1-1.3B ( $T{=}50$ )	131.9	1.00 $\times$	–	–	–	–	81.64
+ PAB Zhao et al. (2024)	74.2	1.78 $\times$	21.42	0.780	0.162	0.164	81.00
+ TeaCache Liu et al. (2025a)	86.2	1.53 $\times$	27.57	0.903	0.055	0.100	81.23
+ FasterCache Lv et al. (2024)	75.9	1.74 $\times$	26.35	0.891	0.063	0.109	81.41

Table 1: Comparison between other acceleration strategies and our AlignCache. We report inference latency, speedup, visual retention metrics (PSNR, SSIM, LPIPS, and GMSD), and VBench Total score on representative video generation models. The baseline is reported with the original

T

denoising steps. “

+

” denotes applying the corresponding acceleration strategy on top of the baseline.

Variant	Lat.(s) $\downarrow$	Spd. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
(a) Warm-up steps $W$
$W{=}5$	42.73	2.29 $\times$	23.68	0.829	0.123
$W{=}7$	43.12	2.26 $\times$	24.40	0.850	0.110
$W{=}9$	43.72	2.22 $\times$	24.51	0.862	0.104
$W{=}12$	46.82	2.06 $\times$	25.97	0.882	0.086
$W{=}14$	49.03	1.97 $\times$	26.34	0.887	0.078

Table 2: Ablations on scheduler-related axes on Wan2.1-1.3B. Representative configurations shown; full grid in App. A.6 .

(a) Motion gain $(\beta_{0}^{(p)},s)$
Variant	Lat.(s) $\downarrow$	Spd. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
$(0.50,\,1.00)$	49.76	1.94 $\times$	25.58	0.873	0.089
$(0.75,\,1.00)$	62.23	1.56 $\times$	30.21	0.937	0.042
$(0.80,\,0.75)$	63.73	1.51 $\times$	29.23	0.930	0.045
$(0.80,\,0.95)$	67.25	1.44 $\times$	32.70	0.954	0.030
$(0.80,\,1.00)$	68.23	1.41 $\times$	32.22	0.954	0.029

Table 3: Ablations on predictor-related axes on Wan2.1-1.3B. Subgroups (a)-(b) vary motion-gain parameters and residual extrapolation order, respectively.

Figure 4: Qualitative comparison on “A shark is swimming in the ocean, pan right.” (left, Wan2.1-1.3B) and “A person is shining shoes.” (right, Open-Sora-1.2) under matched or higher speedup settings. Red boxes highlight regions where competing methods introduce noticeable distortions, while AlignCache faithfully preserves object geometry and fine details at a comparable or higher speedup. More test method case studies can be found in the Appendix A.7 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

$W$	Lat.(s) $\downarrow$	Spd. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
5	42.73	2.29 $\times$	23.68	0.829	0.123
6	44.17	2.19 $\times$	23.51	0.810	0.129
7	43.12	2.26 $\times$	24.40	0.850	0.110
8	44.42	2.18 $\times$	24.05	0.840	0.112
9	43.72	2.22 $\times$	24.51	0.862	0.104
12	46.82	2.06 $\times$	25.97	0.882	0.086

Appendix

Table 4: Full grid for warm-up steps

W

$\theta_{p}$	Lat.(s) $\downarrow$	Spd. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
0.0325	53.77	1.81 $\times$	27.20	0.904	0.070
0.0350	54.06	1.79 $\times$	26.58	0.892	0.076
0.0375	50.29	1.94 $\times$	26.04	0.888	0.082
0.0400	50.42	1.91 $\times$	25.48	0.870	0.090
0.0425	47.21	2.06 $\times$	25.58	0.880	0.088
0.0475	45.06	2.16 $\times$	24.93	0.868	0.098

Appendix

Table 5: Full grid for error base threshold

\theta_{p}

$(\rho_{e},\rho_{l})$	Lat.(s) $\downarrow$	Spd. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
(0.100, 0.86)	45.17	2.14 $\times$	24.70	0.860	0.100
(0.125, 0.84)	44.48	2.18 $\times$	25.13	0.874	0.095
(0.125, 0.88)	44.29	2.19 $\times$	25.11	0.873	0.095
(0.150, 0.80)	45.15	2.14 $\times$	24.61	0.859	0.102
(0.150, 0.82)	44.46	2.18 $\times$	25.00	0.872	0.097
(0.150, 0.85)	45.12	2.14 $\times$	24.61	0.859	0.102

Appendix

Table 6: Full grid for phase ratio

(\rho_{e},\rho_{l})

. The full 11-point grid confirms low sensitivity: PSNR varies by

<0.55

dB across all configurations.

Figure 5: Qualitative comparison on “A person is kissing.” generated by Wan2.1-1.3B. Red boxes highlight facial regions where PAB, TeaCache, FasterCache, and EasyCache introduce distortions in lip contour, skin tone, or identity-bearing details, while AlignCache preserves facial geometry and expression cues at the same $2.38\times$ speedup as EasyCache.

Figure 6: Qualitative comparison on “A bird building a nest from twigs and leaves.” generated by Open-Sora-1.2. Red boxes mark high-frequency texture regions (nest twigs, foliage, feathers) where competing methods produce blurring, structural collapse, or color drift, whereas AlignCache retains fine woven structure and feather details at the highest $1.64\times$ speedup among compared methods.

Diffusion Model Caching

Diffusion model caching reuses features, activations, or attention states across denoising steps to reduce inference cost while preserving output quality.

See all 107 papers

MORCA: Offline-to-Online Reinforcement Learning for Adaptive Cache Reuse in Video Diffusion Acceleration
Oct 7, 2026Yuxiang Xiong, Ruiyan Wang, Wenqiang Wang +5Diffusion Model Caching Diffusion Model Inference Acceleration
Shanghai Jiao Tong University · Alibaba Cloud Computing · Alibaba Token Hub, Alibaba Group
SenCache: Accelerating Diffusion Model Inference via Sensitivity-Aware Caching
Feb 27, 2026Yasaman Haghighi, Alexandre AlahiVideo Diffusion Models Diffusion Model Caching
Ecole Polytechnique Fédérale de Lausanne (EPFL)
ACID: Adaptive Caching for vIDeo generation
Jul 14, 2026Om Agrawal, Saurabh Agarwal, Aditya AkellaVideo Diffusion Models Diffusion Model Caching
UT Austin

Oct 7, 2026J/K move · Enter open · S save

MORCA: Offline-to-Online Reinforcement Learning for Adaptive Cache Reuse in Video Diffusion Acceleration

Yuxiang Xiong, Ruiyan Wang, Wenqiang Wang, Teng Hu+4

Shanghai Jiao Tong University · Alibaba Cloud Computing · Alibaba Token Hub, Alibaba Group

Diffusion Model Caching Diffusion Model Inference Acceleration Video Diffusion Models

Open paper AskPDF arXiv

Diffusion Transformers (DiTs) achieve remarkable performance in video synthesis, but their iterative denoising process suffers from high inference latency. To address this, caching has emerged as an effective acceleration strategy by capitalizing on inter-step redundancy during denoising. Existing dynamic caching methods typically estimate the error that cache reuse would introduce at each denoising step (step error) to guide cache decisions, whereas our concern is how much quality loss cache reuse would cause in the final generated video (terminal error). We show that step error does not directly correspond to terminal error and that latent information helps capture their relationship, thereby informing cache decisions. Moreover, existing threshold-based methods cannot provide precise speedup control, making it difficult to meet practical requirements for user-specified acceleration targets. To address these limitations, we introduce MORCA, a cache scheduling framework trained through offline-to-online reinforcement learning to make latent-aware reuse/recompute decisions under user-specified acceleration targets. Extensive experiments on different video generation models across multiple target acceleration ratios demonstrate that MORCA achieves better generation fidelity than state-of-the-art caching methods under comparable computational budgets. Code is available at https://github.com/x10ngyx/MORCA.

Abstract

Diffusion Model Caching

Explore similar work

MORCA: Offline-to-Online Reinforcement Learning for Adaptive Cache Reuse in Video Diffusion Acceleration

SenCache: Accelerating Diffusion Model Inference via Sensitivity-Aware Caching

ACID: Adaptive Caching for vIDeo generation