Diffusion-based video editing has emerged as an important paradigm for high-quality and flexible content generation. However, despite their generality and strong modeling capacity, Diffusion Transformers (DiT) remain computationally expensive due to the iterative denoising process, posing challenges for practical deployment. Existing video diffusion acceleration methods primarily exploit denoising timestep-level feature reuse, which mitigates the redundancy in denoising process, but overlooks the architectural redundancy within the DiT that many attention operations over spatio-temporal tokens are redundantly executed, offering little to no incremental contribution to the model output. This work introduces HetCache, a training-free diffusion acceleration framework designed to exploit the inherent heterogeneity in diffusion-based masked video-to-video (MV2V) generation and editing. Instead of uniformly reuse or randomly sampling tokens, HetCache assesses the contextual relevance and interaction strength among various types of tokens in designated computing steps. Guided by spatial priors, it divides the spatial-temporal tokens in DiT model into context and generative tokens, and selectively caches the context tokens that exhibit the strongest correlation and most representative semantics with generative ones. This strategy reduces redundant attention operations while maintaining editing consistency and fidelity. Experiments show that HetCache achieves a noticeable acceleration, including a 2.67× latency speedup and FLOPs reduction over commonly used foundation models, with negligible degradation in editing quality.
Figures & tables
Figure 1: (a). Illustration of the acceleration dimensions in Diffusion Transformers (DiTs). Unlike existing methods, the proposed Heterogeneous Caching (HetCache) jointly models denoising-step redundancy in the diffusion process and token redundancy within the Transformer backbone. (b). As a tailored heterogeneous strategy, HetCache accelerates diffusion-based masked video-to-video (MV2V) editing while maintaining generation quality.
Figure 2: The overview of our proposed HetCache scheme. In denoising process, we use the timestep-embeddings-modulated-input [ 18 ] to estimate the computing demand. According to the accumulated distance, Full-Compute anchor step, Reuse step and Partial-Compute step will be executed. In full-computing, HetCache will use spatial prior extracted from editing mask to categorize the DiT tokens into Context , Margin , and Generative Tokens. The Context Tokens which takes high portion and cause redundant computation cost will be cached for partial-compute steps according to its semantic representativeness and interaction strength with the generative tokens.
Video Inpainting on VACE-Benchmark
Method
Step
Efficiency
Visual Quality
FLOPs (P) ↓
Latency (s) ↓
Speed ↑
PSNR ↑
SSIM ↑
VFID ↓
VBench (%) ↑
Wan2.1-VACE
100
145.21
445.52
1.00×
16.06
0.56
57.18
76.54
Timestep Reduction
50
72.60
238.84
1.86×
16.46
0.56
54.96
76.78
PAB
50
43.56
223.20
1.99×
16.46
0.56
56.04
76.73
AdaCache
50
39.93
242.22
1.83×
16.46
0.56
54.96
76.78
Table 1: Quantitative evaluation of inference efficiency and visual quality in video generation models. HetCahce achieves superior efficiency and better visual quality across different base models, sampling schedulers, video resolutions, and lengths.
Figure 3: VBench comparison between HetCache and other methods on different video editing tasks.
Figure 4: Visualization of different video editing tasks. HetCache produces relatively high-quality results while other methods suffer from smoothness, ghosting, and blurring issues.
Method
FLOPs ↓
Lat. ↓
Spd. ↑
PSNR ↑
SSIM ↑
VFID ↓
VB ↑
Higher-resolution (25 × 720P) Video Inpainting on VACE-Benchmark
Wan-VACE-1.3B (100)
252.97
662.49
1.00 ×
12.61
0.40
71.04
74.94
TeaCache (50)
126.48
276.05
2.40 ×
13.01
0.41
73.71
75.03
HetCache (50)
107.89
227.54
2.91 ×
13.03
0.41
71.40
75.68
Longer (57 × 480P) Video Inpainting on VACE-Benchmar
Wan-VACE-1.3B (100)
421.00
892.66
1.00 ×
16.41
0.51
50.33
75.84
Table 2: Additional evaluation results under different settings.
Method
Guidance
Efficiency
Visual Quality
Context
Correlation
Latency ↓
Speed ↑
PSNR ↑
SSIM ↑
VFID ↓
VBench-Score ↑
HetCache - -
×
×
142.46
3.13×
16.60
0.56
54.54
76.19
HetCache -
✓
×
152.14
2.93×
16.54
0.56
54.75
75.80
HetCache -
×
✓
177.46
2.51×
16.60
0.56
55.36
76.24
HetCache
✓
✓
166.81
2.67×
16.58
0.56
54.51
76.29
Table 3: Quantitative ablation study results.
Figure 5: Key metircs comparison of different K and rctx setting in context token sampling.
Figure 6: Visualization of ablation study, with and without clustering and correlation guidance will impact the generation quality.
Video Diffusion Transformer (DiT) models are a dominant approach for high-quality video generation but suffer from high inference cost due to iterative denoising. Existing caching approaches primarily exploit similarity within the diffusion process of a single request to skip redundant denoising steps. In this paper, we introduce Chorus, a caching approach that leverages similarity across requests to accelerate video diffusion model serving. Chorus achieves up to 45% speedup on industrial 4-step distilled models, where prior intra-request caching approaches are ineffective. Particularly, Chorus employs a three-stage caching strategy along the denoising process. Stage 1 performs full reuse of latent features from similar requests. Stage 2 exploits inter-request caching in specific latent regions during intermediate denoising steps. This stage is combined with Token-Guided Attention Amplification to improve semantic alignment between the generated video and the conditional prompts, thereby extending the applicability of full reuse to later denoising steps.
Hao Liu, Ye Huang, Chenghuan Huang +5
Sun Yat-sen University, China · Independent Researcher
Diffusion Transformers (DiTs) achieve remarkable performance in video synthesis, but their iterative denoising process suffers from high inference latency. To address this, caching has emerged as an effective acceleration strategy by capitalizing on inter-step redundancy during denoising. Existing dynamic caching methods typically estimate the error that cache reuse would introduce at each denoising step (step error) to guide cache decisions, whereas our concern is how much quality loss cache reuse would cause in the final generated video (terminal error). We show that step error does not directly correspond to terminal error and that latent information helps capture their relationship, thereby informing cache decisions. Moreover, existing threshold-based methods cannot provide precise speedup control, making it difficult to meet practical requirements for user-specified acceleration targets. To address these limitations, we introduce MORCA, a cache scheduling framework trained through offline-to-online reinforcement learning to make latent-aware reuse/recompute decisions under user-specified acceleration targets. Extensive experiments on different video generation models across multiple target acceleration ratios demonstrate that MORCA achieves better generation fidelity than state-of-the-art caching methods under comparable computational budgets. Code is available at https://github.com/x10ngyx/MORCA.
Yuxiang Xiong, Ruiyan Wang, Wenqiang Wang +5
Shanghai Jiao Tong University · Alibaba Cloud Computing · Alibaba Token Hub, Alibaba Group
Feature caching has become a mainstream training-free strategy for accelerating video diffusion models by reusing intermediate outputs across denoising steps. However, existing adaptive methods rely on heuristic difference measures misaligned with actual generation quality, employ a single global threshold that ignores heterogeneous phase dynamics, and never validate their budget estimates against true approximation error, allowing systematic bias to propagate uncorrected. We propose AlignCache, a closed-loop caching framework that directly aligns scheduling decisions with their true influence on the final generated video through three tightly coupled components: (1) phase-adaptive budget routing that partitions the denoising trajectory into structurally distinct phases, each governed by an independent threshold and a staleness-aware guard; (2) closed-loop threshold calibration that exploits counterfactual error measurement at the classifier-free guided output level to adjust decision boundaries online via an EMA feedback loop; and (3) momentum-augmented draft synthesis combining direction-magnitude gated residual extrapolation with anchor-guided unconditional correction. By transforming scheduling from a static risk accumulator into a feedback-driven, self-correcting controller, AlignCache consistently achieves superior acceleration-quality trade-offs across state-of-the-art video generation models. Our code is available at https://github.com/jiangjyjy/AlignCache.
Jiyue Jiang, Zhiyu Huo, Yuhan Chen +3
Department of Computer Science and Engineering, The Chinese University of Hong Kong · Theory Lab, 2012 Labs, Huawei Technology Co. Ltd