Diffusion Transformer (DiT) models, while having become a powerful backbone for generative synthesis, suffer from high inference cost on generating high-resolution images. Diffusion caching has emerged as a promising acceleration technique to reduce inference cost. However, existing caching strategies fail to accurately identify the varying importance of different computational modules and rarely coordinate cache reusing across the timestep and layer granularities, making it difficult to control error accumulation. To address these limitations, we propose multi-granularity caching (MG-Cache) for DiTs, a novel and fully training-free approach to exploit computational redundancy both on timesteps and Transformer layers. Our core innovation contains a two-stage caching strategy: an offline timestep profiling analysis that identifies redundant timesteps for cache reuse, followed by an adaptive layer-wise caching strategy that combines input stability with importance of each Transformer layer to selectively cache stable Attention and MLP submodules. Extensive experiments on mainstream open-source DiT models validate the effectiveness of our approach, achieving significant acceleration while maintaining high fidelity to the uncached output, with up to a 4.18x acceleration on HunyuanVideo.
Figures & tables
Figure 1: Visual quality comparison. While achieving a higher speedup than baselines, our method also maintains superior fidelity to the original output. For instance, in the first row (Prompt 1), other methods exhibit blurring on the text "stop," whereas our method preserves text sharpness.
Figure 2: Overview of the Multi-Granularity Cache (MG-Cache). We design a two-stage strategy to exploit computational redundancy during inference. Stage 1 performs a static offline analysis to identify and preserve critical timesteps based on a timestep feature table [M^t] constructed from a small batch of calibration dataset. Stage 2 introduces a dynamic layer-wise caching mechanism that computes layer importance scores and input stability to adaptively determine whether to reuse cached features.
Figure 3: Different evolution patterns of intermediate features at each timestep for different prompts. The final model output predominantly exhibits prompt-independent patterns, whereas the intermediate features within layers display prompt-dependent patterns.
Model Output
Layer-15
Layer-35
Layer-55
Std
0.0114
0.1952
0.1903
0.1582
CV
0.1504
0.3419
0.3345
0.4195
Table 1: Prompt dependence of timestep and layer features.
Methods
PSNR
SSIM
LPIPS ↓
FLOPs(p)
Naive Strategy
17.07
0.61
0.43
70.21
With Scores φtl
27.44
0.86
0.13
67.74
Table 2: Comparison with adaptive layer-wise caching and naive strategy(HunyuanVideo 544×960 129frames)
Figure 4: Linear Regression. Instead of directly caching and reusing, a linear regression transformation is applied for cached features.
Figure 5: Mean value of linear regression coefficients. Different value of coefficients in each layer cache and timestep cache on HunyuanVideo.
HunyuanVideo-T2V , 544×960 p, 129 frames
Methods
Quality
Acceleration
VBench Score ↑
PSNR ↑
SSIM ↑
LPIPS ↓
FLOPs(P) ↓
Latency(s) ↓
SpeedUp ↑
Original(50 steps)
75.45%
-
-
-
214.6299
982.03
-
20 steps
75.21%
18.16
0.67
0.38
85.8519
413.38
2.38 ×
FORA( N=3 )
74.75%
17.62
0.66
0.40
101.6018
482.80
2.03 ×
Teacache( δ=0.4 )
75.33%
23.99
0.79
0.19
94.4372
457.01
2.15 ×
Table 3: Quantitative results of generation quality and inference acceleration, evaluated on the HunyuanVideo and FLUX.1-dev
Method
VBench
PSNR
SpeedUp
MG-Cache(slow)
75.14%
27.44
2.74 ×
w/o Offline Analysis
74.79%
24.15
2.41 ×
w/o Linear Regression
74.51%
26.82
2.68 ×
w/o Time- step cache
β=0.3
75.67%
29.49
1.23×
β=0.4
74.86%
26.83
1.42 ×
w/o Layer cache
α=0.04
75.21%
28.65
2.52×
Table 4: Ablation Study of each component on HunyuanVideo
Figure 6: Inference latency and added overhead with different resolution. Since our method primarily targets compute-bound components such as self-attention and MLP modules through diffusion caching and selective skipping, the achievable speedup correspondingly improves with longer sequences, while the added overhead remains below 5% of the total latency.
Figure 7: Correlation between the reuse criterion and the true model output error. Left: raw activation drift Δtl ; right: sensitivity-aware risk proxy Rtl=φtl⋅Δtl .
480×640 p, 65 frames
544×960 p, 129 frames
Sequence length L
20.7k
67.6k
Layer cache
15.2 GB
49.8 GB
Timestep cache
2.6 MB
8.6 MB
Placement
GPU memory
RAM (offloading)
Table 5: Memory overhead of MG-Cache caches (bf16) on HunyuanVideo.
Figure 8: Additional hyperparameter analysis. Left/right axes report PSNR and the skip ratio. Larger α ( β ) increases timestep (layer) skipping, improving acceleration at the cost of a gradual fidelity drop, with the layer-stage budget exhibiting a steeper trade-off.
Layer / Step
Step-0
Step-10
Step-20
Step-30
Step-40
Layer-5
0.0025
0.0043
0.0050
0.0057
0.0056
Layer-15
0.0042
0.0100
0.0095
0.0088
0.0101
Layer-25
0.0037
0.0085
0.0079
0.0089
0.0098
Layer-35
0.0066
0.0091
0.0081
0.0084
0.0097
Layer-45
0.0164
0.0057
0.0061
0.0067
0.0070
Layer-55
0.0078
0.0048
0.0048
0.0054
0.0058
Table 6: Standard deviation of layer importance scores φtl across prompts.
College of Electronics and Information Engineering, Shenzhen University · School of Artificial Intelligence, Shenzhen University · College of Computer Science and Software Engineering, Shenzhen University