IdealCache: Rethinking Cache Scheduling in Diffusion Transformers via Ideal Trajectories
Authors: Shang Xue, Li Zhu, Ping Chen
Organizations: Independent Researcher, Santa Clara, California, United States · Independent Researcher, Mountain View, California, United States · University of Electronic Science and Technology of China
Feature caching accelerates Diffusion Transformers by reusing intermediate computations across denoising timesteps, yet existing methods share a fundamental blind spot: they treat the standard reference denoising trajectory as the optimisation target, approximating its per-step states and thus inheriting its output as an implicit quality ceiling. We argue that this target is arbitrary -- what truly matters is the ideal result, not the intermediate path taken to reach it. We propose IdealCache, a training-free framework that abandons step-wise imitation of the reference trajectory. Instead, it anchors caching on a per-input ensemble of ideal trajectories constructed from candidate target results, aggregates their geometry into a single graph, and discovers optimal non-uniform schedules via a constrained shortest path on this graph. By decoupling from the reference process, IdealCache breaks the performance bottleneck of trajectory-approximation; our schedules can attain near-lossless fidelity in generation and even surpass on task-specific metrics (e.g., maintaining higher semantic consistency in image editing). Without retraining, IdealCache achieves 6.35x speedup on Qwen-Image, 5.84x on Qwen-Image-Edit, and 4.32x on HunyuanVideo, with competitive or superior quality across tasks.
Figures & tables
Figure 1: Representative IdealCache outputs across diverse prompts and tasks (Qwen-Image, Qwen-Image-Edit, HunyuanVideo1.0). Without retraining, IdealCache delivers high-fidelity generations.
Figure 2: Cache deviation as edge weight. Starting from xt∗ on the ideal trajectory, a single full pass populates the cache with residuals Δht,Δet . We then take t−k Euler updates that all reuse those residuals (red dashed), arriving at x^k . The squared distance to the ideal endpoint xk∗ is the cache-deviation edge weight w(k→t) used in our graph.
Method
Acceleration
GEdit-CN (Full)
GEdit-EN (Full)
Latency(s) ↓
Wall-clock Speed ↑
FLOPs(T) ↓
FLOPs Speed ↑
SC ↑
PQ ↑
OS ↑
SC ↑
PQ ↑
OS ↑
Vanilla: 50 steps
268.94
1.00 ×
28190.88
1.00 ×
7.68
7.51
7.41
7.82
7.54
7.54
50% steps
135.44
1.99 ×
14095.44
2.00 ×
7.70
7.53
7.44
7.77
7.52
7.47
20% steps
55.25
4.87 ×
5638.18
5.00 ×
7.65
7.42
7.35
7.73
7.46
7.44
FORA ( N=5 )
59.71
4.51 ×
5643.13
5.00 ×
7.60
7.31
7.25
7.62
7.34
7.28
DuCa ( N=6,R=90% )
67.06
4.01 ×
5897.67
4.78 ×
7.63
7.44
7.44
7.68
7.42
7.39
Table 1: Quantitative comparison of image editing on Qwen-Image-Edit, GEdit-Bench. (SC: Semantic Consistency, PQ: Perceptual Quality, OS: Overall Score.) Wall-clock speedup is the end-to-end latency ratio at batch size 1 and also includes the fixed cost of the VAE, text encoder, and scheduler; FLOPs speedup is the ratio of transformer FLOPs.
ImageReward ↑
Consistency ↑
Background Preservation ↑
50 steps (vanilla)
1.054
-
0.665
Ours ( B=25 )
1.049
0.960
0.687
Ours ( B=10 )
0.991
0.890
0.679
Table 2: Results on the FlowEdit dataset. “Consistency” measures similarity to vanilla 50-step results, while “Background Preservation” evaluates background similarity to the input image using SSIM.
Figure 3: Qualitative editing results on Qwen-Image-Edit. Our method achieves better background preservation in some cases while maintaining comparable editing quality. IdealCache with B=10 ( ∼4.2× speedup) achieves visual quality comparable to the vanilla 50-step baseline.
Method
Acceleration
Quality
Perceptual
Latency(s) ↓
Wall-clock Speed ↑
FLOPs(T) ↓
FLOPs Speed ↑
ImageReward ↑
SSIM ↑
LPIPS ↓
Vanilla: 50 steps
46.54
1.00 ×
10925.18
1.00 ×
1.2092 (+0.00%)
—
—
50% steps
23.44
1.99 ×
5462.59
2.00 ×
1.1612 ( − 3.97%)
0.798
0.249
20% steps
9.51
4.89 ×
2184.56
5.00 ×
0.8996 ( − 25.60%)
0.627
0.504
TaylorSeer ( N=3 )
22.92
2.03 ×
3929.69
2.78 ×
1.0298 ( − 14.83%)
0.504
0.628
HyCa ( N=3 )
21.95
2.12 ×
3929.69
2.78 ×
1.1914 ( − 1.47%)
0.763
0.247
Table 3: Quantitative comparison of text-to-image generation on Qwen-Image, evaluated on the DrawBench benchmark. Wall-clock speedup is the end-to-end latency ratio at batch size 1 and also includes the fixed cost of the VAE, text encoder, and scheduler; FLOPs speedup is the ratio of transformer FLOPs.
Method
Acceleration
Visual Quality
Latency(s) ↓
Wall-clock Speed ↑
FLOPs(T) ↓
FLOPs Speed ↑
LPIPS ↓
SSIM ↑
Vanilla: 50 steps
80.94
1.00 ×
11561.3
1.00 ×
—
—
30% steps
30.21
2.68 ×
—
—
0.381
0.659
ToCa ( N=5 )
27.64
2.93 ×
—
—
0.454
0.590
DuCa ( N=5 )
26.23
3.09 ×
—
—
0.454
0.595
DiCache ( δ=0.8 )
26.03
3.11 ×
—
—
0.382
0.701
Table 4: Quantitative comparison of text-to-video generation on HunyuanVideo, evaluated on VBench. Wall-clock and FLOPs speedups are defined as in Table 1 ; FLOPs are reported for the vanilla pipeline and IdealCache.
Graph Construction
Text-to-Image (DrawBench)
Image Editing (GEdit-EN)
SSIM ↑
LPIPS ↓
ImageReward ↑
SC ↑
PQ ↑
OS ↑
Mean Graph ( R=1000 )
0.755
0.327
1.181
7.86
7.59
7.62
One-Shot ( R=1 )
0.752
0.334
1.168
7.81
7.58
7.58
Table 5: Out-of-distribution transferability on Qwen-Image and Qwen-Image-Edit ( B=9 ). We compare schedules derived from a single random prompt ( R=1 ) against the heavy ensemble mean graph. The performance gap is negligible, demonstrating that extremely low-cost offline planning is highly viable.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Candidate grid M
SSIM ↑
LPIPS ↓
50
0.75
0.33
100
0.78
0.27
Appendix
Table 6: Candidate-grid density ablation on Qwen-Image / DrawBench ( 1664×928 , IdealCache, B=10 ). SSIM and LPIPS are computed against the 50 -step vanilla pipeline output.
Figure 4: Additional qualitative examples on Qwen-Image (text-to-image, left) and Qwen-Image-Edit (image editing, right). IdealCache reaches 6.35× wall-clock speedup at B=7 on text-to-image generation and 5.84× at B=7 on image editing, without retraining.The optimised schedule preserves source-image content in non-edited regions while faithfully applying the requested edit.
Figure 5: Qualitative example on HunyuanVideo with IdealCache(153frames, 320x512 resolution). Sampled frames from a generated clip; temporal coherence and visual fidelity are preserved at substantially reduced compute relative to the 50 -step vanilla pipeline.
Figure 6: Additional HunyuanVideo qualitative example under IdealCache’s accelerated schedule.
Diffusion model caching reuses features, activations, or attention states across denoising steps to reduce inference cost while preserving output quality.
Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom · National Key Laboratory for Novel Software Technology, Nanjing University