IdealCache: Rethinking Cache Scheduling in Diffusion Transformers via Ideal Trajectories
Authors: Shang Xue, Li Zhu, Ping Chen
Organizations: Independent Researcher, Santa Clara, California, United States · Independent Researcher, Mountain View, California, United States · University of Electronic Science and Technology of China
Feature caching accelerates Diffusion Transformers by reusing intermediate computations across denoising timesteps, yet existing methods share a fundamental blind spot: they treat the standard reference denoising trajectory as the optimisation target, approximating its per-step states and thus inheriting its output as an implicit quality ceiling. We argue that this target is arbitrary -- what truly matters is the ideal result, not the intermediate path taken to reach it. We propose IdealCache, a training-free framework that abandons step-wise imitation of the reference trajectory. Instead, it anchors caching on a per-input ensemble of ideal trajectories constructed from candidate target results, aggregates their geometry into a single graph, and discovers optimal non-uniform schedules via a constrained shortest path on this graph. By decoupling from the reference process, IdealCache breaks the performance bottleneck of trajectory-approximation; our schedules can attain near-lossless fidelity in generation and even surpass on task-specific metrics (e.g., maintaining higher semantic consistency in image editing). Without retraining, IdealCache achieves 6.35x speedup on Qwen-Image, 5.84x on Qwen-Image-Edit, and 4.32x on HunyuanVideo, with competitive or superior quality across tasks.
Figures & tables
Figure 1: Representative IdealCache outputs across diverse prompts and tasks (Qwen-Image, Qwen-Image-Edit, HunyuanVideo1.0). Without retraining, IdealCache delivers high-fidelity generations.
Figure 2: Cache deviation as edge weight. Starting from xt∗ on the ideal trajectory, a single full pass populates the cache with residuals Δht,Δet . We then take t−k Euler updates that all reuse those residuals (red dashed), arriving at x^k . The squared distance to the ideal endpoint xk∗ is the cache-deviation edge weight w(k→t) used in our graph.
Method
Acceleration
GEdit-CN (Full)
GEdit-EN (Full)
Latency(s) ↓
Wall-clock Speed ↑
FLOPs(T) ↓
FLOPs Speed ↑
SC ↑
PQ ↑
OS ↑
SC ↑
PQ ↑
OS ↑
Vanilla: 50 steps
268.94
1.00 ×
28190.88
1.00 ×
7.68
7.51
7.41
7.82
7.54
7.54
50% steps
135.44
1.99 ×
14095.44
2.00 ×
7.70
7.53
7.44
7.77
7.52
7.47
20% steps
55.25
4.87 ×
5638.18
5.00 ×
7.65
7.42
7.35
7.73
7.46
7.44
FORA ( N=5 )
59.71
4.51 ×
5643.13
5.00 ×
7.60
7.31
7.25
7.62
7.34
7.28
DuCa ( N=6,R=90% )
67.06
4.01 ×
5897.67
4.78 ×
7.63
7.44
7.44
7.68
7.42
7.39
Table 1: Quantitative comparison of image editing on Qwen-Image-Edit, GEdit-Bench. (SC: Semantic Consistency, PQ: Perceptual Quality, OS: Overall Score.) Wall-clock speedup is the end-to-end latency ratio at batch size 1 and also includes the fixed cost of the VAE, text encoder, and scheduler; FLOPs speedup is the ratio of transformer FLOPs.
ImageReward ↑
Consistency ↑
Background Preservation ↑
50 steps (vanilla)
1.054
-
0.665
Ours ( B=25 )
1.049
0.960
0.687
Ours ( B=10 )
0.991
0.890
0.679
Table 2: Results on the FlowEdit dataset. “Consistency” measures similarity to vanilla 50-step results, while “Background Preservation” evaluates background similarity to the input image using SSIM.
Figure 3: Qualitative editing results on Qwen-Image-Edit. Our method achieves better background preservation in some cases while maintaining comparable editing quality. IdealCache with B=10 ( ∼4.2× speedup) achieves visual quality comparable to the vanilla 50-step baseline.
Method
Acceleration
Quality
Perceptual
Latency(s) ↓
Wall-clock Speed ↑
FLOPs(T) ↓
FLOPs Speed ↑
ImageReward ↑
SSIM ↑
LPIPS ↓
Vanilla: 50 steps
46.54
1.00 ×
10925.18
1.00 ×
1.2092 (+0.00%)
—
—
50% steps
23.44
1.99 ×
5462.59
2.00 ×
1.1612 ( − 3.97%)
0.798
0.249
20% steps
9.51
4.89 ×
2184.56
5.00 ×
0.8996 ( − 25.60%)
0.627
0.504
TaylorSeer ( N=3 )
22.92
2.03 ×
3929.69
2.78 ×
1.0298 ( − 14.83%)
0.504
0.628
HyCa ( N=3 )
21.95
2.12 ×
3929.69
2.78 ×
1.1914 ( − 1.47%)
0.763
0.247
Table 3: Quantitative comparison of text-to-image generation on Qwen-Image, evaluated on the DrawBench benchmark. Wall-clock speedup is the end-to-end latency ratio at batch size 1 and also includes the fixed cost of the VAE, text encoder, and scheduler; FLOPs speedup is the ratio of transformer FLOPs.
Method
Acceleration
Visual Quality
Latency(s) ↓
Wall-clock Speed ↑
FLOPs(T) ↓
FLOPs Speed ↑
LPIPS ↓
SSIM ↑
Vanilla: 50 steps
80.94
1.00 ×
11561.3
1.00 ×
—
—
30% steps
30.21
2.68 ×
—
—
0.381
0.659
ToCa ( N=5 )
27.64
2.93 ×
—
—
0.454
0.590
DuCa ( N=5 )
26.23
3.09 ×
—
—
0.454
0.595
DiCache ( δ=0.8 )
26.03
3.11 ×
—
—
0.382
0.701
Table 4: Quantitative comparison of text-to-video generation on HunyuanVideo, evaluated on VBench. Wall-clock and FLOPs speedups are defined as in Table 1 ; FLOPs are reported for the vanilla pipeline and IdealCache.
Graph Construction
Text-to-Image (DrawBench)
Image Editing (GEdit-EN)
SSIM ↑
LPIPS ↓
ImageReward ↑
SC ↑
PQ ↑
OS ↑
Mean Graph ( R=1000 )
0.755
0.327
1.181
7.86
7.59
7.62
One-Shot ( R=1 )
0.752
0.334
1.168
7.81
7.58
7.58
Table 5: Out-of-distribution transferability on Qwen-Image and Qwen-Image-Edit ( B=9 ). We compare schedules derived from a single random prompt ( R=1 ) against the heavy ensemble mean graph. The performance gap is negligible, demonstrating that extremely low-cost offline planning is highly viable.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Candidate grid M
SSIM ↑
LPIPS ↓
50
0.75
0.33
100
0.78
0.27
Appendix
Table 6: Candidate-grid density ablation on Qwen-Image / DrawBench ( 1664×928 , IdealCache, B=10 ). SSIM and LPIPS are computed against the 50 -step vanilla pipeline output.
Figure 4: Additional qualitative examples on Qwen-Image (text-to-image, left) and Qwen-Image-Edit (image editing, right). IdealCache reaches 6.35× wall-clock speedup at B=7 on text-to-image generation and 5.84× at B=7 on image editing, without retraining.The optimised schedule preserves source-image content in non-edited regions while faithfully applying the requested edit.
Figure 5: Qualitative example on HunyuanVideo with IdealCache(153frames, 320x512 resolution). Sampled frames from a generated clip; temporal coherence and visual fidelity are preserved at substantially reduced compute relative to the 50 -step vanilla pipeline.
Figure 6: Additional HunyuanVideo qualitative example under IdealCache’s accelerated schedule.
Diffusion model caching reuses features, activations, or attention states across denoising steps to reduce inference cost while preserving output quality.
Diffusion Transformers (DiTs) have emerged as the dominant architecture for high-quality image and video generation, yet their iterative denoising process incurs substantial computational cost during inference. Existing caching methods accelerate DiTs by reusing intermediate computations across timesteps, but they share a common limitation: treating the denoising process as uniform across time,depth, and feature dimensions. In this work, we identify three orthogonal axes of non-uniformity in DiT denoising: (1) temporal -- sensitivity to caching errors varies dramatically across the denoising trajectory; (2) depth -- consecutive caching decisions lead to cascading approximation errors; and (3) feature -- different components of the hidden state exhibit heterogeneous temporal dynamics. Based on these observations, we propose SpectralCache, a unified caching framework comprising Timestep-Aware Dynamic Scheduling (TADS), Cumulative Error Budgets (CEB), and Frequency-Decomposed Caching (FDC). On FLUX.1-schnell at 512x512 resolution, SpectralCache achieves 2.46x speedup with LPIPS 0.217 and SSIM 0.727, outperforming TeaCache (2.12x, LPIPS 0.215, SSIM 0.734) by 16% in speed while maintaining comparable quality (LPIPS difference < 1%). Our approach is training-free, plug-and-play, and compatible with existing DiT architectures.
We propose OTCache, a training-free framework for accelerating diffusion sampling via caching schedule prediction. Existing graph-based caching methods reduce redundant computation by optimizing shortest-path objectives, but rely on an additive independence assumption, which often breaks down in the low NFE regime. To address this issue, OTCache models caching schedules across inference budgets as a smooth evolution in policy space, inspired by Optimal Transport (OT). The framework consists of three stages: (1) obtaining a high-fidelity reference schedule using a graph-based caching method under a conservative budget; (2) performing a lightweight anchor search under an extreme low-budget setting via Optuna optimization with an end-to-end perceptual objective; and (3) predicting schedules for target budgets via quantile interpolation between the reference and anchor policies using continuous warping representations. Experiments on FLUX.1 [dev], Qwen-Image, and HunyuanVideo show that OTCache achieves 4.5x, 4.7x, and 3.66x acceleration, respectively, while consistently improving generation fidelity over state-of-the-art caching baselines. This work provides a new perspective on accelerating diffusion models through Optimal-Transport-inspired schedule modeling. Code:https://github.com/UnicomAI/OTCache
Huanlin Gao, Fang Zhao, Qiang Hui +8
Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom · National Key Laboratory for Novel Software Technology, Nanjing University
Diffusion transformers have become the most powerful models for visual generation, but still suffer from massive computation costs. To solve this problem, feature caching has been proposed to cache the features of diffusion models in the previous computation steps and then reuse them in the following caching steps, which brings significant acceleration but also degradation in generation quality. To address this problem, this paper proposes Z-cache as a feature caching method that can maintain high-quality generation through self-reflection. Concretely, we observe that the error from feature caching tends to be sharply reduced after each full computation. Based on this observation, Z-Cache is designed to first predict the features in the future caching steps and then perform a full computation. After that, Z-Cache returns to the caching steps and re-predicts them based on the previous and the current computation steps, which brings correction in features. Experiments demonstrate that with Z-Cache, diffusion transformers achieve comparable generation quality to the original model but with faster inference speed, for instance, 6.22X acceleration on FLUX-dev for text-to-image generation. Our code is released on GitHub.
Zegang Cheng, Zhikai Wang, Jiacheng Liu +8
Shanghai Jiao Tong University · Southern University of Science and Technology · Alibaba Cloud +1