Organizations: Shanghai Jiao Tong University · University of Electronic Science and Technology of China · Xiamen University · Terminal Intelligent Computing Division, Alibaba Cloud · Fudan University
Diffusion Transformers have become the dominant paradigm in generative AI, but their high computational costs severely hinder real-time applications. Prediction-based feature caching is widely used to accelerate diffusion transformers; however, as the number of steps increases, the deviation between its predictions and the reference full-compute trajectory gradually grows. An intuitive idea is to use an online regression model to dynamically correct this deviation, but it faces the issue of label data being unavailable during the acceleration process. This paper presents a statistical observation that the residuals between the features of full computation steps using caching methods and reference full-compute trajectory locally exhibit a zero-mean Gaussian distribution. By treating the features of full computation steps as noisy observations of reference features, the data acquisition problem is resolved. Based on this observation, a plug-and-play GP-Refiner correction framework is proposed. This method utilizes Gaussian Process Regression for correction and, leveraging the properties of GPR, introduces an uncertainty-adaptive computation strategy that triggers necessary full-computation calibration by monitoring the posterior variance in real time. Experiments demonstrate significant improvements across different models when combined with various state-of-the-art methods. Integrating the proposed framework with TaylorSeer reduces the computational load by 19.3% while improving PSNR by 0.9 dB and reducing LPIPS from 0.46 to 0.29. Code is available in https://github.com/Aredstone/GP-Refiner.
Figures & tables
Figure 1: Histogram of the error caused by the feature caching. Across different images, the residuals are centered near zero and closely match a Gaussian fit, consistent with an approximately zero-mean Gaussian noise model.
Figure 2: Q–Q plots of randomly projected feature residuals. The empirical quantiles closely follow a standard normal line ( R2≈0.97 – 0.99 ), supporting an approximately zero-mean Gaussian residual model.
Figure 3: Overall workflow of GP-Refiner. (a) Original cache pipeline: features are computed at selected timesteps and cached, while skipped steps are filled by a cache predictor. (b) Cache + GP-Refiner: we collect an online observation set from full-compute steps, use GPR to rectify the predictor output, and trigger recomputation when the posterior variance exceeds a threshold ( σt2>l ).
Method
Latency(s) ↓
Speed ↑
FLOPs(T) ↓
Speed ↑
PSNR ↑
SSIM ↑
LPIPS ↓
[dev]: 50 steps
11.55
1.00 ×
3719.50
1.00 ×
-
-
-
60% steps
7.12
1.62 ×
2231.70
1.67 ×
30.310
0.7819
0.2461
50% steps
5.93
1.95 ×
1859.75
2.00 ×
29.576
0.7337
0.3106
40% steps
4.82
2.40 ×
1487.80
2.50 ×
29.122
0.6971
0.3619
34% steps
4.15
2.78 ×
1264.63
2.94 ×
28.881
0.6776
0.3913
FORA (N=5)
3.31
3.49 ×
893.54
4.16 ×
28.432
0.6029
0.4918
Table 1: Quantitative comparison in text-to-image generation for FLUX.
Method
Latency(s) ↓
Speed ↑
FLOPs(T) ↓
Speed ↑
PSNR ↑
SSIM ↑
LPIPS ↓
50 steps
36.68
1.00 ×
12917.56
1.00 ×
-
-
-
20% steps
7.82
4.69 ×
2583.51
5.00 ×
28.59
0.61
0.52
FORA ( N=4 )
10.76
3.41 ×
3359.99
3.84 ×
28.66
0.59
0.51
FORA ( N=6 )
8.49
4.32 ×
2326.74
5.55 ×
28.48
0.55
0.59
ToCa ( N=8 )
16.82
2.18 ×
2991.34
4.32 ×
28.93
0.63
0.44
ToCa ( N=12 )
14.56
2.52 ×
2406.20
5.37 ×
28.69
0.57
0.53
Table 2: Quantitative comparison in text-to-image gen. for Qwen-Image.
Figure 4: Visual comparison of generated images before and after combining different baseline cache methods with GP-Refiner.
Method
FLOPs(T) ↓
Speed ↑
PSNR ↑
SSIM ↑
LPIPS ↓
[dev]: 50 steps
3719.50
1.00 ×
-
-
-
TaylorSeer (O=2,N=8)
596.07
6.24×
28.513 (+0.00)
0.5986 (+0.00)
0.4854 (+0.00)
TaylorSeer + ours (O=2,N=8)
596.08
6.24×
29.129 (+0.62)
0.6426 (+0.04)
0.4469 (-0.04)
TaylorSeer (O=2,N=7)
670.44
5.55×
28.671 (+0.00)
0.6237 (+0.00)
0.4542 (+0.00)
TaylorSeer + ours (O=2,N=7)
670.45
5.55×
29.266 (+0.60)
0.6622 (+0.04)
0.4151 (-0.04)
TaylorSeer (O=2,N=6)
744.80
4.99×
28.943 (+0.00)
0.6558 (+0.00)
0.4020 (+0.00)
Table 3: Ablation on GP Rectification and Dynamic Interval Scheduling.
Figure 5: Qualitative results of GP-Refiner (TaylorSeer + ours) on FLUX at NFE-8 and NFE-9.
Training-free feature reuse accelerates diffusion transformer inference by replacing full denoising computation at selected steps with cached approximations. Existing methods mainly improve the reconstruction of skipped features, but we find that once a stable first-order reconstructor is used, increasing reconstruction complexity brings limited gains, whereas the placement of exact-computation steps has a much larger effect on fidelity. More importantly, absolute shallow-probe variation is jointly affected by input-trajectory displacement and network response strength, and therefore does not directly reflect cache risk. We address this confounding by normalizing probe-output variation with the corresponding input displacement, yielding a directional gain per unit trajectory movement. Based on this signal, we propose CacheScope, which identifies high-risk reuse steps online and selectively falls back to full denoising computation while leaving the underlying cache reconstructor unchanged. CacheScope requires no additional training, gradients, auxiliary networks, or prompt-specific offline schedule optimization. Across FLUX.1-dev, Wan2.1-T2V-1.3B, and Open-Sora 1.2, CacheScope achieves up to 3.57x acceleration and reduces median LPIPS by up to 70.1% over the strongest probe-based baseline.
Diffusion Transformers (DiTs) have driven substantial progress in image and video generation but suffer from prohibitive computational costs. Feature caching accelerates inference by reusing intermediate representations. Existing methods rely on historical features for implementation simplicity, yet suffer from severe error accumulation at high acceleration ratios. To address this limitation, we investigate the nature of the requisite feature correction. We demonstrate that the optimal calibration update is characterized by a shared low-rank subspace across diverse prompts. Guided by this structural insight, we propose LearniBridge, a learnable calibration mechanism for feature caching that bridges multiple timesteps through lightweight LoRA updates. This mechanism enables effective calibration requiring only 3-5 training samples. Extensive experiments on image and video generation show that LearniBridge achieves up to 5.87×, 5.75×, and 4.10× acceleration on FLUX, HunyuanVideo, and WAN2.1, respectively. On WAN2.1, it improves VBench by 1.28% over the previous SOTA at 4.10× acceleration. Our code is available at https://github.com/Iiiiiiirene/LearniBridge.
Xuyue Huang, Zhe Chen, Wang Shen +1
Shenzhen Ubiquitous Data Enabling Key Laboratory, Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Central Media Technology Institute, Huawei Technologies Co., Ltd., Shenzhen, China
Diffusion Transformers require repeated denoiser evaluations during iterative sampling, making inference computationally expensive. Cache-based acceleration reduces this cost by reusing intermediate representations across denoising steps, but can introduce representation deviations and degrade generation quality. In this paper, we analyze these deviations and show that effective calibration should consider both the direct mismatch caused by reuse and the subsequent trajectory shift induced by earlier corrections. To address this challenge, we propose Trajectory-Consistent Calibration (TCC), a training-free method that calibrates cached representations toward their full-computation counterparts. Specifically, rather than estimating all calibration priors from a single uncorrected cache trajectory, TCC uses an offline iterative procedure so that each prior accounts for the trajectory shift induced by preceding calibrations. Experiments on PixArt-alpha and DiT-XL/2 show that TCC consistently improves FID across representative cache-based acceleration methods while preserving their underlying reuse policies. Notably, in a representative PixArt-alpha cache-acceleration setting based on FORA, TCC reduces FID from 29.83 to 27.35, slightly surpassing the full-computation baseline.
Mingyu Liang, Dingkun Xu, Jingwei Xu
Laboratory for Novel Software Technology, Nanjing University, China