Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache

Chengjie Lu; Tianchi Deng; Zhengqi He; Zhijian Gao; Huisi Wu; Xueliang Li

cs.CVAug 24, 2026

Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache

Authors: Chengjie Lu, Tianchi Deng, Zhengqi He, Zhijian Gao, Huisi Wu, Xueliang Li

Organizations: College of Electronics and Information Engineering, Shenzhen University · School of Artificial Intelligence, Shenzhen University · College of Computer Science and Software Engineering, Shenzhen University

Abstract

Diffusion Transformers achieve high-fidelity image and video generation, but their iterative sampling remains expensive, for each denoising step requires large matrix operations. Existing cache-based acceleration reduces redundant computation yet increases the VRAM footprint by storing intermediate states, which can directly constrain inference batch size. In this work, we propose a training-free acceleration method that performs stepwise forecasting for DiT sampling using a Barycentric Extrapolator. By leveraging barycentric extrapolation, our predictor is numerically stable and alleviates oscillatory artifacts analogous to the Runge phenomenon during forward forecasting. Across extensive experiments on both image and video generation, our approach provides a favorable trade-off between memory usage and perceptual quality, while delivering up to 3.30x end-to-end sampling speedup compared with baseline DiT inference.

Figures & tables

Figure 1: Caching targets for different strategies. Reusing features across diffusion steps is stepwise caching, while reusing features within a step across layers/modules is blockwise caching. For

N

diffusion steps, blockwise caching may require storing up to

N\times Q

intermediate tensors compared to stepwise caching, which can significantly constrain batch size and parallelism.

Figure 2: Overview of BaryCache. At each diffusion step, the model either performs a full forward pass or a cache forward pass, where the Barycentric Extrapolator yields weights and predicts the current stepwise feature from a cache context of recently computed full-step outputs. Weight post-processing stabilizes the barycentric combination for extrapolation. BaryCache balances memory usage and fidelity via stepwise feature prediction from prior steps under equal-spaced schedules.

Figure 3: Mean square error comparison in the preliminary evaluation. (a) shows that stepwise prediction performs better than blockwise variants. (b) shows that the stepwise version of BaryCache surpasses TaylorSeer in later steps, despite a larger error at step 2, indicating that BaryCache preserves quality while requiring less memory.

Method	Acceleration & Runtime				Image $\uparrow$	Clip $\uparrow$
PixArt- $\mathbf{\Sigma}$ [ 3 ]	FLOPs (T) $\downarrow$	Latency (s) $\downarrow$	Speedup $\uparrow$	VRAM (GB) $\downarrow$	Reward	Score
[Vanilla]: DDIM-50 steps	450.52	4.974	1.00 $\times$	3.363	0.7258	30.89
[Vanilla]: DDIM-15 steps	135.16	1.484	3.35 $\times$	3.363	0.6577	31.04
$\Delta$ -DiT ( $\mathcal{N}=4$ ) [ 5 ]	126.15	1.651	3.01 $\times$	3.651	0.6246	30.96
FORA ( $\mathcal{N}=4$ ) [ 37 ] †	125.68	3.939	1.26 $\times$	19.255	0.5024	30.79
ToCa ( $\mathcal{N}=4$ ) [ 49 ] †	189.08	4.957	1.00 $\times$	27.453	0.5679	30.83

Table 1: Quantitative comparison in text-to-image generation.

Method	Acceleration & Runtime			Generation Quality
HunyuanVideo [ 17 ]	Latency (s) $\downarrow$	Speedup $\uparrow$	VRAM (GB) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
[Vanilla]: Euler-50 steps	231.240	1.000 $\times$	31.171	-	-	-
ToCa ( $\mathcal{N}=5$ ) [ 49 ] †	87.580	2.640 $\times$	46.521	15.8272	0.5069	0.4437
DuCa ( $\mathcal{N}=5$ ) [ 50 ] †	82.480	2.804 $\times$	46.515	15.8641	0.5122	0.4440
TeaCache ( $l=0.4$ ) [ 21 ]	84.920	2.723 $\times$	31.525	17.0962	0.5984	0.3775
TaylorSeer ( $\mathcal{N}=5$ ) [ 22 ] †	87.940	2.630 $\times$	60.771	15.9317	0.5214	0.4269

Table 2: Quantitative comparison in text-to-video generation.

Method	Acceleration & Runtime				Generation Quality
DiT-XL/2 [ 32 ]	FLOPs (T) $\downarrow$	Latency (s) $\downarrow$	Speedup $\uparrow$	VRAM (GB) $\downarrow$	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$
[Vanilla]: DDIM-50 steps	23.74	0.349	1.00 $\times$	6.037	2.27	4.34	239.61
[Vanilla]: DDIM-25 steps	11.87	0.234	1.49 $\times$	6.037	2.91	4.54	229.97
$\Delta$ -DiT ( $\mathcal{N}=3$ ) [ 5 ]	8.07	0.182	1.91 $\times$	6.037	3.58	4.66	217.19
FORA ( $\mathcal{N}=5$ ) [ 37 ] †	5.24	0.190	1.84 $\times$	6.467	5.81	9.74	200.64
ToCa ( $\mathcal{N}=5$ ) [ 49 ] †	7.72	0.303	1.15 $\times$	6.569	4.41	6.21	213.46

Table 3: Quantitative comparison in class-to-image generation for DiT-XL/2.

Figure 4: Qualitative comparison on PixArt- $\Sigma$ .

Figure 5: Qualitative comparison on HunyuanVideo.

Figure 6: Qualitative ablation of $\lambda$ on HunyuanVideo.

Ablation		Image Reward $\uparrow$	Clip Score $\downarrow$
$\mathcal{N}=4$	$\mathit{H}=2$	0.7220	30.92
	$\mathit{H}=3$	0.6936	30.99
	$\mathit{H}=4$	0.6914	30.94
$\mathcal{N}=5$	$\mathit{H}=2$	0.6812	30.94
	$\mathit{H}=3$	0.6557	30.85
	$\mathit{H}=4$	0.6510	30.93

Table 4: Ablation study on cache history length.

Mode	Schedule	Setting	IR $\uparrow$	CS $\uparrow$
Stepwise $\lambda=0$	Eq-spaced	$N=4$	0.7220	30.92
	Adaptive	$\ell=0.13$	0.6594	30.83
	Eq-spaced	$N=5$	0.6812	30.94
	Adaptive	$\ell=0.16$	0.6034	30.84
	Eq-spaced	$N=6$	0.6654	31.10
	Adaptive	$\ell=0.2$	0.5881	30.97

Table 5: Ablations on different modes and schedules.

Ablation	Value	Image Reward $\uparrow$	Clip Score $\uparrow$
$\lambda$ ( stepwise , $\mathcal{N}=4$ , $H=2$ )	0.0	0.7220	30.92
	0.2	0.7002	30.95
	0.4	0.6857	30.95
	0.6	0.6887	31.00
	0.8	0.6566	31.00
	1.0	0.6200	31.01

Table 6: Ablation study on

\lambda

and Br .

Figure 7: Qualitative ablation of the blockwise ratio Br .

Figure 8: Qualitative ablation of $\lambda$ on PixArt- $\Sigma$ .

Figure 9: Qualitative ablation of $\delta$ on PixArt- $\Sigma$ .

Explore similar work

ChebBooster: A Training-Free Approach for Efficient Diffusion Transformer Inference via Chebyshev-Inspired Extrapolation
Aug 24, 2026Chengjie Lu, Tianchi Deng, Zhengqi He +2Diffusion Model Inference Acceleration Diffusion-Based Image Generation
College of Electronics and Information Engineering, Shenzhen University · School of Artificial Intelligence, Shenzhen University
Predict to Skip: Linear Multistep Feature Forecasting for Efficient Diffusion Transformers
Feb 20, 2026Hanshuai Cui, Zhiqing Tang, Qianli Ma +3Diffusion Model Acceleration Diffusion Sampling
School of Artificial Intelligence, Beijing Normal University, Beijing, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai, China · Shenzhen University of Advanced Technology, Shenzhen, China
Frequency-Aware Error-Bounded Caching for Accelerating Diffusion Transformers
Mar 5, 2026Guandong LiDiffusion Model Caching Diffusion Model Inference Acceleration
iFLYTEK

Aug 24, 2026J/K move · Enter open · S save

ChebBooster: A Training-Free Approach for Efficient Diffusion Transformer Inference via Chebyshev-Inspired Extrapolation

Chengjie Lu, Tianchi Deng, Zhengqi He, Chengwen Luo+1

College of Electronics and Information Engineering, Shenzhen University · School of Artificial Intelligence, Shenzhen University

Diffusion Model Inference Acceleration Diffusion-Based Image Generation Diffusion Transformer

Open paper AskPDF arXiv

Diffusion Transformers (DiTs) have shown strong performance in high-fidelity image generation, but their sampling process remains computationally intensive due to full model execution at every timestep. While cache-based acceleration has been explored to mitigate inference cost, naive reuse schemes suffer from low accuracy over long intervals, and Taylor-series-based extrapolation methods often face instability caused by Runge oscillations. In this paper, we propose ChebBooster, a training-free extrapolation framework based on Chebyshev polynomial theory that achieves stable and efficient acceleration for DiTs. Specifically, we adopt the Barycentric formulation to evaluate Chebyshev approximants with high numerical stability and minimal overhead, and further decouple the extrapolation into an offline weight precomputation phase and a lightweight online application stage. Extensive experiments across three representative DiT-based models, including DiT-XL/2, PixArt-

Σ

, and FLUX.1-dev, demonstrate that ChebBooster achieves consistent improvements in visual quality and inference efficiency, reaching up to

3.68\times

latency speedup and

5.12\times

FLOPs reduction, outperforming existing training-free baselines under diverse generation tasks and resolutions.

Abstract

Explore similar work

ChebBooster: A Training-Free Approach for Efficient Diffusion Transformer Inference via Chebyshev-Inspired Extrapolation

Predict to Skip: Linear Multistep Feature Forecasting for Efficient Diffusion Transformers

Frequency-Aware Error-Bounded Caching for Accelerating Diffusion Transformers