SpecKCast: Spectral Kalman Forecasting for Diffusion Acceleration

Javad Rajabi; Hakki Can Karaimer; David B. Lindell; Babak Taati; Michael S. Brown

cs.CVOct 19, 2026

SpecKCast: Spectral Kalman Forecasting for Diffusion Acceleration

Authors: Javad Rajabi, Hakki Can Karaimer, David B. Lindell, Babak Taati, Michael S. Brown

Organizations: University of Toronto · Vector Institute · AI Center–Toronto, Samsung Electronics

Abstract

Diffusion Transformers (DiTs) achieve high-fidelity image and video generation, but they suffer from substantial inference costs due to repeated transformer evaluations across many denoising steps. Feature caching reduces this cost by reusing features across denoising steps. Recent forecasting methods extend this idea by extrapolating features to skipped steps instead of reusing them. However, existing forecasting methods extrapolate the features they predict uniformly, even though the spatial-frequency components of these features change most at different stages of denoising. They also overlook that denoising progresses nonuniformly, so equal step intervals can correspond to very different amounts of change. These mismatches amplify forecast errors as more steps are skipped, limiting the achievable acceleration. We propose SpecKCast, a training-free method that forecasts the model output in the frequency domain. SpecKCast applies a 2D discrete cosine transform (DCT) to the output and tracks each coefficient with a lightweight Kalman filter along a band-specific time coordinate derived from the noise schedule. The filter tracks each coefficient's value and rate of change, and the transformer is evaluated only when the filter's forecast uncertainty exceeds a threshold. Experiments on FLUX.1-dev, Qwen-Image, Wan2.1-14B, and HunyuanVideo show that SpecKCast achieves a better trade-off between acceleration and fidelity than existing caching and forecasting methods, reaching up to 6x speedup with less than 0.05 GiB of additional memory.

Figures & tables

Figure 1: Qualitative comparison on FLUX at different acceleration ratios.

Figure 2: Spatial-frequency bands of the denoising velocity follow different trajectories. PCA trajectories of the 50 velocity predictions from a single FLUX.1-dev generation, projected onto the first two principal components. Larger gaps between consecutive points indicate larger changes. (a) Sampled entries of the full velocity. (b–d) Radial DCT bands:

B_{0}

moves most in the first step,

B_{3}

changes over a broader interval, and

B_{6}

moves most after step 40. Each panel has its own PCA basis and normalization. Additional views, prompts, and Qwen-Image results are in Appendix G .

		FLUX.1-dev (1024 $\times$ 1024)								Qwen-Image (1328 $\times$ 1328)
		Acceleration		Mem. $\Delta$ (GiB) $\downarrow$	Quality			Image Reward $\uparrow$	CLIP $\uparrow$	Acceleration		Mem. $\Delta$ (GiB) $\downarrow$	Quality			Image Reward $\uparrow$	CLIP $\uparrow$
NFE	Method	Latency(s) $\downarrow$	Speedup $\uparrow$	Mem. $\Delta$ (GiB) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Image Reward $\uparrow$	CLIP $\uparrow$	Latency(s) $\downarrow$	Speedup $\uparrow$	Mem. $\Delta$ (GiB) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Image Reward $\uparrow$	CLIP $\uparrow$
$50$	Reference	11.68	1.00	–	–	–	–	0.98	27.20	52.22	1.00	–	–	–	–	1.24	29.14
$\approx 14$	Base	3.35	3.48	0.00	15.30	0.633	0.423	0.98	27.51	14.83	3.52	0.00	15.18	0.640	0.393	1.08	29.19
	FORA	3.98	2.93	2.01	15.03	0.628	0.433	0.96	27.48	16.11	3.24	9.46	14.03	0.571	0.456	1.02	28.97
	TeaCache	3.55	3.29	0.07	16.86	0.684	0.355	1.01	27.57	15.68	3.33	0.20	19.42	0.781	0.229	1.19	29.36

Table 1: Quantitative comparison in text-to-image generation on FLUX.1-dev and Qwen-Image.

Figure 3: Qualitative comparison on text-to-image generation using FLUX.1-dev.

		Wan2.1-14B							HunyuanVideo
		Acceleration		Mem. $\Delta$ (GiB) $\downarrow$	Quality			VBench Quality $\uparrow$	Acceleration		Mem. $\Delta$ (GiB) $\downarrow$	Quality			VBench Quality $\uparrow$
NFE	Method	Latency(s) $\downarrow$	Speedup $\uparrow$	Mem. $\Delta$ (GiB) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VBench Quality $\uparrow$	Latency(s) $\downarrow$	Speedup $\uparrow$	Mem. $\Delta$ (GiB) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VBench Quality $\uparrow$
50	Reference	370	1.00	–	–	–	–	81.11	183	1.00	–	–	–	–	80.26
$\approx 14$	Base	116	3.20	0.00	13.95	0.471	0.518	79.81	58.3	3.14	0.00	17.33	0.641	0.417	78.38
	FORA	108.3	3.42	0.62	21.62	0.753	0.192	80.63	56.8	3.22	0.19	23.46	0.775	0.216	78.90
	TeaCache	105.2	3.52	0.94	22.13	0.772	0.191	80.24	61.2	2.99	0.37	20.56	0.709	0.279	79.35

Table 2: Quantitative comparison in text-to-video generation on Wan2.1-14B and HunyuanVideo.

Figure 4: Qualitative comparison on text-to-video generation using HunyuanVideo.

Forecaster	Scheduler	Time variable	Frequency decomposition	Latency(s) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Image Reward $\uparrow$	CLIP $\uparrow$
Taylor	fixed	warped-logSNR	$\checkmark$	3.46	21.90	0.811	0.171	0.978	27.24
Chebyshev	fixed	warped-logSNR	$\checkmark$	3.97	21.10	0.761	0.252	0.975	27.66
Kalman	fixed	warped-logSNR	$\checkmark$	3.43	22.09	0.810	0.178	0.973	27.27
Kalman	adaptive	raw-logSNR	$\checkmark$	3.52	21.03	0.775	0.210	0.977	27.24
Kalman	adaptive	uniform step index	$\checkmark$	3.51	19.54	0.740	0.256	0.969	27.28
Kalman	adaptive	warped-logSNR	$\times$	3.48	21.19	0.785	0.201	0.966	27.13

Table 3: Ablation of SpecKCast components on FLUX.1-dev at 14 NFE.

Panel of Figure 5 — Figure 5: Qualitative comparison on text-to-image generation using Qwen-Image.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Additional Qwen-Image comparisons. Rows show strawberries in a basket, an astronaut riding a horse, and a fox in snow. The leftmost column is the 50-NFE reference; all accelerated methods use 8 NFE in the first two rows and 10 NFE in the last, with the measured speedup shown below each image. SpecKCast more closely preserves the reference colors and object structure, while the baselines exhibit color shifts, altered content, or degraded texture.

Figure 8: HunyuanVideo: “a bottle and a chair.” Frames 0, 40, and 80 for each method show differences in facial detail, pose, and bottle placement. SpecKCast uses 10 NFE ( $4.27\times$ ), compared with 10–11 NFE ( $3.85$ – $4.24\times$ ) for the baselines. It avoids the severe facial blur visible in FORA and TaylorSeer and keeps the bottle in hand, although the subject’s appearance still differs from the reference.

Figure 9: HunyuanVideo: “a bicycle leaning against a tree.” TaylorSeer (16 NFE, $2.82\times$ ) and Spectrum (14 NFE, $3.13\times$ ) introduce an additional wheel-like structure between the bicycle’s two wheels. SpecKCast (13 NFE, $3.41\times$ ) better preserves the main bicycle geometry across the displayed frames, although the front basket is missing.

Panel of Figure 10 — Figure 10: Wan2.1-14B: a panda drinking coffee with a zoom-in. Frames 0, 40, and 80 show subject appearance and scene layout. All accelerated methods use 8 NFE except SeaCache (7 NFE). SpecKCast achieves $5.61\times$ speedup while avoiding the strong color and boundary artifacts of HiCache and TaylorSeer. TeaCache ( $5.64\times$ ) and SeaCache ( $6.05\times$ ) are faster, but TeaCache softens the panda’s detail and SeaCache changes the scene more substantially.

Figure 12: Joint FLUX.1-dev trajectories across 12 prompts. Columns show the sampled full velocity and DCT bands

B_{0}

B_{3}

, and

B_{6}

. Both rows use the same normalized PCA scores, computed jointly over 12 prompts with 50 measured velocities per prompt, guidance scale 3.5, and full transformer computation. Unlike Figure 2 , which shows a single generation, the PCA here is fitted to all 12 generations together. Top: PC1–PC2 projections with equal axis scaling; labels identify selected steps. Bottom: PC1 (solid blue) and PC2 (dashed gray) against step index. Circles mark measurements, dark to light blue indicates denoising progress, and stars mark the final state (PC1 in the bottom row). Each column has its own PCA basis and scale, constructed as described in Section G.2 .

Figure 13: Spatial-frequency trajectories of Qwen-Image across 12 prompts. Columns show the sampled full velocity and DCT bands

B_{0}

B_{3}

, and

B_{6}

. All rows use the same normalized PCA scores, computed jointly over 12 prompts with 50 velocities passed to the sampler per prompt, guidance scale 4, and full transformer computation. Top: Denoising step increases downward. Middle: The same PC1–PC2 pairs with equal axis scaling; labels identify selected steps. Bottom: PC1 (solid blue) and PC2 (dashed gray) against step index. The low band changes mainly at the start, the middle band evolves over a broader interval, and the high band changes sharply near the end after a small early transient. Bases and scales are fitted independently for each column; see Section G.2 for the construction.

Figure 14: Individual FLUX.1-dev trajectories: prompts 000 and 001. Top two rows: “a cat holding a red apple” (seed 0). Bottom two rows: “a photograph of an astronaut riding a horse on the moon” (seed 1). For each prompt, the same 50 states are shown in 3D with the denoising step as the vertical axis and in the PC1–PC2 plane. Columns show full velocity,

B_{0}

B_{3}

, and

B_{6}

; dark to light blue indicates denoising progress and stars mark the final state. Each panel uses its own PCA basis and scaling, so positions and sizes cannot be compared across panels or prompts.

Figure 15: Individual FLUX.1-dev trajectories: prompts 002 and 003. Top two rows: “a bowl of ramen with chopsticks, steam rising, cinematic lighting” (seed 2). Bottom two rows: “a red panda reading a book in a cozy library” (seed 3). For each prompt, the same 50 states are shown in 3D with the denoising step as the vertical axis and in the PC1–PC2 plane. Columns show full velocity,

B_{0}

B_{3}

, and

B_{6}

; dark to light blue indicates denoising progress and stars mark the final state. Each panel uses its own PCA basis and scaling, so positions and sizes cannot be compared across panels or prompts.

Figure 16: Individual FLUX.1-dev trajectories: prompts 004 and 005. Top two rows: “a futuristic city skyline at sunset with flying cars” (seed 4). Bottom two rows: “a close-up portrait of an old fisherman with a weathered face” (seed 5). For each prompt, the same 50 states are shown in 3D with the denoising step as the vertical axis and in the PC1–PC2 plane. Columns show full velocity,

B_{0}

B_{3}

, and

B_{6}

; dark to light blue indicates denoising progress and stars mark the final state. Each panel uses its own PCA basis and scaling, so positions and sizes cannot be compared across panels or prompts.

Figure 17: Individual Qwen-Image trajectories: prompts 000 and 001. Top two rows: “a cat holding a red apple” (seed 0). Bottom two rows: “a photograph of an astronaut riding a horse on the moon” (seed 1). For each prompt, the same 50 states are shown in 3D with the denoising step as the vertical axis and in the PC1–PC2 plane. Columns show full velocity,

B_{0}

B_{3}

, and

B_{6}

; dark to light blue indicates denoising progress and stars mark the final state. Each panel uses its own PCA basis and scaling, so positions and sizes cannot be compared across panels or prompts.

Figure 18: Individual Qwen-Image trajectories: prompts 002 and 003. Top two rows: “a bowl of ramen with chopsticks, steam rising, cinematic lighting” (seed 2). Bottom two rows: “a red panda reading a book in a cozy library” (seed 3). For each prompt, the same 50 states are shown in 3D with the denoising step as the vertical axis and in the PC1–PC2 plane. Columns show full velocity,

B_{0}

B_{3}

, and

B_{6}

; dark to light blue indicates denoising progress and stars mark the final state. Each panel uses its own PCA basis and scaling, so positions and sizes cannot be compared across panels or prompts.

Figure 19: Individual Qwen-Image trajectories: prompts 004 and 005. Top two rows: “a futuristic city skyline at sunset with flying cars” (seed 4). Bottom two rows: “a close-up portrait of an old fisherman with a weathered face” (seed 5). For each prompt, the same 50 states are shown in 3D with the denoising step as the vertical axis and in the PC1–PC2 plane. Columns show full velocity,

B_{0}

B_{3}

, and

B_{6}

; dark to light blue indicates denoising progress and stars mark the final state. Each panel uses its own PCA basis and scaling, so positions and sizes cannot be compared across panels or prompts.

Diffusion Model Inference Acceleration

Diffusion model inference acceleration reduces the time and compute needed to generate or reconstruct outputs while preserving quality.

See all 506 papers

Tikhonov-Stabilized Bezier Representation Forecasting for Training-free Diffusion Acceleration
Oct 12, 2026Lei Zhu, Mujie Lin, Ruochong Zheng +5Diffusion Model Inference Acceleration Diffusion Model Caching
School of Electronic and Computer Engineering, Peking University · School of Informatics, Xiamen University · Department of Automation and BNRist, Tsinghua University
Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration
Mar 2, 2026Jiaqi Han, Juntong Shi, Puheng Li +3Video Diffusion Models Diffusion-Based Image Generation
Stanford University · ByteDance
Predict to Skip: Linear Multistep Feature Forecasting for Efficient Diffusion Transformers
Feb 20, 2026Hanshuai Cui, Zhiqing Tang, Qianli Ma +3Diffusion Model Acceleration Diffusion Sampling
School of Artificial Intelligence, Beijing Normal University, Beijing, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai, China · Shenzhen University of Advanced Technology, Shenzhen, China

Oct 12, 2026J/K move · Enter open · S save

Tikhonov-Stabilized Bezier Representation Forecasting for Training-free Diffusion Acceleration

Lei Zhu, Mujie Lin, Ruochong Zheng, Guangyi Wang+4

School of Electronic and Computer Engineering, Peking University · School of Informatics, Xiamen University · Department of Automation and BNRist, Tsinghua University

Diffusion Model Inference Acceleration Diffusion Model Caching Diffusion Models Diffusion-Based Image Generation Diffusion Transformer Video Diffusion Models

Open paper AskPDF arXiv

Diffusion models, particularly Diffusion Transformers, achieve strong image and video generation quality but remain expensive at inference time due to repeated denoiser evaluations. Feature caching offers a deployment-friendly acceleration strategy by reusing or predicting intermediate representations without retraining the generator or modifying the sampler. However, existing cache-then-forecast methods often rely on Taylor-style finite-difference extrapolation, which can become unstable over longer cache intervals, and typically predict local module outputs whose errors may accumulate through subsequent denoiser blocks. We propose BeziCast, a training-free diffusion acceleration framework that forecasts output-proximal denoising representations using low-order B'ezier trajectories. BeziCast estimates B'ezier control points via Tikhonov-stabilized fitting, providing a smooth, capacity-controlled temporal parameterization that avoids explicit high-order derivative estimation and decouples trajectory capacity from cache-query density. To moderate aggressive extrapolation, we further derive equivalent forecasting weights and introduce a convex-hull-inspired guardrail that detects high-risk predictions and softly pulls them toward a conservative simplex-projected estimate. Extensive evaluations across advanced image and video diffusion models demonstrate the effectiveness of BeziCast. In particular, BeziCast accelerates FLUX.1 by up to (4.79\times) and HunyuanVideo by up to (4.11\times), while preserving substantially better generation quality than competing acceleration baselines.

Abstract

Diffusion Model Inference Acceleration

Explore similar work

Tikhonov-Stabilized Bezier Representation Forecasting for Training-free Diffusion Acceleration

Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration

Predict to Skip: Linear Multistep Feature Forecasting for Efficient Diffusion Transformers