cs.CVOct 2, 2026

SpectralCache: Accelerating Diffusion-Based World Models via Spectral Feature Caching

Authors: Zhendong Mi, Pu Zhao, Ziyu Hu, Xiaodong Yu, Yanzhi Wang, Grace Li Zhang, Shaoyi Huang

Organizations: Stevens Institute of Technology · Northeastern University · Technische Universität Darmstadt

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Diffusion-based world models enable high-quality interactive environment generation but suffer from substantial inference overhead due to repeated Transformer evaluations during denoising. Existing caching methods mainly exploit temporal redundancy at the feature or token level, leaving the underlying mathematical structure of diffusion features largely unexplored. In this work, we reveal that world-model features exhibit highly stable singular subspaces across nearby denoising steps, while their singular values follow predictable evolution patterns. Building on this observation, we propose SpectralCache, a training-free spectral caching framework that reuses stable singular subspaces and estimates only low-dimensional singular values through linear extrapolation. We further exploit the spectral consistency between neighboring full-computation features to skip selected expensive backbone evaluations via singular value scaling. Extensive experiments on representative world models demonstrate that SpectralCache consistently improves inference efficiency while preserving generation quality. On HunyuanWorld-Voyager-13B, SpectralCache achieves 5.22x acceleration while maintaining a WorldScore of 65.90 for static scenes, substantially outperforming existing training-free caching methods in inference efficiency.

Figures & tables

Figure 1: (a) Visual comparison, (b) latency comparison, and (c) quality comparison of SpectralCache with baselines on HunyuanWorld-Voyager-13B. SpectralCache achieves about 5.22×5.22\times acceleration while preserving high-fidelity video quality .
Figure 2: Spectral dynamics across denoising steps in HunyuanWorld-Voyager. (a) The dominant singular subspaces remain highly stable across consecutive denoising steps. (b) Singular values exhibit a consistent linear extrapolation pattern across denoising steps.
Figure 3: Spectral consistency across nearby full-computation steps. (a) The dominant singular subspaces remain highly consistent. (b) The spectral magnitude ratio remains approximately constant.
Figure 4: Overview of SpectralCache.
Table 5
MethodsPSNR ↑\uparrowSSIM ↑\uparrowWS-Static ↑\uparrowWS-Dynamic ↑\uparrowLatency ↓\downarrowMemory (GB) ↓\downarrow
Voyager∞\infty1.00066.2846.401.00 ×\times50.44
HERO17.710.60162.3743.670.96 ×\times173.42
WorldCache23.490.77064.8945.433.65 ×\times50.58
SpectralCache (Ours)*26.090.83266.2046.313.84 ×\times50.44
SpectralCache (Ours)25.060.81865.9046.135.22 ×\times50.44
Table 5: Ablation study on the effect of singular value scaling for full computation. * denotes our method with singular value scaling disabled at triggered full computation steps.
Figure 7
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: More visual comparison on HunyuanVoyager-13B between SpectralCache and existing methods.
Figure 7: More visual comparison on Aether-5B between SpectralCache and existing methods.
τ\tauPSNR ↑\uparrowLatency (rel. to base) ↓\downarrow
EasyCache21.763.58 ×\times
0.525.394.42 ×\times
0.725.064.83 ×\times
1.025.065.22 ×\times
1.324.805.32 ×\times
1.524.715.36 ×\times
Appendix
Table 7: Ablation study on different cache error threshold τ\tau .
{ps,pc}\{p_{s},p_{c}\}PSNR ↑\uparrowSSIM ↑\uparrow
EasyCache21.760.737
{0.3,0.8}\{0.3,0.8\}24.840.813
{0.3,0.7}\{0.3,0.7\}25.060.818
{0.3,0.6}\{0.3,0.6\}24.980.814
{0.2,0.8}\{0.2,0.8\}24.810.808
{0.2,0.7}\{0.2,0.7\}24.940.811
Appendix
Table 8: Ablation study on token grouping percentiles.
More in this topic

Diffusion Model Caching

Diffusion model caching reuses features, activations, or attention states across denoising steps to reduce inference cost while preserving output quality.

See all 112 papers

Explore similar work

CardsList
Mar 6, 2026

WorldCache: Accelerating World Models for Free via Heterogeneous Token Caching

Diffusion-based world models have shown strong potential for unified world simulation, but the iterative denoising remains too costly for interactive use and long-horizon rollouts. While feature caching can accelerate inference without training, we find that policies designed for single-modal diffusion transfer poorly to world models due to two world-model-specific obstacles: token heterogeneity from multi-modal coupling and spatial variation, and non-uniform temporal dynamics where a small set of hard tokens drives error growth, making uniform skipping either unstable or overly conservative. We propose WorldCache, a caching framework tailored to diffusion world models. We introduce Curvature-guided Heterogeneous Token Prediction, which uses a physics-grounded curvature score to estimate token predictability and applies a Hermite-guided damped predictor for chaotic tokens with abrupt direction changes. We also design Chaotic-prioritized Adaptive Skipping, which accumulates a curvature-normalized, dimensionless drift signal and recomputes only when bottleneck tokens begin to drift. Experiments on diffusion world models show that WorldCache delivers up to \textbf{3.7×\times} end-to-end speedups while maintaining 98% rollout quality, demonstrating the vast advantages and practicality of WorldCache in resource-constrained scenarios. Our code is released in https://github.com/FofGofx/WorldCache.

Weilun Feng, Guoxin Fan, Haotong Qin +10

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · ETH Zürich +4

Diffusion Model Inference AccelerationDiffusion Model CachingInteractive World ModelsWorld Model Learning
AskPDF
Mar 23, 2026

WorldCache: Content-Aware Caching for Accelerated Video World Models

Diffusion Transformers (DiTs) power high-fidelity video world models but remain computationally expensive due to sequential denoising and costly spatio-temporal attention. Training-free feature caching accelerates inference by reusing intermediate activations across denoising steps; however, existing methods largely rely on a Zero-Order Hold assumption i.e., reusing cached features as static snapshots when global drift is small. This often leads to ghosting artifacts, blur, and motion inconsistencies in dynamic scenes. We propose WorldCache, a Perception-Constrained Dynamical Caching framework that improves both when and how to reuse features. WorldCache introduces motion-adaptive thresholds, saliency-weighted drift estimation, optimal approximation via blending and warping, and phase-aware threshold scheduling across diffusion steps. Our cohesive approach enables adaptive, motion-consistent feature reuse without retraining. On Cosmos-Predict2.5-2B evaluated on PAI-Bench, WorldCache achieves \textbf{2.3×\times} inference speedup while preserving 99.4% of baseline quality, substantially outperforming prior training-free caching approaches. Our code can be accessed on World-Cache.

Umair Nawaz, Ahmed Heakl, Ufaq Khan +3

Mohamed bin Zayed University of Artificial Intelligence, UAE · Linköping University, Sweden

Diffusion Model CachingDiffusion Model Inference AccelerationVideo Diffusion ModelsVideo World Models
AskPDF
Feb 22, 2026

SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Models

Diffusion models are a strong backbone for visual generation, but their inherently sequential denoising process leads to slow inference. Previous methods accelerate sampling by caching and reusing intermediate outputs based on feature distances between adjacent timesteps. However, existing caching strategies typically rely on raw feature differences that entangle content and noise. This design overlooks spectral evolution, where low-frequency structure appears early and high-frequency detail is refined later. We introduce Spectral-Evolution-Aware Cache (SeaCache), a training-free cache schedule that bases reuse decisions on a spectrally aligned representation. Through theoretical and empirical analysis, we derive a Spectral-Evolution-Aware (SEA) filter that preserves content-relevant components while suppressing noise. Employing SEA-filtered input features to estimate redundancy leads to dynamic schedules that adapt to content while respecting the spectral priors underlying the diffusion model. Extensive experiments on diverse visual generative models and the baselines show that SeaCache achieves state-of-the-art latency-quality trade-offs.

Jiwoo Chung, Sangeek Hyun, MinKyu Lee +5

Sungkyunkwan University · NAVER Cloud

Diffusion Model CachingDiffusion Model Inference Acceleration
AskPDF