WorldCache: Accelerating World Models for Free via Heterogeneous Token Caching

Weilun Feng; Guoxin Fan; Haotong Qin; Mingqiang Wu; Yuqi Li; Xiangqi Li; Zhulin An; Libo Huang

cs.CVMar 6, 2026

WorldCache: Accelerating World Models for Free via Heterogeneous Token Caching

Authors: Weilun Feng, Guoxin Fan, Haotong Qin, Mingqiang Wu, Yuqi Li, Xiangqi Li, Zhulin An, Libo Huang,

Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · ETH Zürich · City College of New York, City University of New York, USA ·

Abstract

Diffusion-based world models have shown strong potential for unified world simulation, but the iterative denoising remains too costly for interactive use and long-horizon rollouts. While feature caching can accelerate inference without training, we find that policies designed for single-modal diffusion transfer poorly to world models due to two world-model-specific obstacles: token heterogeneity from multi-modal coupling and spatial variation, and non-uniform temporal dynamics where a small set of hard tokens drives error growth, making uniform skipping either unstable or overly conservative. We propose WorldCache, a caching framework tailored to diffusion world models. We introduce Curvature-guided Heterogeneous Token Prediction, which uses a physics-grounded curvature score to estimate token predictability and applies a Hermite-guided damped predictor for chaotic tokens with abrupt direction changes. We also design Chaotic-prioritized Adaptive Skipping, which accumulates a curvature-normalized, dimensionless drift signal and recomputes only when bottleneck tokens begin to drift. Experiments on diffusion world models show that WorldCache delivers up to \textbf{3.7 $\times$ } end-to-end speedups while maintaining 98% rollout quality, demonstrating the vast advantages and practicality of WorldCache in resource-constrained scenarios. Our code is released in https://github.com/FofGofx/WorldCache.

Figures & tables

Figure 2: Overview of the proposed WorldCache framework. The pipeline alternates between FULL backbone evaluation and CACHE approximation. (Top) In each full computation step, tokens are partitioned into Stable, Linear, and Chaotic groups based on their curvature $\kappa$ . (Bottom) During caching steps, heterogeneous predictors (Reuse, Linear Extrapolation, or Damped Update) are applied accordingly. (Left) The Chaotic-prioritized Adaptive Skipping (CAS) mechanism accumulates a curvature-normalized drift score $E_{acc}$ specifically from chaotic tokens, triggering a full computation only when critical drift is detected.

Panel of Figure 3 — Figure 3: An illustration of token heterogeneity. (a) Modality and Spatial Variance: Distinct patterns between modalities and across spatial regions. (b) Trajectory Dynamics: Three trajectory trends: static, predictable, and sharp, non-linear direction shifts that defy simple extrapolation. More analysis in Appendix Sec. E

Panel of Figure 4 — Figure 4: Mechanism and effectiveness of the Damped Update. (a) Trajectory Illustration: Damped update stabilizes prediction through historical $\mathbf{v}_{t^{\star}-1}$ . (b) Quantitative Error Analysis: Damped update reduces chaotic tokens cache error as the prediction window $k$ increases.

Figure 5: An illustration of non-uniform temporal dynamics. We plot the feature difference magnitude across denoising steps for different token percentiles ( $p_{25}$ to $p_{100}$ ). The global drift is dominated by a small subset of ”hard” tokens (top percentile, red line), while the majority remain stable. More analysis in Appendix Sec. F .

Method	Benchmark Evaluation		Perceptual Metrics			Acceleration & Memory
Method	WorldScore Static ↑	WorldScore Dynamic ↑	PSNR ↑	SSIM ↑	LPIPS ↓	Latency(s) ↓	Speed ↑	Memory Overhead(GB) ↓
HunyuanVoyager-13B ( Huang et al., 2025 ) ( $512\times 768p,\texttt{frames}=49$ )
Voyager	66.28	46.40	$\infty$	1.000	0.000	1053.7	1.00 $\times$	50.44
DuCa	53.87	37.71	16.66	0.508	0.486	811.5*	1.30 $\times$	109.70
ToCa	47.49	33.24	15.51	0.409	0.558	1038.4*	1.01 $\times$	107.35
TaylorSeer	62.46	43.72	18.32	0.615	0.293	1195.2*	0.88 $\times$	163.79

Table 1: Quantitative comparison on Image-to-World generation. Bold : the best result. *Note: Layer-wise methods marked with * exceed single-GPU memory limits, requiring CPU offloading which incurs transmission latency.

Method	Video Depth Estimation			Camera Pose Estimation			Acceleration & Memory
Method	Abs Rel ↓	$\delta<1.25_{\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	$\delta<1.25^{2}_{\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	ATE ↓	RPE trans ↓	RPE rot ↓	Latency(s) ↓	Speed ↑	Memory Overhead(GB) ↓
Aether	0.340	0.502	0.738	0.177	0.068	0.780	55.42	1.00 $\times$	50.19
DuCa	0.341	0.475	0.694	0.209	0.069	0.904	28.15	1.97 $\times$	52.70
ToCa	0.341	0.476	0.694	0.209	0.069	0.904	28.02	1.98 $\times$	52.70
TaylorSeer	0.361	0.460	0.718	0.197	0.068	1.134	26.71	2.07 $\times$	58.57
HiCache	0.346	0.472	0.712	0.204	0.069	1.004	26.51	2.09 $\times$	58.57

Table 2: Quantitative comparison on 3D Reconstruction on Aether ( Zhu et al., 2025 ) . Bold : the best result.

$\{p_{s},p_{c}\}$	PSNR ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	SSIM ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	LPIPS ${\mathbf{{\color[rgb]{0.7539,0,0}\downarrow}}}$
EasyCache	21.76	0.737	0.208
{0.3, 0.8}	23.32	0.766	0.179
{0.3, 0.7}	23.49	0.770	0.176
{0.3, 0.6}	23.52	0.768	0.178
{0.2, 0.8}	23.12	0.760	0.185
{0.2, 0.7}	23.12	0.764	0.182

Table 3: Ablation study on token grouping percentiles.

Figure 6: Qualitative comparison of world generation tasks on Voyager ( Huang et al., 2025 ) and Aether ( Zhu et al., 2025 ) .

Method	PSNR ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	SSIM ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	LPIPS ${\mathbf{{\color[rgb]{0.7539,0,0}\downarrow}}}$	Latency(s) ${\mathbf{{\color[rgb]{0.7539,0,0}\downarrow}}}$
TeaCache	26.60	0.843	0.138	114.2
CAS ( $\eta=0.10$ )	31.66	0.922	0.070	109.0
CAS ( $\eta=0.15$ )	31.13	0.916	0.083	108.5
CAS ( $\eta=0.20$ )	30.63	0.908	0.099	107.2
CAS ( $\eta=0.25$ )	29.22	0.907	0.133	99.35
CAS ( $\eta=0.30$ )	28.05	0.894	0.154	93.86

Table 4: Ablation study on adaptive skipping threshold (

\eta

Method	PSNR ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	SSIM ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	LPIPS ${\mathbf{{\color[rgb]{0.7539,0,0}\downarrow}}}$	Latency(s) ${\mathbf{{\color[rgb]{0.7539,0,0}\downarrow}}}$
Reuse	22.74	0.714	0.336	86.32
Linear	18.01	0.537	0.396	87.07
Damped	23.76	0.665	0.276	87.51
Random Group.	22.59	0.710	0.314	86.98
CHTP	25.76	0.791	0.227	86.94

Table 5: Ablation study on token prediction methods.

Method	PSNR ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	SSIM ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	LPIPS ${\mathbf{{\color[rgb]{0.7539,0,0}\downarrow}}}$
Fixed Interval	26.18	0.830	0.216
Difference Guided	26.79	0.824	0.207
Norm Guided	26.02	0.809	0.217
Curvature Guided	25.87	0.788	0.236
CAS	27.10	0.881	0.198

Table 6: Ablation study on steps skipping strategies.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Resolution	PSNR ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	SSIM ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	LPIPS ${\mathbf{{\color[rgb]{0.7539,0,0}\downarrow}}}$	Speed ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$
DuCa	$464\times 832$	19.10	0.716	0.182	1.87 $\times$
ToCa	$464\times 832$	18.85	0.704	0.198	1.84 $\times$
TaylorSeer	$464\times 832$	17.05	0.641	0.286	1.82 $\times$
HiCache	$464\times 832$	17.82	0.678	0.241	1.86 $\times$
TeaCache	$464\times 832$	19.85	0.742	0.136	2.08 $\times$
EasyCache	$464\times 832$	20.28	0.751	0.118	2.16 $\times$

Appendix

Table 7: Broader evaluation on LingBot-14B across two resolutions.

Method	Steps	PSNR ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	SSIM ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	LPIPS ${\mathbf{{\color[rgb]{0.7539,0,0}\downarrow}}}$	Speed ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$
DuCa	70	18.16	0.552	0.452	1.41 $\times$
ToCa	70	17.01	0.453	0.524	1.10 $\times$
TaylorSeer	70	19.82	0.659	0.259	0.98 $\times$
HiCache	70	20.06	0.667	0.247	1.07 $\times$
TeaCache	70	18.05	0.617	0.331	3.82 $\times$
EasyCache	70	23.56	0.789	0.167	4.03 $\times$

Appendix

Table 8: Broader evaluation on Voyager under different denoising schedules.

Method	Frames	PSNR ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	SSIM ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$	LPIPS ${\mathbf{{\color[rgb]{0.7539,0,0}\downarrow}}}$	Speed ${\mathbf{{\color[rgb]{0.7539,0,0}\uparrow}}}$
ToCa	81	19.54	0.666	0.137	1.03 $\times$
TaylorSeer	81	17.32	0.608	0.212	0.97 $\times$
HiCache	81	18.23	0.643	0.171	0.99 $\times$
TeaCache	81	20.89	0.701	0.085	2.23 $\times$
EasyCache	81	21.81	0.705	0.082	2.29 $\times$
WorldCache	81	23.37	0.780	0.001	2.47 $\times$

Appendix

Table 9: Broader evaluation on longer LingBot rollouts beyond the 49-frame main setting.

Model	WorldCache Overhead (s) ↓	End-to-End Latency (s) ↓	Overhead / Latency ↓
LingBot-14B	0.189	348.7	0.054%
Voyager-13B	0.137	288.6	0.047%
Aether-5B	0.064	107.2	0.060%

Appendix

Table 10: Runtime overhead of WorldCache control logic.

Context Length	Baseline Mem.(GB) ↓	WorldCache Mem.(GB) ↓	Extra Mem.(GB) ↓	Speed ↑
49	49.37	49.39	0.02	2.41 $\times$
81	55.87	55.89	0.02	2.47 $\times$
161	71.32	71.35	0.03	2.53 $\times$

Appendix

Table 11: Memory behavior of WorldCache as context length increases.

Panel of Figure 7 — Figure 7: More visualization of token heterogeneity. We visualize both RGB tokens and Depth tokens of different prompts across 50 denoising steps in HunyuanVoyager ( Huang et al., 2025 ) . The distinct patterns between RGB and Depth, as well as the spatial variance within each map, underscore the necessity of heterogeneous processing.

Figure 8: Visualization of non-uniform temporal error dynamics. We plot the prediction error accumulation across different token percentiles ( $p_{25}$ to $p_{100}$ ). The results show that global variance is dominated by the top percentile of “chaotic” tokens (red line), while the majority of tokens ( $p_{50}$ and below) remain stable. This supports our strategy of monitoring the Chaotic group rather than the global mean.

Figure 9: Visualization of temporal scale variance. Histograms show the distribution of feature values at different denoising steps. The numerical range fluctuates by orders of magnitude (e.g., thousands at Step 1 vs. single digits at Step 37), proving that fixed thresholds on raw values are unreliable and validating the necessity of our dimensionless drift indicator.

Figure 10: More visual comparison between WorldCache and existing methods.

Figure 11: More visual comparison between WorldCache and existing methods.

Figure 12: More visual comparison between WorldCache and existing methods.

Figure 13: More visual comparison between WorldCache and existing methods.

Diffusion Model Inference Acceleration

Diffusion model inference acceleration reduces the time and compute needed to generate or reconstruct outputs while preserving quality.

See all 506 papers

WorldCache: Content-Aware Caching for Accelerated Video World Models
Mar 23, 2026Umair Nawaz, Ahmed Heakl, Ufaq Khan +3Diffusion Model Caching Diffusion Model Inference Acceleration
Mohamed bin Zayed University of Artificial Intelligence, UAE · Linköping University, Sweden
SpectralCache: Accelerating Diffusion-Based World Models via Spectral Feature Caching
Oct 2, 2026Zhendong Mi, Pu Zhao, Ziyu Hu +4Diffusion Model Caching Diffusion Model Inference Acceleration
Stevens Institute of Technology · Northeastern University · Technische Universität Darmstadt
WorldDynCache: Risk-Controlled Latent Dynamics Approximation for Diffusion World Model
Aug 3, 2026Leyang Chen, Junyi Wu, Shaoqiu Zhang +1World Model Learning Latent Dynamics Modeling
Shanghai Jiao Tong University

Mar 23, 2026J/K move · Enter open · S save

WorldCache: Content-Aware Caching for Accelerated Video World Models

Umair Nawaz, Ahmed Heakl, Ufaq Khan, Abdelrahman Shaker+2

Mohamed bin Zayed University of Artificial Intelligence, UAE · Linköping University, Sweden

Diffusion Model Caching Diffusion Model Inference Acceleration Video Diffusion Models Video World Models

Open paper AskPDF arXiv

Diffusion Transformers (DiTs) power high-fidelity video world models but remain computationally expensive due to sequential denoising and costly spatio-temporal attention. Training-free feature caching accelerates inference by reusing intermediate activations across denoising steps; however, existing methods largely rely on a Zero-Order Hold assumption i.e., reusing cached features as static snapshots when global drift is small. This often leads to ghosting artifacts, blur, and motion inconsistencies in dynamic scenes. We propose WorldCache, a Perception-Constrained Dynamical Caching framework that improves both when and how to reuse features. WorldCache introduces motion-adaptive thresholds, saliency-weighted drift estimation, optimal approximation via blending and warping, and phase-aware threshold scheduling across diffusion steps. Our cohesive approach enables adaptive, motion-consistent feature reuse without retraining. On Cosmos-Predict2.5-2B evaluated on PAI-Bench, WorldCache achieves \textbf{2.3

\times

} inference speedup while preserving 99.4% of baseline quality, substantially outperforming prior training-free caching approaches. Our code can be accessed on World-Cache.

Abstract

Diffusion Model Inference Acceleration

Explore similar work

WorldCache: Content-Aware Caching for Accelerated Video World Models

SpectralCache: Accelerating Diffusion-Based World Models via Spectral Feature Caching

WorldDynCache: Risk-Controlled Latent Dynamics Approximation for Diffusion World Model