CAST: Reconstruction-Coupled Acceleration of Interactive World Models

Leyang Chen; Junyi Wu; Fanqing Kong; Shaoqiu Zhang; Yulun Zhang

cs.CVSep 28, 2026

CAST: Reconstruction-Coupled Acceleration of Interactive World Models

Authors: Leyang Chen, Junyi Wu, Fanqing Kong, Shaoqiu Zhang, Yulun Zhang

Organizations: Shanghai Jiao Tong University

Abstract

Interactive world models must respond quickly to controls while preserving scene consistency. Existing acceleration methods can miss heterogeneous control responses and spatial transport when recovering skipped features. We observe that interaction-induced feature changes correlate with approximation error, while low-frequency interpolation errors are phase-sensitive and show more predictable phase progression. These findings motivate CAST, a reconstruction-coupled inference framework. CAST selects anchors by interaction sensitivity and cross-layer coverage, reconstructs skipped residuals with frequency- and confidence-aware Phase-Aware Reconstruction (PAR), and coordinates historical KV routing according to downstream reconstruction responsibility. On Matrix-Game 3.0 and HY-World 1.5, CAST achieves 2.15x and 3.48x speedups, respectively, while maintaining visual quality close to Native (Figure 1). It also attains the highest VBench scores among compared methods and leads non-native baselines on seven and six of thirteen WorldMark dimensions, demonstrating a balance of generation speed, visual quality, and interactive responsiveness under real-time control. Code is available at https://github.com/lokiniuniu/CAST.

Figures & tables

Figure 2: Speed and interactive quality. Speedup over Native (left) and selected WorldMark dimensions (right), min–max normalized per backbone across all methods including Native; colors match the speedup bars.

Figure 3: Control sensitivity and skipping error. Top: native rollout. Middle: independently scaled sensitivity at block $\ell$ and error at $\ell+1$ . Bottom: pooled percentiles (Spearman $\rho=0.74$ , $n=3{,}072$ ), with counts and binned medians.

Figure 4: Spectral error and phase progression. Top: interpolation error with oracle phase/magnitude corrections. Bottom: phase progress across frequency bands; dashed lines denote uniform progression.

Figure 5: CAST pipeline. Select anchors and historical KV blocks, compute anchor residuals, and reconstruct skipped updates while retaining each frame’s own skip connection.

Figure 6: Control responses and coverage debt guide selection.

Figure 7: Phase-aware reconstruction with

Q=\overline{Z}

, where

Z

is defined in Eq. ( 6 ).

	Action Dynamics $\uparrow$								World Memory $\uparrow$			Visual Quality $\uparrow$		Efficiency
Method	Direction Accuracy		Direction Purity		Motion Stability		Response Latency		Local	Global	Revisit	Perceptual	Aesthetic	Speedup $\uparrow$
	Trans.	Rot.	Trans.	Rot.	Trans.	Rot.	Trans.	Rot.
Matrix-Game 3.0
Native	98.053	87.203	83.605	82.966	79.832	98.184	93.636	96.296	76.763	47.755	80.124	77.174	58.688	$1\times$
SVG	97.443	82.404	84.411	77.974	80.687	97.674	92.508	96.287	76.253	45.632	78.826	75.809	58.118	$0.62\times$
BSA	97.306	86.877	84.061	82.903	82.822	98.427	93.748	96.268	76.502	44.809	80.591	77.362	59.192	$0.94\times$

Table 1: WorldMark results; higher is better. Bold marks the best non-native score per backbone. Speedups are taken from our matched runtime evaluation in Table 2 and are not measured on the WorldMark trajectories used to compute the quality and control scores.

Figure 8: Initialize each anchor’s mask, then refine it using local and neighboring reconstruction gains while preserving a fixed historical KV budget at every anchor.

Figure 9: Baseline rollouts on WorldMark. Matrix-Game 3.0 (left) and HY-World 1.5 (right).

Method	vs. Original			Self-Comparison			VBench $\uparrow$	Speedup $\uparrow$
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VBench $\uparrow$	Speedup $\uparrow$
Matrix-Game 3.0
Native	—	—	—	17.07	0.5264	0.3283	0.7504	$1\times$
SVG	12.38	0.4170	0.5587	14.48	0.4949	0.4406	0.7511	$0.62\times$
BSA	13.34	0.4228	0.5795	16.66	0.5326	0.4094	0.7336	$0.94\times$
MagCache	15.02	0.4902	0.4504	14.06	0.5398	0.3837	0.7192	$1.38\times$

Table 2: VBench and other quality results. Bold: best non-native score per backbone; arrows: preferred direction.

Variant	VBench $\uparrow$	LPIPS $\downarrow$	T-Motion $\uparrow$	Local $\uparrow$	Global $\uparrow$	Revisit $\uparrow$	Latency $\downarrow$	Speedup $\uparrow$
A: Uniform	0.7218	0.5128	74.218	70.864	38.927	72.991	1,934	$2.23\times$
B: + CAFS	0.7346	0.4974	76.482	72.943	41.586	75.648	1,951	$2.21\times$
C: + PAR	0.7479	0.4776	78.219	75.046	44.387	79.206	1,976	$2.19\times$
Native	0.7504	—	79.832	76.763	47.755	80.124	4,319	$1.00\times$
+ CAFS	0.7370	0.4960	76.800	74.000	43.200	76.700	2,800	$1.54\times$
+ CAFS+ PAR	0.7600	0.4550	80.200	76.900	47.500	82.000	2,825	$1.53\times$

Table 3: Progressive configurations and fixed-budget ablations on Matrix-Game 3.0. A: uniform/interleaved anchors + linear reconstruction; B: CAFS + linear reconstruction; C: CAFS + PAR. A–C use independent KV masks. Fixed-budget variants A–D use frame retention

r_{f}=0.50

and historical KV retention

r_{\mathrm{KV}}=0.20

. T-Motion, Local, Global, and Revisit are WorldMark scores. Speedup is relative to Native; latency is measured in ms/chunk and excludes VAE decoding for every configuration.

Variant	Core	Route	Recon.	Total	OH	Speedup
Matrix-Game 3.0
Native	4,319	0	0	4,319	0.0%	$1.00\times$
Indep. (C)	1,887	43	46	1,976	4.5%	$2.19\times$
CAST	1,891	74	42	2,007	5.8%	$\mathbf{2.15\times}$
HY-World 1.5
Native	8,713	0	0	8,713	0.0%	$1.00\times$

Table 4: Mean steady-state runtime (ms/chunk); OH includes routing and reconstruction.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

1	Input: ordered anchors, omission scores $u$ , fixed candidates $\mathcal{C}_{i}$ , interval coefficients, budget $K_{\mathrm{KV}}$ , boundary coefficients from Eq. ( 21 ), and directional-pass cap $L_{s}$ .
2	Initialize $\mathcal{M}_{i}\leftarrow\TopK_{m\in\mathcal{C}_{i}}(u_{i,m},K_{\mathrm{KV}})$ for every anchor; construct binary masks $x$ .
3	For directional refinement pass $s=1,\ldots,L_{s}$ :
4	Set $\mathrm{changed}\leftarrow\mathrm{false}$ .
5	For $i$ in forward order if $s$ is odd, backward order otherwise:
6	Compute $G_{i,m}$ using the latest neighbor masks (Eq. ( 23 )).
7	Let $\mathcal{M}_{i}^{\prime}\leftarrow\TopK_{m\in\mathcal{C}_{i}}(G_{i,m},K_{\mathrm{KV}})$ .

Appendix

Algorithm 1 Fixed-budget reconstruction-coupled routing

Field	Matrix-Game 3.0	HY-World 1.5
Checkpoint and code provenance	Skywork/Matrix-Game-3.0 , base_distilled_model ; released code with CAST integration	tencent/HY-WorldPlay , ar_distilled_action_model ; HunyuanVideo 8B path
GPU model and count	$1\times$ RTX A6000 per rollout; batch 1	$2\times$ RTX A6000 per rollout; model/tensor parallel inference; batch 1
Software and attention backend	Python 3.12; PyTorch 2.6; CUDA 12.4; Triton 3.2; FlashAttention 2.7 for dense paths; released Light Interaction block-sparse execution backend	Python 3.10; PyTorch 2.6; CUDA 12.4; Triton 3.2; FlashAttention 2.7 for dense paths; released Light Interaction block-sparse execution backend
Model / attention / FFT precision	BF16 model and QKV; FP32 softmax and pooled reductions; FP32 / complex64 FFT; no INT8/FP8	BF16 model and QKV; FP32 softmax and pooled reductions; FP32 / complex64 FFT; no FP8
Resolution and native chunk	$704\times 1280$ ; steady-state 40 decoded / 10 new latent frames; separate 57-frame startup	$480\times 832$ ; steady-state 16 decoded / 4 latent frames; first chunk has 13 decoded frames
Denoising schedule	3 evaluations; retain the released distilled scheduler and its native timestep ordering	4 evaluations; native few-step AR scheduler; no denoising cache

Appendix

Table 5: Backbone and execution configuration (

8\times

RTX A6000 server).

Field	Matrix-Game 3.0	HY-World 1.5
$r_{f}$ , $K_{f}$ , $\lambda_{\mathrm{cov}}$	$0.50$ , $5$ of $10$ , $0.50$ ; dynamic Top-5 over all current frames; boundary extrapolation uses the two nearest anchors	$0.50$ , $2$ of $4$ , $0.50$ ; dynamic Top-2 over all current frames; boundary extrapolation uses the two nearest anchors
Tiles and reliable spectral bins	$8\times 8$ spatial token tiles, disjoint; $W_{p},\|Z_{p}\|>10^{-6}\max W_{p}+10^{-8}$ ; FP32	$8\times 8$ spatial token tiles, disjoint; $W_{p},\|Z_{p}\|>10^{-6}\max W_{p}+10^{-8}$ ; FP32
$\tau_{\mathrm{low}},\tau_{\mathrm{high}},q,T$	$(0.25,0.75,2,0.10)$ ; principal phase branch; DC/Nyquist transport disabled	$(0.25,0.75,2,0.10)$ ; principal phase branch; DC/Nyquist transport disabled
$K_{\mathrm{KV}}$ , $r_{\mathrm{KV}}$ , $M_{c}$	$K_{\mathrm{KV}}=\lceil 0.20M\rceil$ ; target $r_{\mathrm{KV}}=0.20$ ; $M_{c}=M$ (full candidate pool); bypass if $M=0$	$K_{\mathrm{KV}}=\lceil 0.20M\rceil$ ; target $r_{\mathrm{KV}}=0.20$ ; $M_{c}=M$ ; report realized block fraction
Projection $P$ and scaling	$P=W_{O}^{(h)}$ , native attention output-projection head slice; no additional learned/sketched projection; FP32 pooled reductions	Same native output-projection head slice and FP32 reductions; squared Euclidean norm
Routing weights / stopping	$(\lambda_{\mathrm{anc}},\lambda_{\mathrm{rec}})=(1,1)$ ; $L_{s}=2$ directional passes; accept $\Delta J>10^{-6}\max(J,10^{-8})$	$(\lambda_{\mathrm{anc}},\lambda_{\mathrm{rec}})=(1,1)$ ; $L_{s}=2$ directional passes; same relative tolerance

Appendix

Table 6: CAST and evaluation settings for both backbones under the shared comparison protocol.

Frame selection; PAR and coupled routing fixed
			WorldMark $\uparrow$
Variant	VBench $\uparrow$	LPIPS $\downarrow$	T-Motion	Local	Global	Revisit
Uniform/interleaved	0.7437	0.4884	76.914	73.478	42.806	77.138
Latent-motion magnitude	0.7506	0.4769	79.312	75.087	44.578	79.341
Control sensitivity	0.7570	0.4657	79.830	76.531	46.645	81.514
Reconstruction; control-aware selection and coupled routing fixed

Appendix

Table 7: Controlled frame-selection and reconstruction ablations on Matrix-Game 3.0. Every row uses

r_{f}=0.50

r_{\mathrm{KV}}=0.20

, and the same end-to-end evaluation. T-Motion, Local, Global, and Revisit are representative WorldMark scores; all are higher-is-better.

KV routing	Anchor error $\downarrow$	Interval error $\downarrow$	VBench $\uparrow$	Local $\uparrow$	Global $\uparrow$	Revisit $\uparrow$
Independent (C)	0.0831	0.1127	0.7479	75.046	44.387	79.206
Coupled, $L_{s}=2$ (D)	0.0774	0.1013	0.7570	76.531	46.645	81.514

Appendix

Table 8: Fixed-input coupling ablation on Matrix-Game 3.0. Anchor and interval errors are normalized residual errors under identical anchors and KV budgets. Local, Global, and Revisit are WorldMark memory scores measured on the paired rollouts for both routing variants.

$r_{f}$	$r_{\mathrm{KV}}$	VBench $\uparrow$	LPIPS $\downarrow$	Global	Revisit	Latency $\downarrow$	Speedup $\uparrow$
				WorldMark $\uparrow$
0.30	0.20	0.7441	0.4936	42.738	77.246	1,717	$2.52\times$
0.50	0.10	0.7512	0.4788	42.911	77.926	1,909	$2.26\times$
0.50	0.20	0.7570	0.4657	46.645	81.514	2,007	$\mathbf{2.15\times}$
0.50	0.30	0.7584	0.4608	47.386	82.263	2,078	$2.08\times$
0.60	0.20	0.7601	0.4529	47.219	82.487	2,363	$1.83\times$

Appendix

Table 9: Budget sensitivity on Matrix-Game 3.0 with two directional refinement passes. Ratios are retained fractions; Global and Revisit are WorldMark memory scores, and latency is milliseconds per chunk. All settings use identical prompts, actions, and paired random seeds.

$L_{s}$	VBench $\uparrow$	LPIPS $\downarrow$	Global	Revisit	Interval NRMSE $\downarrow$	Latency $\downarrow$	Speedup $\uparrow$
			WorldMark $\uparrow$
0	0.7479	0.4776	44.387	79.206	0.1127	1,976	$2.19\times$
1	0.7549	0.4689	45.916	80.721	0.1058	1,991	$2.17\times$
2	0.7570	0.4657	46.645	81.514	0.1013	2,007	$\mathbf{2.15\times}$
4	0.7574	0.4649	46.781	81.672	0.1008	2,038	$2.12\times$

Appendix

Table 10: Directional refinement sensitivity on Matrix-Game 3.0 at

r_{f}=0.50

and

r_{\mathrm{KV}}=0.20

. Global and Revisit are WorldMark memory scores; interval NRMSE is measured against fully computed residuals, and latency is milliseconds per chunk.

Figure 10: WorldMark rollouts: Matrix-Game 3.0, fr012_007 (L/R, left), and HY-World 1.5, ts018_006 (W/S, right). All methods use matched frame indices. Columns show the initial view, the midpoint and end of the first action, and the end of the second action, including its reversal of the initial command.

Figure 11: WorldMark rollouts: Matrix-Game 3.0, fr019_007 (L/R, left), and HY-World 1.5, tr006_006 (W/S, right). Methods and frame selection follow Figure 10 .

Figure 12: WorldMark rollouts: Matrix-Game 3.0, fr007_006 (W/S, left), and HY-World 1.5, fr018_007 (L/R, right). Methods and frame selection follow Figure 10 .

Figure 13: WorldMark rollouts: Matrix-Game 3.0, ts020_007 (L/R, left), and HY-World 1.5, ts024_007 (L/R, right). Methods and frame selection follow Figure 10 .

Figure 14: WorldMark rollouts: Matrix-Game 3.0, fr003_010 (S/L, left), and HY-World 1.5, fr012_007 (L/R, right). Methods and frame selection follow Figure 10 .

Explore similar work

Light Interaction: Training-Free Inference Acceleration for Interactive Video World Models
May 29, 2026Jiacheng Lu, Haoyi Zhu, Sipei Yi +3Video World Models Dynamic Sparse Attention
1Zhejiang University · 2NVIDIA
MoWorld: A Flash World Model
Jul 7, 2026Team Moxin, Deyi Ji, Tianrun Chen +26Video World Models World Models
MoWorld Team, Moxin Technology
Matrix-game 2.0: An open-source, real-time, and streaming interactive world model
Aug 18, 2025Xianglong He, Chunli Peng, Zexiang Liu +16Video World Models Interactive Video Generation
Skywork AI

May 29, 2026, cs.CVJ/K move · Enter open · S save

Light Interaction: Training-Free Inference Acceleration for Interactive Video World Models

Jiacheng Lu, Haoyi Zhu, Sipei Yi, Enze Xie+2

1Zhejiang University · 2NVIDIA

Video World Models Dynamic Sparse Attention Peak Memory

Open paper AskPDF arXiv

Interactive video world models generate video chunk by chunk in response to user-controlled camera movements, enabling applications such as real-time game simulation, virtual scene navigation, and embodied AI training. However, scaling to long interactive trajectories is prohibitively expensive due to growing context memory, quadratic attention complexity, and repeated denoising steps. We present Light Interaction, a training-free inference acceleration framework for interactive video world models. Our key insight is that interaction naturally enables trajectory-dependent adaptive computation: retrieved spatial memory can be discarded during novel exploration, temporal context can be adjusted according to local latent dynamics, and early-step model outputs can be reused when the camera revisits familiar regions. Based on this insight, Light Interaction combines adaptive context management, denoising cache acceleration, and hardware-software co-designed 3D block sparse attention with fused Triton kernels. Evaluated on HY-WorldPlay and Matrix-Game-3.0, Light Interaction achieves up to 2.59x speedup without model retraining while maintaining competitive visual quality.

Abstract

Explore similar work

Light Interaction: Training-Free Inference Acceleration for Interactive Video World Models

MoWorld: A Flash World Model

Matrix-game 2.0: An open-source, real-time, and streaming interactive world model