Mix Forcing: Causal Pre-Training via Mixture Attention for Interactive World Models

Zelin Gao; Hao Ouyang; Jiapeng Zhu; Qiuyu Wang; Jingye Chen; Zichen Liu; Xing Zhu; Yujun Shen

cs.CVOct 24, 2026

Mix Forcing: Causal Pre-Training via Mixture Attention for Interactive World Models

Authors: Zelin Gao, Hao Ouyang, Jiapeng Zhu, Qiuyu Wang, Jingye Chen, Zichen Liu, Xing Zhu, Yujun Shen,

Organizations: The Hong Kong University of Science and Technology · Robbyant

Abstract

While interactive video world models require autoregressive generation, directly adapting pretrained bidirectional video diffusion models into autoregressive ones with teacher forcing can disrupt their visual priors and degrade autoregressive generation quality. In this paper, we introduce Mix Forcing, a causal pre-training paradigm for interactive video world models, implemented with the Mixture-of-Bidirectional-and-Autoregressive (MoBA) attention. Mix Forcing jointly trains a video diffusion model with both teacher forcing and bidirectional training, yielding a strong causal initialization for few-step post-training. In self-attention, Mix Forcing leverages teacher forcing to learn causal next-chunk prediction for autoregressive rollout, while using bidirectional attention to preserve pretrained visual priors. To support online text interaction, Mix Forcing further adopts a chunk-wise cross-attention design that conditions each video chunk on a shared environment prompt and its own local prompt, without exposing future instructions to the current chunk. Following causal pre-training on a 14B image-to-video model, we apply consistency and distribution matching distillation to obtain a four-step autoregressive generator. Experiments on our internal benchmark and public benchmarks, including WBench and SANA-WM-Bench, show that our model achieves state-of-the-art visual quality and timely event following with precise camera control. Notably, our model supports high-fidelity long-horizon generation without quality degradation even over one-hour rollouts. Project page is here: https://zelingao98.github.io/mixforcing

Figures & tables

Figure 1: Long-horizon interactive video rollout with camera control and memory. Top: a single one-hour autoregressive rollout generated by our model trained with Mix Forcing , following successive prompt updates under keyboard camera control. Bottom: a memory case, where the camera looks away and later returns, and previously observed regions (A: waterfall; B: red foliage) remain consistent when revisited.

Figure 2: Pipeline of our interactive video world model. Given an initial image and an environment prompt $\mathbf{p}_{e}$ , an interactive world is simulated by our four-step causal video model. Based on the KV-cached history context and user inputs, including the camera pose $\mathbf{e}_{i}$ and the local prompt $\mathbf{p}_{i}$ , each latent chunk is generated autoregressively and decoded into a video clip by the VAE decoder.

Figure 3: MoBA attention block. For self-attention (left), each noisy TF chunk attends only to itself and its clean history, while the appended BID noisy latents attend only to one another bidirectionally. For cross-attention (right), TF chunks attend to the shared environment prompt

\mathbf{p}_{e}

and their corresponding local prompt

\mathbf{p}_{i}

, while BID chunks attend to the global prompt

\mathbf{p}_{g}

Method	Quality $\bm{\uparrow}$						Event Following $\bm{\uparrow}$			Camera Error $\bm{\downarrow}$
Method	AQ	IQ	TFlk	DD	HPSv3	Avg	EA	OT 1.0s	OT 1.5s	Rot	Trans	CMC
HY-WorldPlay	59.4	62.4	96.8	55.6	58.4	66.5	0.0	0.0	0.0	1.795	0.553	1.968
JoyAI-Echo-WM	62.7	62.5	95.4	74.6	62.2	71.5	45.2	5.6	26.2	0.928	0.463	1.126
Matrix-Game 3.0	60.6	65.2	95.3	87.3	60.7	73.8	0.0	0.0	0.0	0.441	0.329	0.694
SANA-WM-Streaming	61.0	56.9	96.6	79.4	59.1	70.6	41.3	4.8	23.0	2.022	0.466	2.067
LingBot-World-Fast	63.8	61.7	96.3	73.0	67.4	72.4	86.5	38.1	87.3	1.065	0.433	1.312

Table 1: Quantitative comparison on InBench. The quality, event-following, and camera-error groups are evaluated on the full InBench, the interactive subset, and the roaming subset, respectively. Metrics follow Sec. 4.1 , and Avg denotes the average of the quality metrics. EA denotes event adherence, and OT 1.0s and OT 1.5s denote on-time accuracy within 1.0 and 1.5 seconds. The best results are in bold, and the top three are highlighted from dark to light blue.

Table 2: Quantitative comparison on SANA-WM-Bench. 60-second generation under (a) simple and (b) hard camera trajectories.

\mathrm{IQ}_{0\text{--}10}

and

\mathrm{IQ}_{50\text{--}60}

denote imaging quality over the first and last 10 seconds. Other metrics follow Sec. 4.1 . The best results are in bold, and the top three are highlighted from dark to light blue.

Figure 4: Pairwise user study on InBench. Each bar compares our model with one baseline and shows the percentages of judgments preferring ours, rating a tie, and preferring the baseline.

Panel of Figure 5 — Figure 5: Qualitative comparisons with state-of-the-art methods. (a) Camera-controlled roaming: our model preserves structure and fine details during viewpoint changes, whereas other methods suffer from blurring or distortion. (b) Online prompt interaction: our model follows each event prompt with a consistent scene, whereas other methods respond with delays or break scene consistency.

Attention Pattern	Chunk	Quality $\bm{\uparrow}$	Temporal IQ $\bm{\uparrow}$			Camera Error $\bm{\downarrow}$
Attention Pattern	Chunk		$\bm{\mathrm{IQ}_{0\text{--}2}}$	$\bm{\mathrm{IQ}_{8\text{--}10}}$	$\bm{\mathrm{SIQ}}$	Rot	Trans	CMC
TF	1	68.9	51.5	48.3	49.9	1.098	0.445	1.425
TF+DF	1	65.6	52.8	50.9	51.9	0.655	0.249	0.871
TF+DF+BID	1	67.7	52.1	48.8	50.5	0.751	0.258	0.986
TF+BID	1	70.8	55.9	61.0	58.4	0.449	0.231	0.647
TF+BID	4	69.1	55.2	58.8	57.0	0.884	0.323	1.165

Table 3: Ablation of MoBA self-attention design. Combinations of teacher forcing (TF), diffusion forcing (DF), and bidirectional (BID) training with different training chunk sizes.

\mathrm{IQ}_{0\text{--}2}

and

\mathrm{IQ}_{8\text{--}10}

denote imaging quality over the first and last 2 seconds. The best results are in bold, and the top three are highlighted from dark to light blue.

Prompt	Quality $\bm{\uparrow}$	Event Following $\bm{\uparrow}$		Camera Error $\bm{\downarrow}$
Prompt		EA	OT 1.5s	Rot	Trans	CMC
Global	70.8	23.4	18.3	0.449	0.231	0.647
Chunk-Wise	71.1	50.0	62.1	0.319	0.139	0.431

Table 4: Ablation of MoBA cross-attention design. Global versus chunk-wise prompt conditioning, both built on the best self-attention setting in Tab. 3 . Metrics follow Sec. 4.1 . The best results are in bold and highlighted in blue.

Pre-training	Post-training	Steps	Quality $\uparrow\kern 0.0pt"22$	Event Following $\bm{\uparrow}$		Camera Error $\bm{\downarrow}$
Pre-training	Post-training	Steps		EA	OT 1.5s	Rot	Trans	CMC
Mix Forcing	None	50	71.1	50.0	62.1	0.319	0.139	0.431
TF only	CD $\rightarrow$ DMD	4	64.1	33.7	19.8	1.077	0.446	1.331
Mix Forcing	CD $\rightarrow$ DMD	4	70.9	55.1	64.3	0.435	0.234	0.593

Table 5: Ablation of Mix Forcing for few-step post-training. Four-step models obtained from TF-only and Mix Forcing pre-training under the same CD and DMD pipeline, with the 50-step Mix Forcing model as a reference. Metrics follow Sec. 4.1 . The best results are in bold and highlighted in blue.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Quality $\bm{\uparrow}$						Setting $\bm{\uparrow}$			Interaction $\bm{\uparrow}$
Method	AQ	IQ	TFlk	DD	HPSv3	Avg	Scene	Subject	Avg	Acc	Cons	Avg
HY-WorldPlay	63.8	67.0	93.7	66.5	71.8	72.6	56.5	89.7	73.1	83.0	91.5	87.2
JoyAI-Echo-WM	62.8	65.9	92.2	94.9	64.4	76.0	60.6	89.0	74.8	85.3	87.7	86.5
Matrix-Game 3.0	57.7	64.9	91.1	94.3	56.1	72.8	50.0	80.7	65.3	82.2	82.7	82.5
SANA-WM-Streaming	60.6	62.8	94.7	95.6	61.2	75.0	62.3	87.7	75.0	82.3	80.6	81.5
LingBot-World-Fast	60.1	63.1	92.4	98.7	61.9	75.3	70.2	90.1	80.2	83.2	79.6	81.4

Appendix

Table S1: Quantitative comparison on WBench Navi-158. The Setting group measures scene- and subject-level adherence, while the Interaction group reports navigation accuracy (Acc) and consistency (Cons). Avg denotes the mean within each group. Darker highlights indicate higher ranks among the three best averages.

Simple Trajectory Split (60s)
Method	Quality $\bm{\uparrow}$
Method	AQ	IQ	TFlk	DD	HPSv3	Avg
HY-WorldPlay	56.7	69.3	97.9	25.0	59.7	61.7
JoyAI-Echo-WM	55.4	69.5	96.3	66.3	59.6	69.4
Matrix-Game 3.0	50.4	68.4	93.1	97.5	42.8	70.4
SANA-WM-Streaming	51.1	63.5	97.7	90.0	46.1	69.7

Appendix

Table S2: Detailed quality results on SANA-WM-Bench. Quality metrics are evaluated on 60-second videos. Avg denotes their arithmetic mean; higher scores are better. Darker highlights indicate higher ranks among the three best averages within each split. Metric definitions are provided in Sec. B.1 .

Figure S2: Qualitative comparisons with state-of-the-art methods (1/9). Each group shows one case, with six methods evaluated at matched generated-video timestamps. Column headers summarize the active event instructions; “Scene” denotes environment-only conditioning. Pressed WASD/IJKL keys are shown with a light background according to the recorded keyboard inputs.

Figure S3: Qualitative comparisons with state-of-the-art methods (2/9). Each group shows one case, with six methods evaluated at matched generated-video timestamps. Column headers summarize the active event instructions; “Scene” denotes environment-only conditioning. Pressed WASD/IJKL keys are shown with a light background according to the recorded keyboard inputs.

Figure S4: Qualitative comparisons with state-of-the-art methods (3/9). Each group shows one case, with six methods evaluated at matched generated-video timestamps. Column headers summarize the active event instructions; “Scene” denotes environment-only conditioning. Pressed WASD/IJKL keys are shown with a light background according to the recorded keyboard inputs.

Figure S5: Qualitative comparisons with state-of-the-art methods (4/9). Each group shows one case, with six methods evaluated at matched generated-video timestamps. Column headers summarize the active event instructions; “Scene” denotes environment-only conditioning. Pressed WASD/IJKL keys are shown with a light background according to the recorded keyboard inputs.

Figure S6: Qualitative comparisons with state-of-the-art methods (5/9). Each group shows one case, with six methods evaluated at matched generated-video timestamps. Column headers summarize the active event instructions; “Scene” denotes environment-only conditioning. Pressed WASD/IJKL keys are shown with a light background according to the recorded keyboard inputs.

Figure S7: Qualitative comparisons with state-of-the-art methods (6/9). Each group shows one case, with six methods evaluated at matched generated-video timestamps. Column headers summarize the active event instructions; “Scene” denotes environment-only conditioning. Pressed WASD/IJKL keys are shown with a light background according to the recorded keyboard inputs.

Figure S8: Qualitative comparisons with state-of-the-art methods (7/9). Each group shows one case, with six methods evaluated at matched generated-video timestamps. Column headers summarize the active event instructions; “Scene” denotes environment-only conditioning. Pressed WASD/IJKL keys are shown with a light background according to the recorded keyboard inputs.

Figure S9: Qualitative comparisons with state-of-the-art methods (8/9). Each group shows one case, with six methods evaluated at matched generated-video timestamps. Column headers summarize the active event instructions; “Scene” denotes environment-only conditioning. Pressed WASD/IJKL keys are shown with a light background according to the recorded keyboard inputs.

Figure S10: Qualitative comparisons with state-of-the-art methods (9/9). Each group shows one case, with six methods evaluated at matched generated-video timestamps. Column headers summarize the active event instructions; “Scene” denotes environment-only conditioning. Pressed WASD/IJKL keys are shown with a light background according to the recorded keyboard inputs.

Panel of Figure 23 — Figure S11: Qualitative results of causal pre-training. Top: comparisons with MAGI and HY-World 1.5 over 60-second rollouts, with the initial image and full prompt above each group. Bottom: a one-minute camera-controlled rollout from our causal checkpoint. All our results are generated before few-step post-training.

Panel of Figure 24 — Figure S12: One-Hour autoregressive generation. Top: representative frames from a one-hour rollout of our interactive video world model. Bottom: mean Quality scores across one-hour rollouts, evaluated in consecutive five-minute intervals. The shaded region indicates the standard deviation across rollouts. Scores are on a 0–100 scale; higher is better.

Figure S13: Additional interactive results (1/2). (a) Magical effects around a wizard. (b) An office chair on ice, with aurora effects and the appearance of penguins. Each row shows six frames from a prompt interval in temporal order, with the prompt excerpt above. Rows follow the rollout sequence within each group.

Figure S14: Additional interactive results (2/2). (a) Character actions including casting magic, riding a horse, and running. (b) First-person exploration across the Egyptian pyramids, West Lake, the Eiffel Tower, Mount Fuji, and the Taj Mahal. Each row shows a prompt excerpt and six frames in temporal order, following the layout of Fig. S13 .

Video Generation

Video generation studies models that create and edit moving images from text, images, audio, and structured controls.

See all 364 papers

Next Forcing: Causal World Modeling with Multi-Chunk Prediction
Jun 9, 2026Gangwei Xu, Qihang Zhang, Jiaming Zhou +4Causal World Models Autoregressive Video Generation
Robbyant · HUST · HKUST (GZ) +1
Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation
May 14, 2026Min Zhao, Hongzhou Zhu, Kaiwen Zheng +6Autoregressive Diffusion Real-Time Interactive Video Generation
Tsinghua University · ShengShu · Renmin University of China
minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models
May 28, 2026Min Zhao, Hongzhou Zhu, Bokai Yan +9Interactive World Models Video Diffusion Models
1ShengShu 2THU · 1ShengShu 3RUC · 1ShengShu +4

Jun 9, 2026J/K move · Enter open · S save

Next Forcing: Causal World Modeling with Multi-Chunk Prediction

Gangwei Xu, Qihang Zhang, Jiaming Zhou, Xing Zhu+3

Robbyant · HUST · HKUST (GZ)+1

Causal World Models Autoregressive Video Generation Video World Models Causal Video Generation

Open paper AskPDF arXiv

Autoregressive video generation has emerged as a powerful paradigm for World Action Models (WAMs). However, existing approaches suffer from slow training convergence and limited converged accuracy, particularly at high frame rates, as the training supervision is confined to the current chunk without explicit signals about future dynamics; they also suffer from slow inference due to iterative video denoising. In this paper, we present Next Forcing, a multi-chunk prediction (MCP) framework for causal world modeling that enables faster training, higher accuracy, and accelerated inference. Inspired by multi-token prediction in large language models, Next Forcing introduces an MCP training objective that augments the main model with lightweight auxiliary MCP modules to simultaneously denoise video chunks at multiple future temporal horizons (next

^1

, next

^2

, next

^3

chunks). These MCP modules form a causal chain across prediction depths, where intermediate features fused from multiple layers of the main model are leveraged to predict future dynamics, allowing near-future predictions to inform farther-future ones and providing dense multi-scale temporal supervision back to the main model. During training, the MCP modules significantly accelerate convergence and improve converged accuracy, especially at high frame rates: at 50 fps, Next Forcing achieves a 93.1% relative improvement over LingBot-VA at 5k training steps and 2.3x faster convergence, and establishes new state-of-the-art results on the RoboTwin benchmark (94.1/93.5% on Clean/Random). At inference, the MCP modules can be retained to predict the next video chunk in parallel with the current one, achieving 2x inference acceleration. Next Forcing also demonstrates significant improvements on PhyWorld, a benchmark evaluating adherence to physical laws in video generation, and over 50% FVD reduction on general video pretraining.

Abstract

Video Generation

Explore similar work

Next Forcing: Causal World Modeling with Multi-Chunk Prediction

Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation

minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models