Stream4D: 4D-Consistency for Streaming Autoregressive Diffusion Video Models

Yuanhao Ban; Jiaqi Feng; Hengguang Zhou; Xiaohuan Pei; Justin Cui; Cho-Jui Hsieh

cs.CVAug 20, 2026

Stream4D: 4D-Consistency for Streaming Autoregressive Diffusion Video Models

Authors: Yuanhao Ban, Jiaqi Feng, Hengguang Zhou, Xiaohuan Pei, Justin Cui, Cho-Jui Hsieh

Organizations: UCLA · Tsinghua University

Abstract

Streaming autoregressive diffusion models enable real-time, long-horizon video generation, but their training objectives optimize local frame prediction rather than the geometry and dynamics of a coherent world: long rollouts accumulate geometric drift and degrade into static or unnatural motion. Recent bidirectional approaches address this problem using rewards signals built upon 3D Gaussian-Splatting reconstruction. However, a single rigid 3d reconstruction cannot model a dynamic scene, so this critic penalizes genuine object motion as reconstruction error and is maximized by freezing the video. This shortcut is especially detrimental in the AR setting, where each chunk can propagate an already-static configuration. In this work, we propose Stream4D, which replaces the static critic with a feed-forward 4D reconstruction reward that explicitly models scene dynamics, allowing coherent motion to receive high consistency rewards. To further guide motion magnitude and quality, we add a motion prior that rewards natural scene-flow magnitude while penalizing jitter and non-rigid artifacts. Our final recipe combines these two terms with a lightweight perceptual anchor. Across various autoregressive video backbones and various generation horizons, Stream4D improves 4D reconstruction quality, preserves motion more effectively, and achieves higher human-aligned preference. Project page: https://banyuanhao.github.io/Stream4D/

Figures & tables

Figure 1: Static-3D rewards freeze the scene; a 4D reward keeps it moving. Five uniformly-spaced frames from a $10.3$ s LongLive rollout (“a cat running away with a fish while people chase behind”). The distilled base contains large motion but the cat and fish drift across frames; the static-3DGS rewards World-R1 and VideoGPA reduce the scene to a more rigid, low-motion configuration. Stream4D keeps the cat running with clear forward motion while preserving a coherent subject and scene.

Figure 2: Unlike a static 3D reward, which punishes motion outright, our dynamic 4D-GS backbone does not penalize motion. Per-prompt 4D-PSNR only drifts weakly downward with motion magnitude $m$ .

Reference: full-diffusion T2V baselines (non-distilled, no RL)
Method	MoVieS Recon.			4DGT Recon.			LLM judge		VideoReward win%
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion $\uparrow$	Consist. $\uparrow$	VQ $\uparrow$	MQ $\uparrow$	TA $\uparrow$	Ovr $\uparrow$
Wan2.1-T2V-1.3B Wan et al. (2025)	19.15	0.828	0.258	17.14	0.614	0.399	—	—	—	—	—	—
CogVideoX-1.5-5B Yang et al. (2024)	17.02	0.768	0.279	15.06	0.518	0.454	—	—	—	—	—	—
Distilled AR backbone: Self-Forcing Huang et al. (2025) (5 s)
Base (B0)	16.88	0.785	0.282	16.28	0.559	0.419	—	—	—	—	—	—

Table 1: Main results on the 500 object-prominent testing set . Metrics as defined in Sec. 4.1 : MoVieS 4D reconstruction score; the 4DGT reconstruction Xu et al. (2025) score (a reconstructor-independent cross-check, Appendix D ); the order-debiased LLM judge evaluating Motion preservation and Consistency ); and VideoReward Liu et al. (2025b) paired win% vs. base.

Stream4D vs.	Human win% $\uparrow$		Gemini judge win% $\uparrow$
Stream4D vs.	Motion	Overall	Motion	Consist.
Base	$43$	$60$	$63$	$70$
World-R1 Wang et al. (2026a)	$72$	$76$	$93$	$68$
VideoGPA Du et al. (2026)	$87$	$80$	$95$	$82$

Table 2: Human study on LongLive. We focus the human evaluation on the long-horizon LongLive backbone, where accumulated drift and the static-collapse shortcut matter most. For each pair raters pick which video shows more natural motion and which is better overall (jointly weighing motion amount, motion quality, and object consistency); win% is Stream4D’s rate (ties count

\tfrac{1}{2}

{>}50\%

prefers ours). For reference we list the Gemini judge’s Motion and Consistency win% on the same

50

prompts: the judge agrees with humans on every ranking (Stream4D preferred over both reward baselines, and below

50\%

on raw motion vs. base), corroborating that the automatic scores are not an artifact, though the judge is somewhat more generous to ours in absolute terms.

Reward configuration	MoVieS Recon.			4DGT Recon.			LLM judge		VideoReward win%
Reward configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion $\uparrow$	Consist. $\uparrow$	VQ $\uparrow$	MQ $\uparrow$	TA $\uparrow$	Ovr $\uparrow$
Backbone: Self-Forcing (5 s)
Base (no RL)	16.88	0.785	0.282	16.28	0.559	0.419	—	—	—	—	—	—
Stream4D (Ours)	20.34	0.874	0.195	17.24	0.650	0.417	0.833	82.2	62.4	62.8	56.2	66.2
$-$ perceptual anchor (a)	22.39	0.905	0.163	18.53	0.723	0.371	0.830	81.7	66.4	64.8	52.8	66.6
$-$ motion term (b)	24.37	0.952	0.096	19.36	0.738	0.328	0.341	31.1	63.0	56.8	44.0	59.2

Table 3: Reward-axis ablation: dropping one of the three reward axes, on all three backbones. Each block lists the distilled base, the deployed Stream4D recipe, then the three single-axis drops. Metric conventions match Table 1 .

Figure 3: Reconstruction-motion trade-off . Deployed recipes (stars) sit on the upper-right frontier.

Motion-term $R_{\text{mot}}$	MoVieS Recon.			4DGT Recon.			LLM judge		VideoReward win%
Motion-term $R_{\text{mot}}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion $\uparrow$	Consist. $\uparrow$	VQ $\uparrow$	MQ $\uparrow$	TA $\uparrow$	Ovr $\uparrow$
Backbone: Self-Forcing (5 s)
Base (no RL)	16.88	0.785	0.282	16.28	0.559	0.419	—	—	—	—	—	—
$g(m)\!\cdot\!\mathrm{smooth}\!\cdot\!\mathrm{rigid}$	20.34	0.874	0.195	17.24	0.650	0.417	0.833	82.2	62.4	62.8	56.2	66.2
$\mathrm{smooth}\!\cdot\!\mathrm{rigid}$ ,	23.43	0.947	0.119	18.02	0.671	0.405	0.406	36.4	78.4	60.4	53.8	74.6
$m\!\cdot\!\mathrm{smooth}\!\cdot\!\mathrm{rigid}$ (e)	18.46	0.833	0.234	16.85	0.632	0.392	0.923	71.2	46.4	47.4	41.8	46.0

Table 4: Motion-term formulation ablation: varying the shape of the motion term

R_{\text{mot}}

while keeping the reconstruction and perceptual axes fixed, on all three backbones.

$m_{\text{nat}}$	MoVieS Recon.			4DGT Recon.			LLM judge		VideoReward win% vs base
$m_{\text{nat}}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion $\uparrow$	Consist. $\uparrow$	VQ $\uparrow$	MQ $\uparrow$	TA $\uparrow$	Ovr $\uparrow$
Self-Forcing base	16.88	0.785	0.282	16.28	0.559	0.419	—	—	—	—	—	—
$0.010$ ( $0.5\times$ )	21.10	0.924	0.135	17.86	0.672	0.345	0.522	44.7	62.8	59.2	65.6	63.8
$\mathbf{0.020}$ (deployed)	20.34	0.874	0.195	17.24	0.650	0.417	0.833	82.2	62.4	62.8	56.2	66.2
$0.030$ ( $1.5\times$ )	18.87	0.836	0.228	17.50	0.645	0.374	0.916	74.4	54.0	51.0	45.6	53.2

Table 5: Sensitivity to the gate target

m_{\text{nat}}

on Self-Forcing. The

0.020

row is the deployed recipe, landing on the knee of the motion-fidelity trade-off, taking the best Consistency and VideoReward-Overall.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Hyperparameter	Value
Backbone	Architecture	Causal Wan 2.1 (Self-Forcing / Causal-Forcing / LongLive)
Backbone	Video resolution ( $H\times W$ )	$480\times 832$
LoRA fine-tuning	Rank $r$ / scaling $\alpha$	256 / 256
	Dropout	0.0
	Gradient checkpointing	True
Optimization	Hardware	4 nodes $\times$ 8 $\times$ H200, DDP

Appendix

Table S1 : Stream4D training hyperparameters. A single training-hyperparameter set is used across Self-Forcing, Causal-Forcing, and LongLive; only the three reward-axis weights vary per backbone.

Distilled AR backbone: Self-Forcing Huang et al. (2025) (5 s)
Method	MoVieS Recon.			4DGT Recon.			LLM judge		VideoReward win%
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion $\uparrow$	Consist. $\uparrow$	VQ $\uparrow$	MQ $\uparrow$	TA $\uparrow$	Ovr $\uparrow$
Base (B0)	18.19	0.831	0.239	16.23	0.546	0.439	—	—	—	—	—	—
World-R1 Wang et al. (2026a)	19.35	0.885	0.182	16.60	0.560	0.412	0.773	68.2	61.4	49.4	54.2	56.8
VideoGPA Du et al. (2026) ‡†	24.88	0.908	0.143	19.87	0.621	0.396	0.477	38.3	63.0	63.4	65.0	67.4
$+$ Stream4D (Ours)	21.23	0.895	0.177	17.70	0.657	0.411	0.816	75.1	59.8	56.6	54.4	59.8

Appendix

Table S2: Robustness check: main comparison on a uniformly random 500-prompt subset of the same held-out VidProM set (seed-0 sample, no motion filter;

122

prompts overlap the high-motion subset of Table 1 ). Metrics and conventions as in Table 1 ( 4DGT : reconstructor-independent dynamic-reconstruction cross-check, Appendix D ); all RL’d rows at ckpt-150, judged against the same backbone’s base. ‡ The VideoGPA row uses the standard VideoGPA recipe (the best-weight variant of Table 1 was generated only on the high-motion subset); on this unfiltered, low-motion-dominated subset its clean freeze sweeps both reconstructors. † marks methods whose Consistency win-rate falls below the

50\,\%

break-even against their own base: on this low-motion-dominated subset, their reconstruction and VideoReward advantages are bought by suppressing motion (motion scores

0.14

–

0.48

), not by generating better video. Bold = best RL’d row per backbone block; shaded = ours.

Panel of Figure 11 — Figure S1 : Motion preserved vs. frozen: the static-reward shortcut, one prompt per backbone. Top: Causal-Forcing (a WW3 battle); middle: Self-Forcing (a futuristic highway); bottom: LongLive (fish in a river). Three uniformly-spaced frames per method, all RL’d rows at ckpt-150; read each row across time. The static-3DGS baselines (VideoGPA, World-R1) are frozen—subjects hold the same pose from $t{=}0$ to $t{=}1$ —while the distilled bases move but deform, and Stream4D keeps the motion while holding object identity and scene structure. Table 1 quantifies motion preservation over the full prompt set.

Figure S2 : Object-identity under motion: the base breaks the object apart, the static-reward baselines freeze the scene, and only ours keeps a moving object intact.

Figure S3 : Consistency companion on Self-Forcing: an ancient-battlefield beast Red outlines mark failure frames and the coloured badge names each row’s behaviour. The failures are complementary: the base moves but morphs ; the static-reward baselines are either frozen . Stream4D stays coherent : one beast, one identity, approaching the camera.

Figure S4 : Consistency companion on LongLive: a planet descending over a city The base deforms into spreading debris/fire; VideoGPA’s explosion flickers in and out over an otherwise static scene; World-R1 is frozen . Stream4D keeps a coherent descending planet over a stable city.

Streaming Video Generation

Streaming video generation creates video incrementally as frames or chunks arrive, enabling low-latency interactive synthesis and editing.

See all 98 papers

Streaming4D: Accelerate 4D World Models via Block-wise Video Generation and Incremental Reconstruction
Sep 1, 2026Xiaoyan Liu, Jiaxin Liu, Kangrui Li +14D Reconstruction 3D Reconstruction
The Chinese University of Hong Kong · The Hong Kong Polytechnic University · The University of New South Wales +1
AdaState: Self-Evolving Anchors for Streaming Video Generation
May 28, 2026Yusuf Dalva, Pinar YanardagVideo Diffusion Models Streaming Video Generation
Virginia Tech
Steady-Forcing: Balancing Spatial Persistence and Motion Continuity in Long-Horizon Nature Video Diffusion
Jun 2, 2026Matiur Rahman Minar, Seunghun Oh, GangHyeon Jeong +1Physical Consistency in Video Generation Long-Horizon Video Generation
Department of Computer Science and Engineering Sogang University Seoul, Korea · Department of Artifical Intelligence Sogang University Seoul, Korea

Sep 1, 2026J/K move · Enter open · S save

Streaming4D: Accelerate 4D World Models via Block-wise Video Generation and Incremental Reconstruction

Xiaoyan Liu, Jiaxin Liu, Kangrui Li, Sifan Zhou

The Chinese University of Hong Kong · The Hong Kong Polytechnic University · The University of New South Wales+1

4D Reconstruction 3D Reconstruction Streaming Video Generation Dynamic Scene Reconstruction 4D Scene Generation Video World Models

Open paper AskPDF arXiv

Current 4D generation paradigms are often bottlenecked by a sequential decoupling design: video is generated first, followed by 3D reconstruction, leading to high interaction latency. This limits applications in interactive real-time scenarios. To this end, we propose Streaming4D, a tightly coupled synchronous pipeline that integrates block-wise autoregressive video generation with incremental 3D reconstruction. Unlike traditional frame-by-frame emission and delayed geometry recovery, Streaming4D generates temporal video blocks and immediately triggers reconstruction for each completed block, enabling parallel execution between synthesis and geometric updates. This approach allows the world representation to evolve online with the video stream, reducing feedback latency while preserving geometric fidelity. We instantiate Streaming4D using a Self-Forcing-style autoregressive generator and an incremental reconstruction backend. Experiments show consistent runtime improvements across resolutions on a single RTX 4090 (1.24

\times

speedup), while maintaining high-quality 4D geometry and multi-view consistency.

Abstract

Streaming Video Generation

Explore similar work

Streaming4D: Accelerate 4D World Models via Block-wise Video Generation and Incremental Reconstruction

AdaState: Self-Evolving Anchors for Streaming Video Generation

Steady-Forcing: Balancing Spatial Persistence and Motion Continuity in Long-Horizon Nature Video Diffusion