Stream4D: 4D-Consistency for Streaming Autoregressive Diffusion Video Models

Yuanhao Ban; Jiaqi Feng; Hengguang Zhou; Xiaohuan Pei; Justin Cui; Cho-Jui Hsieh

cs.CVAug 20, 2026

Stream4D: 4D-Consistency for Streaming Autoregressive Diffusion Video Models

Authors: Yuanhao Ban, Jiaqi Feng, Hengguang Zhou, Xiaohuan Pei, Justin Cui, Cho-Jui Hsieh

Organizations: UCLA · Tsinghua University

Abstract

Streaming autoregressive diffusion models enable real-time, long-horizon video generation, but their training objectives optimize local frame prediction rather than the geometry and dynamics of a coherent world: long rollouts accumulate geometric drift and degrade into static or unnatural motion. Recent bidirectional approaches address this problem using rewards signals built upon 3D Gaussian-Splatting reconstruction. However, a single rigid 3d reconstruction cannot model a dynamic scene, so this critic penalizes genuine object motion as reconstruction error and is maximized by freezing the video. This shortcut is especially detrimental in the AR setting, where each chunk can propagate an already-static configuration. In this work, we propose Stream4D, which replaces the static critic with a feed-forward 4D reconstruction reward that explicitly models scene dynamics, allowing coherent motion to receive high consistency rewards. To further guide motion magnitude and quality, we add a motion prior that rewards natural scene-flow magnitude while penalizing jitter and non-rigid artifacts. Our final recipe combines these two terms with a lightweight perceptual anchor. Across various autoregressive video backbones and various generation horizons, Stream4D improves 4D reconstruction quality, preserves motion more effectively, and achieves higher human-aligned preference. Project page: https://banyuanhao.github.io/Stream4D/

Figures & tables

Figure 1: Static-3D rewards freeze the scene; a 4D reward keeps it moving. Five uniformly-spaced frames from a $10.3$ s LongLive rollout (“a cat running away with a fish while people chase behind”). The distilled base contains large motion but the cat and fish drift across frames; the static-3DGS rewards World-R1 and VideoGPA reduce the scene to a more rigid, low-motion configuration. Stream4D keeps the cat running with clear forward motion while preserving a coherent subject and scene.

Figure 2: Unlike a static 3D reward, which punishes motion outright, our dynamic 4D-GS backbone does not penalize motion. Per-prompt 4D-PSNR only drifts weakly downward with motion magnitude $m$ .

Reference: full-diffusion T2V baselines (non-distilled, no RL)
Method	MoVieS Recon.			4DGT Recon.			LLM judge		VideoReward win%
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion $\uparrow$	Consist. $\uparrow$	VQ $\uparrow$	MQ $\uparrow$	TA $\uparrow$	Ovr $\uparrow$
Wan2.1-T2V-1.3B Wan et al. (2025)	19.15	0.828	0.258	17.14	0.614	0.399	—	—	—	—	—	—
CogVideoX-1.5-5B Yang et al. (2024)	17.02	0.768	0.279	15.06	0.518	0.454	—	—	—	—	—	—
Distilled AR backbone: Self-Forcing Huang et al. (2025) (5 s)
Base (B0)	16.88	0.785	0.282	16.28	0.559	0.419	—	—	—	—	—	—

Table 1: Main results on the 500 object-prominent testing set . Metrics as defined in Sec. 4.1 : MoVieS 4D reconstruction score; the 4DGT reconstruction Xu et al. (2025) score (a reconstructor-independent cross-check, Appendix D ); the order-debiased LLM judge evaluating Motion preservation and Consistency ); and VideoReward Liu et al. (2025b) paired win% vs. base.

Stream4D vs.	Human win% $\uparrow$		Gemini judge win% $\uparrow$
Stream4D vs.	Motion	Overall	Motion	Consist.
Base	$43$	$60$	$63$	$70$
World-R1 Wang et al. (2026a)	$72$	$76$	$93$	$68$
VideoGPA Du et al. (2026)	$87$	$80$	$95$	$82$

Table 2: Human study on LongLive. We focus the human evaluation on the long-horizon LongLive backbone, where accumulated drift and the static-collapse shortcut matter most. For each pair raters pick which video shows more natural motion and which is better overall (jointly weighing motion amount, motion quality, and object consistency); win% is Stream4D’s rate (ties count

\tfrac{1}{2}

{>}50\%

prefers ours). For reference we list the Gemini judge’s Motion and Consistency win% on the same

50

prompts: the judge agrees with humans on every ranking (Stream4D preferred over both reward baselines, and below

50\%

on raw motion vs. base), corroborating that the automatic scores are not an artifact, though the judge is somewhat more generous to ours in absolute terms.

Reward configuration	MoVieS Recon.			4DGT Recon.			LLM judge		VideoReward win%
Reward configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion $\uparrow$	Consist. $\uparrow$	VQ $\uparrow$	MQ $\uparrow$	TA $\uparrow$	Ovr $\uparrow$
Backbone: Self-Forcing (5 s)
Base (no RL)	16.88	0.785	0.282	16.28	0.559	0.419	—	—	—	—	—	—
Stream4D (Ours)	20.34	0.874	0.195	17.24	0.650	0.417	0.833	82.2	62.4	62.8	56.2	66.2
$-$ perceptual anchor (a)	22.39	0.905	0.163	18.53	0.723	0.371	0.830	81.7	66.4	64.8	52.8	66.6
$-$ motion term (b)	24.37	0.952	0.096	19.36	0.738	0.328	0.341	31.1	63.0	56.8	44.0	59.2

Table 3: Reward-axis ablation: dropping one of the three reward axes, on all three backbones. Each block lists the distilled base, the deployed Stream4D recipe, then the three single-axis drops. Metric conventions match Table 1 .

Figure 3: Reconstruction-motion trade-off . Deployed recipes (stars) sit on the upper-right frontier.

Motion-term $R_{\text{mot}}$	MoVieS Recon.			4DGT Recon.			LLM judge		VideoReward win%
Motion-term $R_{\text{mot}}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion $\uparrow$	Consist. $\uparrow$	VQ $\uparrow$	MQ $\uparrow$	TA $\uparrow$	Ovr $\uparrow$
Backbone: Self-Forcing (5 s)
Base (no RL)	16.88	0.785	0.282	16.28	0.559	0.419	—	—	—	—	—	—
$g(m)\!\cdot\!\mathrm{smooth}\!\cdot\!\mathrm{rigid}$	20.34	0.874	0.195	17.24	0.650	0.417	0.833	82.2	62.4	62.8	56.2	66.2
$\mathrm{smooth}\!\cdot\!\mathrm{rigid}$ ,	23.43	0.947	0.119	18.02	0.671	0.405	0.406	36.4	78.4	60.4	53.8	74.6
$m\!\cdot\!\mathrm{smooth}\!\cdot\!\mathrm{rigid}$ (e)	18.46	0.833	0.234	16.85	0.632	0.392	0.923	71.2	46.4	47.4	41.8	46.0

Table 4: Motion-term formulation ablation: varying the shape of the motion term

R_{\text{mot}}

while keeping the reconstruction and perceptual axes fixed, on all three backbones.

$m_{\text{nat}}$	MoVieS Recon.			4DGT Recon.			LLM judge		VideoReward win% vs base
$m_{\text{nat}}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion $\uparrow$	Consist. $\uparrow$	VQ $\uparrow$	MQ $\uparrow$	TA $\uparrow$	Ovr $\uparrow$
Self-Forcing base	16.88	0.785	0.282	16.28	0.559	0.419	—	—	—	—	—	—
$0.010$ ( $0.5\times$ )	21.10	0.924	0.135	17.86	0.672	0.345	0.522	44.7	62.8	59.2	65.6	63.8
$\mathbf{0.020}$ (deployed)	20.34	0.874	0.195	17.24	0.650	0.417	0.833	82.2	62.4	62.8	56.2	66.2
$0.030$ ( $1.5\times$ )	18.87	0.836	0.228	17.50	0.645	0.374	0.916	74.4	54.0	51.0	45.6	53.2

Table 5: Sensitivity to the gate target

m_{\text{nat}}

on Self-Forcing. The

0.020

row is the deployed recipe, landing on the knee of the motion-fidelity trade-off, taking the best Consistency and VideoReward-Overall.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Hyperparameter	Value
Backbone	Architecture	Causal Wan 2.1 (Self-Forcing / Causal-Forcing / LongLive)
Backbone	Video resolution ( $H\times W$ )	$480\times 832$
LoRA fine-tuning	Rank $r$ / scaling $\alpha$	256 / 256
	Dropout	0.0
	Gradient checkpointing	True
Optimization	Hardware	4 nodes $\times$ 8 $\times$ H200, DDP

Appendix

Table S1 : Stream4D training hyperparameters. A single training-hyperparameter set is used across Self-Forcing, Causal-Forcing, and LongLive; only the three reward-axis weights vary per backbone.

Distilled AR backbone: Self-Forcing Huang et al. (2025) (5 s)
Method	MoVieS Recon.			4DGT Recon.			LLM judge		VideoReward win%
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion $\uparrow$	Consist. $\uparrow$	VQ $\uparrow$	MQ $\uparrow$	TA $\uparrow$	Ovr $\uparrow$
Base (B0)	18.19	0.831	0.239	16.23	0.546	0.439	—	—	—	—	—	—
World-R1 Wang et al. (2026a)	19.35	0.885	0.182	16.60	0.560	0.412	0.773	68.2	61.4	49.4	54.2	56.8
VideoGPA Du et al. (2026) ‡†	24.88	0.908	0.143	19.87	0.621	0.396	0.477	38.3	63.0	63.4	65.0	67.4
$+$ Stream4D (Ours)	21.23	0.895	0.177	17.70	0.657	0.411	0.816	75.1	59.8	56.6	54.4	59.8

Appendix

Table S2: Robustness check: main comparison on a uniformly random 500-prompt subset of the same held-out VidProM set (seed-0 sample, no motion filter;

122

prompts overlap the high-motion subset of Table 1 ). Metrics and conventions as in Table 1 ( 4DGT : reconstructor-independent dynamic-reconstruction cross-check, Appendix D ); all RL’d rows at ckpt-150, judged against the same backbone’s base. ‡ The VideoGPA row uses the standard VideoGPA recipe (the best-weight variant of Table 1 was generated only on the high-motion subset); on this unfiltered, low-motion-dominated subset its clean freeze sweeps both reconstructors. † marks methods whose Consistency win-rate falls below the

50\,\%

break-even against their own base: on this low-motion-dominated subset, their reconstruction and VideoReward advantages are bought by suppressing motion (motion scores

0.14

–

0.48

), not by generating better video. Bold = best RL’d row per backbone block; shaded = ours.

Panel of Figure 11 — Figure S1 : Motion preserved vs. frozen: the static-reward shortcut, one prompt per backbone. Top: Causal-Forcing (a WW3 battle); middle: Self-Forcing (a futuristic highway); bottom: LongLive (fish in a river). Three uniformly-spaced frames per method, all RL’d rows at ckpt-150; read each row across time. The static-3DGS baselines (VideoGPA, World-R1) are frozen—subjects hold the same pose from $t{=}0$ to $t{=}1$ —while the distilled bases move but deform, and Stream4D keeps the motion while holding object identity and scene structure. Table 1 quantifies motion preservation over the full prompt set.

Figure S2 : Object-identity under motion: the base breaks the object apart, the static-reward baselines freeze the scene, and only ours keeps a moving object intact.

Figure S3 : Consistency companion on Self-Forcing: an ancient-battlefield beast Red outlines mark failure frames and the coloured badge names each row’s behaviour. The failures are complementary: the base moves but morphs ; the static-reward baselines are either frozen . Stream4D stays coherent : one beast, one identity, approaching the camera.

Figure S4 : Consistency companion on LongLive: a planet descending over a city The base deforms into spreading debris/fire; VideoGPA’s explosion flickers in and out over an otherwise static scene; World-R1 is frozen . Stream4D keeps a coherent descending planet over a stable city.

Streaming Video Generation

Streaming video generation creates video incrementally as frames or chunks arrive, enabling low-latency interactive synthesis and editing.

See all 98 papers

Sep 1, 2026

Streaming4D: Accelerate 4D World Models via Block-wise Video Generation and Incremental Reconstruction

Current 4D generation paradigms are often bottlenecked by a sequential decoupling design: video is generated first, followed by 3D reconstruction, leading to high interaction latency. This limits applications in interactive real-time scenarios. To this end, we propose Streaming4D, a tightly coupled synchronous pipeline that integrates block-wise autoregressive video generation with incremental 3D reconstruction. Unlike traditional frame-by-frame emission and delayed geometry recovery, Streaming4D generates temporal video blocks and immediately triggers reconstruction for each completed block, enabling parallel execution between synthesis and geometric updates. This approach allows the world representation to evolve online with the video stream, reducing feedback latency while preserving geometric fidelity. We instantiate Streaming4D using a Self-Forcing-style autoregressive generator and an incremental reconstruction backend. Experiments show consistent runtime improvements across resolutions on a single RTX 4090 (1.24

\times

speedup), while maintaining high-quality 4D geometry and multi-view consistency.

Xiaoyan Liu, Jiaxin Liu, Kangrui Li +1

The Chinese University of Hong Kong · The Hong Kong Polytechnic University · The University of New South Wales +1

4D Reconstruction 3D Reconstruction Streaming Video Generation Dynamic Scene Reconstruction 4D Scene Generation Video World Models

Ask PDF

May 28, 2026

AdaState: Self-Evolving Anchors for Streaming Video Generation

Autoregressive video diffusion models generate streaming video by producing frames sequentially, conditioning each chunk on previously generated content. These models are structurally anchored to the first frame: its key-value representation occupies a privileged position in the attention cache and serves as the primary scene reference throughout generation. As the cleanest and most error-free position in the cache, this anchor draws disproportionate attention, suppressing video dynamics, and locking scene composition to the initial viewpoint even as the scene naturally evolves. The result is a temporally shallow video in which motion, camera movement, and scene progression are dampened in favor of static consistency. To address this, we replace the static anchor with an adaptive state, a hidden latent that the model denoises alongside content at every chunk but never renders. Rather than referencing a frozen first frame, the model generates its own scene anchor at each step by attending to both the previous state and the current content, producing a reference that evolves with the generated content. Unlike standard video generation, which encodes an absolute notion of time, our formulation treats time as relative: every generation step sees the same positional structure regardless of how far generation has progressed, and the state transition is identical at every chunk. Together, these properties introduce a recurrence into the generation process, where denoising serves as the transition function, and the KV cache serves as the carrier, requiring no external module. Experiments demonstrate that the adaptive state substantially improves video dynamics, enabling richer motion and natural scene progression within generated videos.

Yusuf Dalva, Pinar Yanardag

Virginia Tech

Video Diffusion Models Streaming Video Generation Video Generation Autoregressive Video Generation Temporal Consistency in Video Generation

Ask PDF

Jun 2, 2026

Steady-Forcing: Balancing Spatial Persistence and Motion Continuity in Long-Horizon Nature Video Diffusion

Autoregressive video diffusion models enable streaming generation but often degrade over long rollouts: static scene layouts drift, while mechanisms that improve spatial stability tend to suppress motion, causing natural flows such as water, fire, or smoke to stagnate. We study this stability-motion trade-off in fixed-camera long-horizon nature video generation, where the two failure modes can be more clearly separated than in moving-camera settings. We propose Steady-Forcing, a memory and training framework combining a persistent visual anchor (V-Sink), an exponential moving-average motion memory (EMA-Sink), block-relative temporal encoding, periodic cache purification, and distillation from a Wan2.1-14B teacher with motion-rewarded priors under task-focused configurations. Together, these components are designed to preserve background identity while sustaining visually plausible fluid dynamics over multi-minute autoregressive rollouts. Evaluations across seven baselines show that Steady-Forcing improves long horizon background consistency and imaging quality, while a blind user study indicates stronger perceived stability and motion continuity. The benchmark evaluation further suggest that generic VBench aggregate scores under-penalize fixed-camera artifacts as well as rewarding drift-induced optical flow as Dynamic Degree while not directly penalizing texture hardening or flow stagnation - motivating future task-specific benchmarks for static-camera nature-flow evaluation. Project page: https://minar09.github.io/steadyforcing/

Matiur Rahman Minar, Seunghun Oh, GangHyeon Jeong +1

Department of Computer Science and Engineering Sogang University Seoul, Korea · Department of Artifical Intelligence Sogang University Seoul, Korea

Physical Consistency in Video Generation Long-Horizon Video Generation Memory-Augmented Video Generation Video Generation Autoregressive Video Generation Diffusion Models for Motion Generation+1

Ask PDF

Abstract

Streaming Video Generation

Explore similar work

Streaming4D: Accelerate 4D World Models via Block-wise Video Generation and Incremental Reconstruction

AdaState: Self-Evolving Anchors for Streaming Video Generation

Steady-Forcing: Balancing Spatial Persistence and Motion Continuity in Long-Horizon Nature Video Diffusion