RealWonder: Real-Time Physical Action-Conditioned Video Generation

Wei Liu; Ziyu Chen; Zizhang Li; Yue Wang; Hong-Xing Yu; Jiajun Wu

cs.CVMar 5, 2026

RealWonder: Real-Time Physical Action-Conditioned Video Generation

Authors: Wei Liu, Ziyu Chen, Zizhang Li, Yue Wang, Hong-Xing Yu, Jiajun Wu

Organizations: Stanford University · University of Southern California

Abstract

Current video generation models cannot simulate physical consequences of 3D actions like forces and robotic manipulations, as they lack structural understanding of how actions affect 3D scenes. We present RealWonder, the first real-time system for action-conditioned video generation from a single image. Our key insight is using physics simulation as an intermediate bridge: instead of directly encoding continuous actions, we translate them through physics simulation into visual representations (optical flow and RGB) that video models can process. RealWonder integrates three components: 3D reconstruction from single images, physics simulation, and a distilled video generator requiring only 4 diffusion steps. Our system achieves 13.2 FPS at 480x832 resolution, enabling interactive exploration of forces, robot actions, and camera controls on rigid objects, deformable bodies, fluids, and granular materials. We envision RealWonder opens new opportunities to apply video models in immersive experiences, AR/VR, and robot learning. Our code and model weights are publicly available in our project website: https://liuwei283.github.io/RealWonder/

Figures & tables

Figure 2: Overview of RealWonder. ( Left ) Given a single image and a sequence of actions as input, we first reconstruct the 3D scene as point clouds, ( Middle ) estimate material for the objects to interact with, and then maintain a physics simulation stream using the actions. ( Right ) Meanwhile, we maintain another stream of rendering optical flow and RGB preview to condition a few-step conditional video generator, producing the physical action-conditioned video streaming.

Figure 3: Training of Real-time Flow-Conditioned Video Model. A pretrained image-to-video model [ 57 ] is first adapted to optical-flow conditioning through LoRA [ 22 ] post-training. Next, it is distilled via distribution-matching training into a causal, flow-conditioned real-time video generator.

Figure 4: Qualitative results . In the left column we show the input scene image and initial actions, where the arrow indicates 3D force and the blue wind icon indicates force fields. Note that we always apply gravity in our simulation.

Figure 5: Comparison to baselines . In the first row, we show the input images and actions (we use arrows for 3D point forces and wind icon for force fields) together with the text prompts. We always apply gravity in our simulation.

Methods	Visuals ( $\uparrow$ )	Aesthetics ( $\uparrow$ )	Consistency ( $\uparrow$ )	PhysReal ( $\uparrow$ )
PhysGaussian	0.454	0.517	0.221	0.468
CogVideoX	0.696	0.603	0.234	0.624
Tora	0.700	0.588	0.223	0.578
RealWonder (ours)	0.708	0.593	0.265	0.705

Table 1: Quantitative comparison to baselines.

	Action Following	Motion Fidelity	Visual Quality	Physical Plausibility
over PhysGaussian [ 64 ]	88.4%	82.0%	88.6%	87.1%
over CogVideoX-I2V [ 67 ]	89.6%	71.0%	75.3%	85.9%
over Tora [ 75 ]	83.9%	67.9%	75.4%	79.7%

Table 2: 2AFC human study results of the favor rate of our RealWonder over baseline methods. The responses are collected from 400 participants.

	Tora	CogVideoX-I2V	PhysGaussian	RealWonder (Ours)
FPS	0.107	0.225	0.207	13.2
Latency	-	-	4.84s	0.73s

Table 3: Comparison of runtime performance.

Figure 6: Different actions on the same scene, leading to different generated physical outcomes.

Figure 7: Ablation on physics simulator. Top : RealWonder without the physics simulator (conditioned only on input image and text, where we specify a wind blowing from the right in the text). Bottom : Our full model with wind (3D force field) as action.

Figure 8: Ablation on the flow condition and RGB preview condition. We apply a 3D point force upward.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Consistent performance under reconstruction errors

Figure S2 : Left: video generator fixes ambient dynamics (i.e., the dynamics on the river). Right: video generator refines deformable object motion.

Methods	Imaging ( $\uparrow$ )	Aesthetic ( $\uparrow$ )	Consistency ( $\uparrow$ )	PhysReal ( $\uparrow$ )
Teacher	0.713	0.605	0.271	0.698
Student	0.708	0.593	0.265	0.705

Appendix

Table S1 : Quantitative comparison with teacher model

Figure S3 : Additional cases across diverse materials. In the left column we show the input scene image and initial actions, where the arrow indicates 3D force, blue wind icon indicates force fields and robot icon indicates robot actions. Gravity is applied.

Figure S4 : Comparison to baselines on long streaming video generation . We use "N/A" to denote frames that these methods cannot generate at later timestamps. We strongly encourage the reader to see video results in our website.

Figure S5 : Comparison to baselines on long streaming video generation . We use "N/A" to denote frames that these methods cannot generate at later timestamps. We strongly encourage the reader to see video results in our website.

Parameter	Default Value
General simulation
Step time	$1e^{-2}$
Sub-steps number	$10$
Sampled particle size	$1e^{-2}$
Gravity	$(0,0,-9.8)$
Rigid body solver

Appendix

Table S2 : Simulation parameters and default values

Action-Conditioned Video Generation

Action-conditioned video generation creates videos whose motion and scene changes are guided by controls, actions, or interaction signals.

See all 103 papers

Wonder: Video World Model Done Better
Jul 28, 2026Jiacong Xu, Hanwen Jiang, Zhixin Shu +3Camera-Controlled Video Generation Real-Time Interactive Video Generation
Adobe Research · Johns Hopkins University
TelePhysics: Physics-Grounded Multi-Object Scene Generation from a Single Image with Real-Time Interaction
May 19, 2026Xin Zhang, Yabo Chen, Yijie Fang +5Physical Consistency in Video Generation Real-Time Interactive Video Generation
Fudan University · Institute of Artificial Intelligence, China Telecom (TeleAI)
Fluid-Gen-Zero: Grounding Pretrained Video Generators in Physics without Training
Oct 7, 2026Hong Huang, Yuqiu Liu, Chenyu You +3Video Generation Physical Consistency in Video Generation
Simon Fraser University · Stony Brook University · Lawrence Berkeley National Laboratory +1

Jul 28, 2026J/K move · Enter open · S save

Wonder: Video World Model Done Better

Jiacong Xu, Hanwen Jiang, Zhixin Shu, Kalyan Sunkavalli+2

Adobe Research · Johns Hopkins University

Camera-Controlled Video Generation Real-Time Interactive Video Generation Long-Horizon Video Generation Memory-Augmented Video Generation Video World Models

Open paper AskPDF arXiv

We present Wonder, a general-purpose video world model for real-time, camera-controllable world exploration. Given an image or a conditional video, Wonder constructs a playable world where users can navigate interactively by moving the camera, discovering unseen regions, and revisiting previously observed areas in real time and over a long-term horizon. Achieving this capability requires a system-level co-design of control method, memory mechanism, and training strategy. We introduce a novel camera conditioning with a dense coordinate field whose renderings provide spatially aligned motion and orientation cues, allowing the model to interpret camera motion directly as visual evidence. To support fast and precise memory retrieval over a growing generation context, we propose an efficient sparse attention-based memory mechanism, enabling the model to selectively attend to a small set of relevant context tokens at inference time, regardless of actual context length. We further develop several techniques to rectify the self-forcing-style distillation pipeline, improving the student model's ability to respect control signals, as well as maintaining diverse generation modes and long-term memory from the teacher. Together, these components enable Wonder to synthesize diverse, minute-scale videos at 16 FPS while preserving coherent geometry, appearance, and dynamics across long rollouts. Beyond image-to-video generation, Wonder naturally supports video-conditioned generation, allowing existing dynamic scenes to be re-shot in real time.

Abstract

Action-Conditioned Video Generation

Explore similar work

Wonder: Video World Model Done Better

TelePhysics: Physics-Grounded Multi-Object Scene Generation from a Single Image with Real-Time Interaction

Fluid-Gen-Zero: Grounding Pretrained Video Generators in Physics without Training