Agentic Physics-Grounded Video Generation via Memory-Augmented In-Context Reasoning
Authors: Xikang Guan, Shaocong Yang, Yanqi Bao, Can Wang, Zheng Gu, Zhengqing Chen, Kai Chen, Jing Huo, Yang Gao
Organizations: State Key Laboratory for Novel Software Technology, Nanjing University · Horizon Robotics · School of Computing and Data Science, The University of Hong Kong · College of Computer Science and Software Engineering, Shenzhen University
Physical plausibility is essential for realistic video generation. Existing methods impose physical guidance before synthesis via simulations or refined prompts, but this can limit dynamic diversity, miss errors that emerge only after generation, and require repeated regeneration when inconsistencies remain. We therefore study a generate-diagnose-repair paradigm that preserves the original generation capability and selectively repairs physical inconsistencies afterward, raising two key questions: what to repair and how to repair? To address them, we introduce PhysRepair, an agentic video generation framework that combines causal reasoning over the observed physical failure with prior repair experience to enable localized and adaptive correction. Specifically, Spatiotemporal Causal Constraints identify the first invalid physical state and its causally affected states, producing a repair specification that guides physically plausible local regeneration while preserving unaffected content. Experience-augmented In-context Learning combines the ongoing repair context with relevant experience from previous repairs, allowing the agent to adapt its repair decisions across diverse physical failures and accumulate completed trajectories for future cases. Experiments on Physics-IQ Verified across solid mechanics, fluid dynamics, optics, thermodynamics, and magnetism demonstrate that PhysRepair consistently improves the physical plausibility of generated videos over existing methods. Our project page is available at https://xxkkxxx.github.io/PhysRepair-Website/.
Figures & tables
Figure 1: Comparison of physics-grounded video generation paradigms. (a) Plan and Prompt Refine iteratively updates generation plans or physics-aware prompts, while (b) Simulator Program Refine iteratively updates physical programs or simulator configurations; both impose physical guidance before synthesis and may require repeated regeneration. (c) PhysRepair instead follows a generate–diagnose–repair paradigm: it diagnoses physical failures from the generated video (what to repair?) and performs localized, causally constrained repair, while leveraging prior repair experience to adapt the intervention and preserve generated already-valid content (how to repair?).
Figure 2: Overview of PhysRepair. Starting from an initially generated video, a Physics Repair Agent diagnoses the realized physical failure and performs localized counterfactual repair. Spatiotemporal causal constraints determine what may change and what must remain fixed , while experience-augmented in-context learning retrieves relevant repair experience to adapt how to repair . The resulting counterfactual physical state guides video regeneration, followed by physics-and-preservation verification and experience distillation for subsequent repair.
Method
SP ↑
ST ↑
WS ↑
MSE score ↑
Physics-IQ ↑
Video generation models
MAGI-1 4.5B ( Sand AI, 2025 )
14.3
13.3
7.0
–
18.8
Cosmos-Predict2-2B ( NVIDIA, 2025 )
–
–
–
–
27.7
Wan2.2-TI2V-5B ( Wan Team, 2025 )
35.9
26.4
23.0
25.5
27.7
CogVideoX-I2V-5B ( Yang et al., 2024 )
37.8
35.5
21.8
32.0
31.8
Wan2.2-I2V-14B ( Wan Team, 2025 )
51.1
20.5
28.5
28.9
32.2
Table 1: Quantitative results on Physics-IQ Verified. SP, ST, and WS denote Spatial IoU, Spatiotemporal IoU, and Weighted Spatial IoU, respectively. The best, second-best, and third-best results for each metric are marked in bold , single underline , and double underline , respectively.
Figure 4: Qualitative ablation of Spatiotemporal Causal Constraints. Without causal constraints, repair may correct the immediate target while leaving causally affected states unresolved or disrupting valid interactions. Full PhysRepair jointly constrains editable and preserved states across the interaction, enabling complete and localized physical repair.
Figure 5: Analysis of PhysRepair. Memory efficiency and user-perceived physical consistency of PhysRepair.
Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.
Recent advances in physics-grounded video generation leverage physics simulation as a physical prior to guide video synthesis toward physically plausible outcomes. The simulation process is controlled by physical specifications, which are typically generated by a vision-language model in a single pass. Such one-shot prediction often fails to accurately translate user intent into executable simulations, particularly for fine-grained object dynamics, complex motion trajectories, and temporally structured interactions. In this paper, we propose PhysAgent, a reflective agentic framework that closes the loop among physical program generation, physics simulation, stage-specific verification, and targeted program repair. Beyond improving the control of coupled physical parameters, our framework enables the agent to progressively realize complex trajectories, multi-stage interactions, and precise event outcomes by treating each physical program as an executable hypothesis. In addition, we design a set of physics-control APIs to support more stable and complex motion behaviors. Extensive experiments demonstrate that PhysAgent produces more physically plausible videos, achieves better prompt alignment, and generalizes more effectively across diverse physical scenarios.
Qirui Li, Jinkun Hao, Yibo Li +3
Shanghai Jiao Tong University · Cardiff University
Generative video models achieve high visual fidelity but often violate basic physical principles, limiting reliability in real-world settings. Prior attempts to inject physics rely on conditioning: frame-level signals are domain-specific and short-horizon, while global text prompts are coarse and noisy, missing fine-grained dynamics. We present PhysVid, a physics-aware local conditioning scheme that operates over temporally contiguous chunks of frames. Each chunk is annotated with physics-grounded descriptions of states, interactions, and constraints, which are fused with the global prompt via chunk-aware cross-attention during training. At inference, we introduce negative physics prompts (descriptions of locally relevant law violations) to steer generation away from implausible trajectories. On VideoPhy, PhysVid improves physical commonsense scores by ≈33% over baseline video generators, and by up to ≈8% on VideoPhy2. These results show that local, physics-aware guidance substantially increases physical plausibility in generative video and marks a step toward physics-grounded video models.
Video generation models produce visually compelling results but systematically violate physical commonsense -- on VideoPhy-2, the best model achieves only 32.6% joint accuracy. We identify a specification bottleneck: text prompts are lossy compression of the physical world, omitting the parameters that fully determine dynamics, and no amount of model scaling can recover what was never specified. From this diagnosis we derive three properties that physics conditioning must satisfy -- sufficiency, dynamism, and verifiability -- and show that no existing approach satisfies all three. We present NEWTON, in which video generation is demoted from the system output to one action inside an agent's toolbox: a learned planner orchestrates physics-aware tools (keyframe generation, scientific computation, prompt refinement) to construct rich conditioning, and a verifier closes the loop for iterative re-planning. The planner is the sole trainable component, optimized on-policy via Flow-GRPO inside the live multi-turn loop. On VideoPhy-2, NEWTON improves joint accuracy from 21.4% to 29.7% on LTX-Video and from 30.7% to 37.4% on Veo-3.1, without modifying either generator. Our project page: https://Newton026.github.io/newton
Yuxiang Feng, Juncheng Wang, Chao Xu +7
Zhejiang University · The Hong Kong Polytechnic University · IROOTECH TECHNOLOGY +1