4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

Ruihong Shen; Žiga Kovačič; Peter Kulits; Xingrui Wang; Zizhang Li; Joshua B. Tenenbaum; Alan Yuille; Jieneng Chen

cs.CVOct 2, 2026

4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

Authors: Ruihong Shen, Žiga Kovačič, Peter Kulits, Xingrui Wang, Zizhang Li, Joshua B. Tenenbaum, Alan Yuille, Jieneng Chen, +1 more

Organizations: Johns Hopkins University · Stanford University · Max Planck Institute for Intelligent Systems · Massachusetts Institute of Technology

Abstract

We introduce 4DCodeBench, a benchmark for 4D inverse graphics through code generation, in which agents reconstruct dynamic scenes from video as executable graphics programs. To accomplish this, agents must translate visual observations into compact representations of scene structure and dynamics, by implementing abstractions such as physical simulations to reproduce complex behavior. To evaluate this capability, we curate a set of real-world videos and construct synthetic scenes spanning diverse physical phenomena, including deformation, fluid flow, and fracture. We perform extensive benchmarking of frontier models, finding that strong static reconstruction capabilities do not yet translate into reliable reconstruction of complex dynamics. 4DCodeBench provides a testbed for tracking progress toward agents that can interpret the dynamics of the world through code. Our benchmark is available at https://github.com/4DCodeBench/4DCodeBench

Explore similar work

CP4D: Compositional Physics-aware 4D Scene Generation
Jun 8, 2026Hanxin Zhu, Cong Wang, Tianyu He +4Physical Consistency in Video Generation 3D Scene Generation
School of Information Science and Technology, University of Science and Technology of China · Zhongguancun Academy · the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences. +2
Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes
Sep 29, 2026Xiaokang Ye, Siddhant Hitesh Mantri, Zimeng Chen +6AI Coding Agents 3D Scene Editing
UC San Diego · UC Berkeley
Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
Oct 3, 2026Jinzhou Tang, Zijun Zhang, Jing Yang +10Interactive World Models Coding Agents
Aether AI · University of California, San Diego

Jun 8, 2026, cs.CVJ/K move · Enter open · S save

CP4D: Compositional Physics-aware 4D Scene Generation

Hanxin Zhu, Cong Wang, Tianyu He, Long Chen+3

School of Information Science and Technology, University of Science and Technology of China · Zhongguancun Academy · the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences.+2

Physical Consistency in Video Generation 3D Scene Generation Functional 3D Scene Generation 4D Scene Generation

Open paper AskPDF arXiv

4D generation (\textit{i.e.}, dynamic 3D generation) has recently emerged as a rapidly growing research frontier due to its powerful spatiotemporal modeling capabilities. However, despite notable advances, existing approaches typically fail to capture the underlying physical principles, producing results that are both physically inconsistent and visually implausible. To overcome this limitation, we present CP4D, a novel paradigm for photorealistic 4D scene synthesis with faithful adherence to complex physical dynamics. Drawing inspiration from the compositional nature of real-world scenes, where immutable static backgrounds coexist with dynamic, physically plausible foregrounds, CP4D reformulates 4D generation as the integration of a static 3D environment with physically grounded dynamic objects. On this basis, our framework follows a three-stage pipeline: \textbf{1)} Firstly, we leverage pre-trained expert models to generate high-fidelity 3D representations of the environment and foreground objects respectively. \textbf{2)} Subsequently, to produce physically plausible trajectories and realistic interactions for these objects, we propose a hybrid motion synthesis strategy that integrates priors from physical simulators with the common sense embedded in video diffusion models. \textbf{3)} Finally, we develop an automated composition mechanism that seamlessly fuses the static environment and dynamic objects into coherent, physically consistent 4D scenes. Extensive experiments demonstrate that CP4D can generate explorable and interactive 4D scenes with high visual fidelity, strong physical plausibility, and fine-grained controllability, significantly outperforming existing methods. The project page: https://anonymous.4open.science/w/CP4D/.

Abstract

Explore similar work

CP4D: Compositional Physics-aware 4D Scene Generation

Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes

Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos