Organizations: Nanjing University · KlingAI Research · Peking University · Hong Kong University of Science and Technology (Guangzhou) · University of Hong Kong · Hong Kong University of Science and Technology · Tsinghua University
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Figures & tables
Figure 1: Selective 3D modeling. Each method pairs 3D proxy frames with generated video frames at 3, 5, and 7 seconds. Codex + Blender and VideoCoCo explicitly model hands and arms. Coda-DaVinci models the objects involved in cutting and unwrapping, while leaving detailed hand articulation to the video generator.
Figure 2: Qualitative visualization of component configurations. Within each labeled group, the upper sequence shows the proxy and the lower sequence shows the corresponding generated video; columns follow the same eight sampled timestamps. The groups labeled P0 and P1 correspond to G1_P1_PE1 and G0_P1_PE0 , respectively. Both enable explicit physics simulation and differ in geometry and prompt enhancement.
Figure 3: Effects of proxy granularity, physics simulation, and prompt enhancement on three benchmarks. Each bar shows the mean of the four human evaluation scores (C/I/P/N) reported for one configuration on one benchmark in Table 1 . Higher scores are better. From left to right, the groups vary granularity at fixed P1/PE0 , physics simulation at fixed G1/PE0 , and prompt enhancement at fixed G1/P1 . All three groups share the reference configuration G1/P1/PE0 .
WorldMemBench
VGIF-Bench
Principia
Average
Configuration
C ↑
I ↑
P ↑
N ↑
C ↑
I ↑
P ↑
N ↑
C ↑
I ↑
P ↑
N ↑
C ↑
I ↑
P ↑
N ↑
Direct generation
2.182
1.242
2.485
2.333
2.483
1.828
2.207
2.000
2.562
2.406
2.000
2.156
2.404
1.819
2.234
2.170
G0/P0/PE0
2.758
3.000
2.606
2.455
2.429
2.276
1.897
1.414
2.969
2.938
2.562
2.594
2.731
2.755
2.372
2.181
G1/P0/PE0
2.606
3.000
2.576
2.455
2.690
2.207
1.448
1.034
2.969
3.000
2.656
2.719
2.755
2.755
2.255
2.106
G2/P0/PE0
2.788
3.000
2.606
2.455
2.552
2.069
1.448
1.000
2.906
2.875
2.531
2.562
2.755
2.670
2.223
2.043
G0/P1/PE0
2.606
2.970
2.758
2.697
2.448
2.276
1.759
1.345
2.969
2.969
2.531
2.625
2.681
2.755
2.372
2.255
Table 1: Human evaluation on three benchmarks and per-metric averages over their pooled samples. C/I/P/N denote consistency, instruction following, physical plausibility, and naturalness and visual realism, respectively.
Comparison
Config.
C
I
P
N
Mean
P1-P0
G0/PE0
-0.054
0.000
0.000
+0.075
+0.005
G1/PE0
+0.043
0.000
+0.160
+0.128
+0.082
G2/PE0
-0.043
+0.074
+0.096
+0.011
+0.035
G1/PE1
+0.200
+0.250
+0.062
+0.012
+0.131
G1/PE2
+0.149
+0.191
+0.223
+0.096
+0.165
Table 2: Changes in per-metric averages over pooled samples from three benchmarks ( P1 minus P0 ); Mean: C/I/P/N average.
Comparison
Config.
C
I
P
N
Mean
PE2-PE1
G1/P0
+0.468
+0.224
+0.043
+0.149
+0.221
G1/P1
+0.364
+0.131
+0.261
+0.263
+0.255
PE2-PE0
G1/P0
+0.022
+0.022
-0.042
+0.011
+0.003
G1/P1
+0.128
+0.213
+0.021
-0.021
+0.085
Table 3: Changes in per-metric averages over pooled samples from three benchmarks using PE2 at fixed G1 ; Mean: C/I/P/N average.
Figure 4: Prompt enhancement under two physics settings at fixed proxy granularity ( G1 ). Each point is the unweighted mean of the four human evaluation scores (C/I/P/N) reported in Table 1 for one benchmark and configuration. Dashed lines with open circles denote optional physics simulation ( P0 ), and solid lines with diamonds denote required physics simulation ( P1 ).
Figure 5: Overview of Coda-DaVinci. Coda-DaVinci combines task-adaptive proxy construction with proxy-grounded video synthesis. The harness maintains the working scene and supports visual feedback throughout generation.
Last Exam
R2M
VTR
T2AV-Compass
Method
Score ↑
Scene ID ↑
Local Geo. ↑
State ↑
Object ↑
Checklist ↑
WER ↓
VR ↑
IF ↑
Direct generation
47.16
0.347
0.351
0.159
0.343
85.15
59.21
71.38
65.11
Agentic Video Generation
Codex
53.18
0.575
0.583
0.520
0.587
89.72
25.73
72.21
68.05
Claude
51.95
0.594
0.590
0.523
0.522
87.07
53.62
71.50
68.22
Codex + VideoWeaver
55.11
0.739
0.705
0.646
0.665
90.88
23.58
71.83
67.60
Table 4: Main results across four benchmarks. R2M reports normalized memory ratios (NMR). WER: word error rate; VR: video realism; IF: visual instruction following. Green cells mark the best result in each column.
Agent
P
I
C
A&F
Mean
Seedance 2.5
Direct generation
3.242
3.364
3.758
3.667
3.508
GPT-6-Astra
3.455
3.758
3.879
3.758
3.712
GPT-6.1-Sol
3.394
3.545
3.818
3.667
3.606
Claude Opus 5.5
3.548
3.742
3.839
3.645
3.694
Gemini 4 Argon
3.562
3.719
3.656
3.656
3.648
Table 5: Ablations on the test subset. P: physics; I: instruction following; C: consistency; A&F: aesthetics and fluency. Green marks column maxima (including ties) per video model.
Figure 6: Performance and ablation studies. (a) Normalized benchmark results. (b) Ablations of adaptive routing, independent review, and modeling granularity. (c) Video quality and token cost across rendering engines. (d) On LongAV-Compass, Coda-DaVinci improves event fulfillment and long-form structure over direct generation, with lower transition stability. (e) On WBench, it improves interaction adherence and consistency, with lower setting adherence.
Figure 7: Camera instruction following and state persistence. The camera turns away while leaves are being cleared, then returns to the steps. Each method occupies two rows, read consecutively from left to right. Frames are sampled at matching timestamps, with denser sampling during the return. Both code2video methods show cleared steps on revisitation, while Seedance retains substantial leaf coverage.
Figure 8: Hand articulation and visual fluency. Consecutive frames from a dough-flattening task, sampled at 4 fps over 4.50–8.25 seconds. Each method occupies two rows, with the second continuing the first. Compared with the nearly fixed pose of Codex + Blender, Coda-DaVinci shows changing wrist and finger poses through repeated contact with the dough. Seedance provides the direct-generation comparison.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Scene geometry during navigation. R2M-Bench case WSLRRLL_0046 . The portal, curved roof, and surrounding station provide landmarks for comparing scene structure through the camera movement. Proxy renders and generated frames reveal how each modeled scene carries through synthesis.
Figure 10: Scene geometry during navigation (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 11: Preserving an edited scene state. WBench case 248. The monitor goes black before the player stands up and turns around. Direct generation brings back the game display in later turns, while Coda-DaVinci preserves the black-screen state in both the proxy and the generated video.
Figure 12: Scene consistency across successive views. WBench case 92. The camera moves through an autumn park. Coda-DaVinci retains the wooden bench across views; the direct-generation sequence introduces a bench with a different armrest structure. Corresponding proxy frames show the shared scene used for continuation.
Figure 13: Coordinating objects and interactions. VTR-Bench case CULT-0034 . Two friends compare a set of written notes and a tracklist before playing a cassette. Coda-DaVinci represents the tabletop objects and subject identities in the proxy, while the final video supplies natural hand movements and human appearance.
Figure 14: Coordinating objects and interactions (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 15: Preserving task-critical object structure. VTR-Bench case AD-0059 . The task involves adjusting a split keyboard, inspecting its layout diagram, and showing its engraved badge. The proxy specifies the keyboard components and camera views, which guide the corresponding actions in the generated video.
Figure 16: Preserving task-critical object structure (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 17: Following actions within a continuous shot. VTR-Bench case LIFE-0008 . The owner hands over a leash and explains the care instructions before the neighbor hides a key under the frog-shaped planter. Coda-DaVinci keeps these actions within the requested wide view, preserving the room layout throughout.
Figure 18: Following actions within a continuous shot (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 19: Deformation with natural hand motion. T2AV-Compass case 0348 . The proxy controls the dough flattening beneath the paper doilies. Coda-DaVinci leaves the hands to the video model, producing the requested manipulation around the changing dough shape.
Figure 20: Deformation with natural hand motion (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 21: Natural interaction from a simplified scene. T2AV-Compass case 0333 . An elderly wheelchair user stops to greet a small dog. The proxy establishes their relative positions and motion, while the generated video supplies the finer posture and appearance of the interaction.
Figure 22: Natural interaction from a simplified scene (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 23: From modeled objects to fine visual detail. T2AV-Compass case 0064 . The camera approaches a bowl of steaming dumplings in a palace courtyard. Paired proxy and generated frames show how the modeled arrangement is rendered into food texture, pleats, and steam.
Figure 24: From modeled objects to fine visual detail (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 25: Subject identity across alternating close-ups. T2AV-Compass case 0424 . The sequence alternates between a stressed pilot and his focused co-pilot. Coda-DaVinci uses simple subject proxies to establish identity and framing, allowing the video model to generate facial detail and expression.
Figure 26: Subject identity across alternating close-ups (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 27: Composition of the 94-sample evaluation set retained across all evaluated configurations. Each sector shows the benchmark name, percentage, and sample count.
Tool
Function
Skills and source files
read_skill
Load the instructions for a selected skill.
read_file
Read scene code, manifests, or other workspace files.
write_file
Create or overwrite a workspace file.
edit_file
Apply targeted edits to an existing file.
Scene execution and rendering
Appendix
Table 6: Harness tools and their functions.
Setting
Configuration
Agent coordination and review
Lead generation agent
Plans the task, integrates scene assets, and reviews the generated video.
Modeling subagents
Receive scoped construction tasks and return editable assets; their sessions can be resumed for revisions.
Acceptance agent
Independently checks the submitted video against task-specific criteria using read-only inspection tools.
Acceptance repairs
Up to three repairs per interaction. The generation agent then inspects and submits the candidate, recording remaining issues.
Persistent workspace
Retains scene code, reusable assets, subject references, and scene state across revisions and continuation requests.
Appendix
Table 7: Harness settings for task execution and video review.
Figure 28: Per-metric differences between proxy-guided and direct generation. Each cell shows the human evaluation score for G1/P1/PE2 minus the direct-generation score on the same benchmark and metric, using the 0–3 scale in Table 1 . Positive values favor the proxy-guided configuration.
Figure 29: Per-metric changes from coarse to medium proxy granularity. Each point shows the score for G1/P1/PE0 minus the score for G0/P1/PE0 , computed from the 0–3 human evaluation scores in Table 1 . Points to the right of zero favor medium granularity.
Figure 30: Mean score gains over direct generation for each coding-agent and video-generator pair. Each cell subtracts the direct-generation mean for its column from the mean for the indicated pair in Table 5 . Scores use the 1–5 diagnostic scale and average physical plausibility, instruction following, visual consistency, and aesthetics and fluency over the 33-task subset.
Recent years have witnessed remarkable progress in video generation in terms of visual quality and temporal consistency. However, generating videos with precise spatial structure, coherent motion dynamics, and controllable scene composition remains challenging. In this paper, we introduce Video-Coder, the first attempt to train a code-driven video generation agent, which autonomously writes and executes code to construct a dynamic 3D scene (Draft Video) that provides structure and motion signals to downstream video generators. To support this paradigm, we design a dedicated data pipeline and curate two high-quality datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k, which contain diverse prompts and ground-truth reference videos. We further develop the Video- Coder Harness, a unified interactive environment integrating code execution, 3D scene rendering, and execution feedback for agent training and inference. We then train Video-Coder with SFT fol- lowed by harness-native agentic reinforcement learning with dual reward feedback, which combines execution-based and video-based rewards. Additionally, we design an annealing training strategy that gradually reduces the execution reward over time, allowing the model to rely more on the video reward in the later training stages. Experiments show that Video-Coder achieves significant gains, improving Minimax-H3 by approximately 7 points on PhyGenBench and 24 points on VBench-2.0. We hope this work can serve as an open foundation for coding agents in video generation, and we fully open-source our data, models, and code
Chaoyang Wang, Sicheng Gao, Junan Hu +7
University of Central Florida · University of Würzburg · Microsoft Research +3
Text-to-video models have achieved remarkable visual quality, yet they still struggle to generate physically consistent dynamics because the temporal evolution of a scene must be inferred implicitly from a highly compressed text prompt. Existing chain-of-thought approaches introduce intermediate plans or visual states, but these representations are typically non-executable or temporally sparse, limiting their ability to instantiate and control the complete spatiotemporal process. To address this limitation, we introduce VideoCoCo, an agentic dual-engine framework in which executable Blender code serves as a process-level chain of thought. Given a text prompt, a coding agent synthesizes a Blender program that explicitly specifies the scene and its temporal evolution. The executable simulation engine runs the program to produce a deterministic spatiotemporal draft, which is subsequently transformed into a photorealistic video by a generative video engine through draft-conditioned editing. This decomposition separates process-level reasoning from high-fidelity visual realization. To adapt the video editor to simulated drafts, we construct VideoCoCo-3K, a curated dataset of draft-instruction-target triplets. VideoCoCo improves the OmniWeaving baseline from 0.475 to 0.558 on PhyGenBench and from 52.18 to 77.88 on VBench-2.0, achieving the best average score on both benchmarks. These results demonstrate that executable code provides an effective, controllable, and inspectable intermediate representation for physically consistent video generation.
Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.