Organizations: Nanjing University · KlingAI Research · Peking University · Hong Kong University of Science and Technology (Guangzhou) · University of Hong Kong · Hong Kong University of Science and Technology · Tsinghua University
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Figures & tables
Figure 1: Selective 3D modeling. Each method pairs 3D proxy frames with generated video frames at 3, 5, and 7 seconds. Codex + Blender and VideoCoCo explicitly model hands and arms. Coda-DaVinci models the objects involved in cutting and unwrapping, while leaving detailed hand articulation to the video generator.
Figure 2: Qualitative visualization of component configurations. Within each labeled group, the upper sequence shows the proxy and the lower sequence shows the corresponding generated video; columns follow the same eight sampled timestamps. The groups labeled P0 and P1 correspond to G1_P1_PE1 and G0_P1_PE0 , respectively. Both enable explicit physics simulation and differ in geometry and prompt enhancement.
Figure 3: Effects of proxy granularity, physics simulation, and prompt enhancement on three benchmarks. Each bar shows the mean of the four human evaluation scores (C/I/P/N) reported for one configuration on one benchmark in Table 1 . Higher scores are better. From left to right, the groups vary granularity at fixed P1/PE0 , physics simulation at fixed G1/PE0 , and prompt enhancement at fixed G1/P1 . All three groups share the reference configuration G1/P1/PE0 .
WorldMemBench
VGIF-Bench
Principia
Average
Configuration
C ↑
I ↑
P ↑
N ↑
C ↑
I ↑
P ↑
N ↑
C ↑
I ↑
P ↑
N ↑
C ↑
I ↑
P ↑
N ↑
Direct generation
2.182
1.242
2.485
2.333
2.483
1.828
2.207
2.000
2.562
2.406
2.000
2.156
2.404
1.819
2.234
2.170
G0/P0/PE0
2.758
3.000
2.606
2.455
2.429
2.276
1.897
1.414
2.969
2.938
2.562
2.594
2.731
2.755
2.372
2.181
G1/P0/PE0
2.606
3.000
2.576
2.455
2.690
2.207
1.448
1.034
2.969
3.000
2.656
2.719
2.755
2.755
2.255
2.106
G2/P0/PE0
2.788
3.000
2.606
2.455
2.552
2.069
1.448
1.000
2.906
2.875
2.531
2.562
2.755
2.670
2.223
2.043
G0/P1/PE0
2.606
2.970
2.758
2.697
2.448
2.276
1.759
1.345
2.969
2.969
2.531
2.625
2.681
2.755
2.372
2.255
Table 1: Human evaluation on three benchmarks and per-metric averages over their pooled samples. C/I/P/N denote consistency, instruction following, physical plausibility, and naturalness and visual realism, respectively.
Comparison
Config.
C
I
P
N
Mean
P1-P0
G0/PE0
-0.054
0.000
0.000
+0.075
+0.005
G1/PE0
+0.043
0.000
+0.160
+0.128
+0.082
G2/PE0
-0.043
+0.074
+0.096
+0.011
+0.035
G1/PE1
+0.200
+0.250
+0.062
+0.012
+0.131
G1/PE2
+0.149
+0.191
+0.223
+0.096
+0.165
Table 2: Changes in per-metric averages over pooled samples from three benchmarks ( P1 minus P0 ); Mean: C/I/P/N average.
Comparison
Config.
C
I
P
N
Mean
PE2-PE1
G1/P0
+0.468
+0.224
+0.043
+0.149
+0.221
G1/P1
+0.364
+0.131
+0.261
+0.263
+0.255
PE2-PE0
G1/P0
+0.022
+0.022
-0.042
+0.011
+0.003
G1/P1
+0.128
+0.213
+0.021
-0.021
+0.085
Table 3: Changes in per-metric averages over pooled samples from three benchmarks using PE2 at fixed G1 ; Mean: C/I/P/N average.
Figure 4: Prompt enhancement under two physics settings at fixed proxy granularity ( G1 ). Each point is the unweighted mean of the four human evaluation scores (C/I/P/N) reported in Table 1 for one benchmark and configuration. Dashed lines with open circles denote optional physics simulation ( P0 ), and solid lines with diamonds denote required physics simulation ( P1 ).
Figure 5: Overview of Coda-DaVinci. Coda-DaVinci combines task-adaptive proxy construction with proxy-grounded video synthesis. The harness maintains the working scene and supports visual feedback throughout generation.
Last Exam
R2M
VTR
T2AV-Compass
Method
Score ↑
Scene ID ↑
Local Geo. ↑
State ↑
Object ↑
Checklist ↑
WER ↓
VR ↑
IF ↑
Direct generation
47.16
0.347
0.351
0.159
0.343
85.15
59.21
71.38
65.11
Agentic Video Generation
Codex
53.18
0.575
0.583
0.520
0.587
89.72
25.73
72.21
68.05
Claude
51.95
0.594
0.590
0.523
0.522
87.07
53.62
71.50
68.22
Codex + VideoWeaver
55.11
0.739
0.705
0.646
0.665
90.88
23.58
71.83
67.60
Table 4: Main results across four benchmarks. R2M reports normalized memory ratios (NMR). WER: word error rate; VR: video realism; IF: visual instruction following. Green cells mark the best result in each column.
Agent
P
I
C
A&F
Mean
Seedance 2.5
Direct generation
3.242
3.364
3.758
3.667
3.508
GPT-6-Astra
3.455
3.758
3.879
3.758
3.712
GPT-6.1-Sol
3.394
3.545
3.818
3.667
3.606
Claude Opus 5.5
3.548
3.742
3.839
3.645
3.694
Gemini 4 Argon
3.562
3.719
3.656
3.656
3.648
Table 5: Ablations on the test subset. P: physics; I: instruction following; C: consistency; A&F: aesthetics and fluency. Green marks column maxima (including ties) per video model.
Figure 6: Performance and ablation studies. (a) Normalized benchmark results. (b) Ablations of adaptive routing, independent review, and modeling granularity. (c) Video quality and token cost across rendering engines. (d) On LongAV-Compass, Coda-DaVinci improves event fulfillment and long-form structure over direct generation, with lower transition stability. (e) On WBench, it improves interaction adherence and consistency, with lower setting adherence.
Figure 7: Camera instruction following and state persistence. The camera turns away while leaves are being cleared, then returns to the steps. Each method occupies two rows, read consecutively from left to right. Frames are sampled at matching timestamps, with denser sampling during the return. Both code2video methods show cleared steps on revisitation, while Seedance retains substantial leaf coverage.
Figure 8: Hand articulation and visual fluency. Consecutive frames from a dough-flattening task, sampled at 4 fps over 4.50–8.25 seconds. Each method occupies two rows, with the second continuing the first. Compared with the nearly fixed pose of Codex + Blender, Coda-DaVinci shows changing wrist and finger poses through repeated contact with the dough. Seedance provides the direct-generation comparison.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Scene geometry during navigation. R2M-Bench case WSLRRLL_0046 . The portal, curved roof, and surrounding station provide landmarks for comparing scene structure through the camera movement. Proxy renders and generated frames reveal how each modeled scene carries through synthesis.
Figure 10: Scene geometry during navigation (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 11: Preserving an edited scene state. WBench case 248. The monitor goes black before the player stands up and turns around. Direct generation brings back the game display in later turns, while Coda-DaVinci preserves the black-screen state in both the proxy and the generated video.
Figure 12: Scene consistency across successive views. WBench case 92. The camera moves through an autumn park. Coda-DaVinci retains the wooden bench across views; the direct-generation sequence introduces a bench with a different armrest structure. Corresponding proxy frames show the shared scene used for continuation.
Figure 13: Coordinating objects and interactions. VTR-Bench case CULT-0034 . Two friends compare a set of written notes and a tracklist before playing a cassette. Coda-DaVinci represents the tabletop objects and subject identities in the proxy, while the final video supplies natural hand movements and human appearance.
Figure 14: Coordinating objects and interactions (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 15: Preserving task-critical object structure. VTR-Bench case AD-0059 . The task involves adjusting a split keyboard, inspecting its layout diagram, and showing its engraved badge. The proxy specifies the keyboard components and camera views, which guide the corresponding actions in the generated video.
Figure 16: Preserving task-critical object structure (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 17: Following actions within a continuous shot. VTR-Bench case LIFE-0008 . The owner hands over a leash and explains the care instructions before the neighbor hides a key under the frog-shaped planter. Coda-DaVinci keeps these actions within the requested wide view, preserving the room layout throughout.
Figure 18: Following actions within a continuous shot (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 19: Deformation with natural hand motion. T2AV-Compass case 0348 . The proxy controls the dough flattening beneath the paper doilies. Coda-DaVinci leaves the hands to the video model, producing the requested manipulation around the changing dough shape.
Figure 20: Deformation with natural hand motion (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 21: Natural interaction from a simplified scene. T2AV-Compass case 0333 . An elderly wheelchair user stops to greet a small dog. The proxy establishes their relative positions and motion, while the generated video supplies the finer posture and appearance of the interaction.
Figure 22: Natural interaction from a simplified scene (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 23: From modeled objects to fine visual detail. T2AV-Compass case 0064 . The camera approaches a bowl of steaming dumplings in a palace courtyard. Paired proxy and generated frames show how the modeled arrangement is rendered into food texture, pleats, and steam.
Figure 24: From modeled objects to fine visual detail (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 25: Subject identity across alternating close-ups. T2AV-Compass case 0424 . The sequence alternates between a stressed pilot and his focused co-pilot. Coda-DaVinci uses simple subject proxies to establish identity and framing, allowing the video model to generate facial detail and expression.
Figure 26: Subject identity across alternating close-ups (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.
Figure 27: Composition of the 94-sample evaluation set retained across all evaluated configurations. Each sector shows the benchmark name, percentage, and sample count.
Tool
Function
Skills and source files
read_skill
Load the instructions for a selected skill.
read_file
Read scene code, manifests, or other workspace files.
write_file
Create or overwrite a workspace file.
edit_file
Apply targeted edits to an existing file.
Scene execution and rendering
Appendix
Table 6: Harness tools and their functions.
Setting
Configuration
Agent coordination and review
Lead generation agent
Plans the task, integrates scene assets, and reviews the generated video.
Modeling subagents
Receive scoped construction tasks and return editable assets; their sessions can be resumed for revisions.
Acceptance agent
Independently checks the submitted video against task-specific criteria using read-only inspection tools.
Acceptance repairs
Up to three repairs per interaction. The generation agent then inspects and submits the candidate, recording remaining issues.
Persistent workspace
Retains scene code, reusable assets, subject references, and scene state across revisions and continuation requests.
Appendix
Table 7: Harness settings for task execution and video review.
Figure 28: Per-metric differences between proxy-guided and direct generation. Each cell shows the human evaluation score for G1/P1/PE2 minus the direct-generation score on the same benchmark and metric, using the 0–3 scale in Table 1 . Positive values favor the proxy-guided configuration.
Figure 29: Per-metric changes from coarse to medium proxy granularity. Each point shows the score for G1/P1/PE0 minus the score for G0/P1/PE0 , computed from the 0–3 human evaluation scores in Table 1 . Points to the right of zero favor medium granularity.
Figure 30: Mean score gains over direct generation for each coding-agent and video-generator pair. Each cell subtracts the direct-generation mean for its column from the mean for the indicated pair in Table 5 . Scores use the 1–5 diagnostic scale and average physical plausibility, instruction following, visual consistency, and aesthetics and fluency over the 33-task subset.