Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a ruleconsistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated with blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos.
Figures & tables
Figure 2: Overview of VibeVideoBench. (a) Requests from four kinds of real-world sources are drafted under a shared contract, given the assets they need (searched for or synthesized by an LLM), and curated by humans, yielding a benchmark that is diverse in content, form, length and sound. (b) Each task gets a three-level rubric traced to its prompt and its input assets; L2 checks quote their basis and carry modality tags; rubrics go through rounds of model QC and human review until no confirmed finding remains. (c) Agents run in native harnesses in identical one-hour sandboxes; programs check L1, and three VLM judge panels—one routing each check by modality—score L2 and L3.
Benchmark
Output
Pixels from
General agents
Open domain
Audio judged
Input assets
Task-spec. rubric
VBench [ 21 ]
T2V clip
diffusion
✗
✓
✗
✗
✗
EvalCrafter [ 30 ]
T2V clip
diffusion
✗
✓
✗
✗
✗
VideoArgus [ 46 ]
T2V clip
diffusion
✗
✓
✗
✗
✓
ArtifactsBench [ 47 ]
web page
LLM code
✗
✓
✗
✗
✓
TheoremExplain [ 27 ]
Manim video
pipeline
∘
✗
∘
✗
✗
Paper2Video [ 50 ]
talk video
pipeline
✗
✗
∘
✓
∘
Table 1: Positioning. Among the benchmarks we know of, only VibeVideoBench has general-purpose agents build complete audio-visual videos from scratch with code across open domains, graded with task-specific rubrics. ✓ : yes; ✗ : no; ∘ : partially.
Figure 3: Benchmark composition (239 tasks). Top: the inner ring gives the 12 categories, with spans proportional to task counts. The outer ring names 24 of the 34 presentation forms, each shown once, in the category where it is most characteristic (equal angles, not counts); every task can carry several of the 43 subcategory tags, and the 9 knowledge domains are not drawn. The six cards are real deliveries, one per model family. Bottom: (a) prompt length of the 209 English tasks (lessons and papers keep their content in the supplied note or PDF, so their prompts are short); (b) target runtime, 8 s–15 min; (c) L2 checks per task, 13–110; (d) share of tasks with each requirement.
Table 2: Main results on 239 tasks. All judge-based scores (L2, L3) are means over the three judge panels. L1 : playable videos (of 239) and spec compliance (% of tasks passing every program check). Delivered videos : L2 pass rate and L3 overall (1–5) of playable videos, and human Elo from blind pairwise preferences between playable videos (§ 5.4 ); rows follow L3 overall. No video =0 : a missing video fails; L2 strict is the share of tasks with no failed core check (§ 4 ); grey numbers are ranks. Full results in App. E .
Figure 4: Failures and process. (a) Main cause of the 224 runs without a playable video. (b) Median minutes left at the final write of successful runs (core tasks). (c) Primary renderer (% of runs). (d) Images self-inspected per run vs. delivered quality.
Figure 5: Where the money goes (239 tasks). (a) Mean list-price API cost per task, split by token type: the Claude agents spend most on prompt-cache writes, GPT-5.6 Sol on fresh input. (b) Overall score (no video =0 ) per USD of mean task cost (log scale).
J1–J2
J1–J3
J2–J3
System Spearman ρ (overall)
0.933
0.950
0.950
Check-level agreement / Cohen’s κ
94.6 / 0.56
91.3 / 0.52
91.7 / 0.44
Video-level Pearson r (overall)
0.45
0.51
0.51
Same-task pair ordering accuracy
69.3
69.7
65.7
J1
J2
J3
Mean
Human vs. judge, system ρ
0.98
0.80
0.90
0.98
Table 3: Judge reliability on the 239 tasks. Top: agreement between judge panels (about 1,920 videos judged by each pair). Bottom: agreement with blind pairwise human preferences; Spearman between human Elo and the panel’s delivered-only overall ( n=9 ), and agreement on human-decisive pairs the panel does not tie.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Description
Count
Stream (i): web prompts
Collected
prompts written for code-based video tools (452 social posts, 25 Remotion gallery, 7 other)
484
Kept after strict review
every prompt read in full; nine removal codes
152 (31%)
Stream (ii): dense captions of real videos
Scanned
caption records (HD-VILA, Molmo2, LLaVA-Video, ActivityNet sources)
508,993
Lexical recall
26 content directions
6,711
Appendix
Table A.1 : Construction funnel. Counts at each stage; “kept” rates in parentheses.
Category
Tasks
Most frequent subcategories (tasks; multi-label)
C01 Brand & marketing
9
Contextual selling point & call to action (7), Brand & product showcase (3), Musical structure & AV choreography (2), Visual metaphor (1)
History & society (12), Statistical charts & comparison (11), History & chronology (11), Geography & earth science (8)
C06 Visual art & form
26
Spatial tour & scene experience (9), Character performance & choreography (5), Visual metaphor (5), Abstract form & scale (3)
Appendix
Table A.2 : Categories (239 tasks) with their most frequent subcategories.
ID
Presentation form
n
Cat.
ID
Presentation form
n
Cat.
S09
Mechanism & causal chain
51
6
S27
Abstract form & scale
6
3
S29
Musical structure & AV choreography
28
7
S28
Visual metaphor
6
2
S07
Making, assembly & modification
25
3
S38
Worked examples & problem solving
6
2
S08
Mathematical derivation & counting
23
4
S34
Conceptual frameworks & decision paths
5
1
S14
Statistical charts & comparison
18
5
S01
Software & device tutorials
4
1
S10
System architecture & information flow
17
3
S25
Time-span, identity & emotional change
4
2
Appendix
Table A.3 : Subcategories (multi-label). n : tasks carrying the subcategory; Cat.: number of categories it appears in.
Language
209 English / 30 Chinese [core 199 / 6; lessons 24 Chinese; papers 10 English]
Prompt length
core English prompts: median 1,638 words, IQR 967–2,179, range 503–5,581; lesson and paper prompts are short (median 262 words/characters) and carry their content in the supplied note or PDF
Table C.1 : Evaluated systems. Each runs in its own native harness in unattended mode at the highest effort the provider accepted.
Panel
L3 (deliv.)
L2 pass
Unknown
Leniency
Spread nv = 0
Spread deliv.
1
2
3
4
5
unk.
J1
3.61
90.9%
0.9%
− 0.41
1.47
0.89
0.8
6.0
28.6
57.4
6.0
1.3
J2
4.01
95.7%
0.4%
+ 0.19
1.19
0.46
0.2
2.5
9.2
69.9
17.3
0.8
J3
4.03
88.2%
6.3%
+ 0.22
1.39
0.65
2.1
2.2
16.3
46.8
31.4
1.2
Appendix
Table D.1 : Judge scale use (239 tasks). L3, L2 and unknown: averages over the nine systems (delivered runs). Leniency: mean of the panel’s five-dimension score minus the mean of the other two panels on the same run (1,927 runs). Spread: best minus worst system overall. Right: distribution of all dimension scores (%; unk.: unknown or n/a).
System
J1 (Gemini 4 Argon)
J2 (Gemini 3.8 Flash)
J3 vision (GPT-5.6 Sol)
GPT-6 Astra
+0.17 [0.11, 0.23]
−0.23 [ − 0.29, − 0.17]
+0.06 [ − 0.02, 0.13]
Claude Opus 5.5
+0.21 [0.15, 0.27]
−0.11 [ − 0.17, − 0.04]
−0.10 [ − 0.18, − 0.03]
DeepSeek-V4.1-Flash
−0.03 [ − 0.10, 0.05]
+0.05 [ − 0.02, 0.13]
−0.03 [ − 0.09, 0.04]
Claude Sonnet 5.5
+0.18 [0.11, 0.25]
−0.08 [ − 0.15, − 0.01]
−0.10 [ − 0.17, − 0.03]
Kimi K3
+0.02 [ − 0.04, 0.08]
+0.03 [ − 0.04, 0.10]
−0.05 [ − 0.12, 0.03]
GPT-5.6 Sol
−0.32 [ − 0.40, − 0.24]
+0.12 [0.05, 0.20]
+0.20 [0.12, 0.27]
Appendix
Table D.2 : Self-preference. Relative score of each system under each panel (visual dimensions; positive = the panel rates the system higher than the other two panels do). Shaded: the panel’s own model. 95% bootstrap CIs.
Table E.1 : Full leaderboard on 239 tasks (mean of three judge panels). L1: playable-delivery rate and spec compliance (all program checks pass: the checks of Tab. E.6 on the 205 core tasks and the rule-based checks on lessons and papers; a non-delivery fails). L2: check pass rate; strict success = share of tasks whose core checks have no fail and at least one pass. L3: per-dimension and overall quality (1–5). Rows are ordered by delivered-only overall; the grey number is the rank with no video =0 . Human: Bradley–Terry Elo from blind pairwise preferences between playable videos (App. H ). $/task: mean list-price API cost over the 239 tasks (App. G ). Best in bold , second underlined (ties on the shown digits share the mark).
#
System
L2 pass (%)
Task success (%)
L3 dims (delivered only)
Overall
Unk.
deliv.
nv = 0
all core
strict
VQ
VA
CL
TM
AQ
deliv.
nv = 0
(%)
J1: Gemini 4 Argon (full video + audio)
1
GPT-6 Astra
98.4
98.2
76.6
83.7
4.00
4.06
4.07
3.87
3.80
3.963
3.946
0.6
2
Claude Opus 5.5
98.4
98.4
76.6
83.7
3.92
4.01
4.04
3.92
3.68
3.915
3.915
0.7
3
Claude Sonnet 5.5
95.2
92.5
67.8
73.6
3.79
3.92
3.97
3.83
3.62
3.826
3.712
0.7
4
DeepSeek-V4.1-Flash
91.6
90.5
49.0
54.4
3.64
3.82
3.78
3.50
3.60
3.667
3.620
0.9
Appendix
Table E.2 : Per-panel results (239 tasks; rows in the panel’s no-video- =0 order). nv = 0: no video =0 . Task success: % of tasks whose core checks all pass or are n/a, with at least one pass (“all core”, unknown counts against), and strict success (no core check fails, at least one passes); no video counts as failure. Unk.: unknown share of delivered checks.
System
Mean
95% CI
Comparison
P
W/L/T
GPT-6 Astra
4.109
[4.052, 4.156]
Astra > Opus
0.911
127/89/23
Claude Opus 5.5
4.074
[4.037, 4.109]
Opus > DeepSeek
1.000
133/74/32
DeepSeek-V4.1-Flash
3.907
[3.826, 3.978]
DeepSeek > Sonnet
0.697
89/121/29
Claude Sonnet 5.5
3.878
[3.769, 3.977]
Sonnet > Kimi
1.000
152/68/19
Kimi K3
3.441
[3.267, 3.593]
Kimi > GPT-5.6 Sol
0.860
143/86/10
GPT-5.6 Sol
3.328
[3.174, 3.479]
GPT-5.6 Sol > Argon
0.980
82/142/15
Appendix
Table E.3 : Uncertainty. Left: 95% bootstrap CI over tasks ( B=2000 ) of the per-task three-panel overall (no video =0 ); the per-task mean differs from the pooled leaderboard value by ≤ 0.011. Right: paired bootstrap probability that rank k beats rank k+1 , with per-task head-to-head wins/losses/ties.
Figure E.1 : Quality profile per system (239 tasks; three-panel mean, no video =0 ). (a) The five L3 dimensions; (b) overall score per category (mean of per-task scores; n tasks in parentheses), the top block of Tab. E.4 . Radial axes start at 2.0 in (a) and 1.0 in (b), not at 0; rings every 0.5. Circles: each lab’s flagship; squares and dashed lines: its second model.
Category
GPT-6 Astra
Opus 5.5
Sonnet 5.5
Gem. 4 Argon
DeepSeek
Kimi K3
GLM-5.3-Flash
GPT-5.6 Sol
Gem. 3.8 Flash
Mean
Overall, no video =0 (1–5)
Brand & marketing (9)
4.18
4.18
3.94
2.69
4.11
3.60
2.96
3.14
2.19
3.44
Product & feature demos (6)
4.20
4.24
3.51
2.73
3.50
4.04
4.00
4.02
3.31
3.73
Events & personal expression (7)
4.03
3.93
3.82
3.15
3.78
3.22
2.92
3.60
3.59
3.56
Concept & pipeline explainers (57)
4.19
4.15
3.94
3.13
4.03
3.46
2.53
3.17
2.92
3.50
Data & fact stories (24)
4.14
4.01
3.85
3.04
4.03
3.62
2.52
3.52
2.88
3.51
Appendix
Table E.4 : Per-category results (239 tasks, 12 categories; three-panel mean of per-task scores).
Pass rate (%)
Unknown rate (%)
System
Visual
Audio
Audio-visual
Visual
Audio
Audio-visual
GPT-6 Astra
96.4
99.4
98.6
2.5
1.7
1.9
Claude Opus 5.5
96.6
98.9
98.9
2.5
1.2
1.0
Claude Sonnet 5.5
92.3
96.6
97.9
3.0
2.7
1.5
Gemini 4 Argon
95.7
98.8
98.2
3.1
2.9
1.2
DeepSeek-V4.1-Flash
91.5
97.1
96.8
3.1
1.7
1.6
Appendix
Table E.5 : Check pass and unknown rates by modality tag (delivered runs, mean of three panels). Visual checks are the hardest for every system. J3’s frame-sampled vision judge abstains far more on visual checks (6.8–9.1%) than the full-video judges (0.1–1.5%).
System
Playable
Resolution
FPS
Duration
Frame count †
Audio ‡
Project
All pass
GPT-6 Astra
99.5
98.5
96.6
98.0
98.1
100.0
96.1
190/205
92.7
Claude Opus 5.5
100.0
100.0
99.5
100.0
100.0
100.0
100.0
204/205
99.5
Claude Sonnet 5.5
96.6
96.6
96.6
94.1
92.6
98.0
95.1
192/205
93.7
Gemini 4 Argon
75.6
75.6
75.6
75.6
79.6
74.5
75.6
155/205
75.6
DeepSeek-V4.1-Flash
98.5
98.5
97.6
97.1
96.3
99.0
96.6
192/205
93.7
Kimi K3
87.8
87.8
87.8
84.9
87.0
86.7
87.3
172/205
83.9
Appendix
Table E.6 : Program checks on the 205 core tasks (% of runs passing; non-delivery fails all checks). † 54 tasks require an exact frame count. ‡ 98 tasks whose audio policy marks at least one channel required (strict parse); pass = audio stream with peak above −50 dBFS.
System
Self-ended
Timed out
Error
Video
Playable
Check pass
Median s
Calls
Tool calls
Out. tok (k)
Reas.
GPT-6 Astra
229
10
0
238
238
220
2,464
51
49
60.9
49%
Claude Opus 5.5
238
1
0
239
239
228
2,283
51
49
130.7
56%
DeepSeek-V4.1-Flash
217
22
0
239
236
220
3,070
117
130
166.8
59%
Claude Sonnet 5.5
218
20
1
235
232
215
3,065
56
63
254.7
75%
Kimi K3
180
58
1
218
213
192
2,874
63
80
75.6
51%
GPT-5.6 Sol
147
90
2
224
213
187
3,370
109
101
129.7
57%
Appendix
Table F.1 : Run outcomes and effort (239 tasks). Video: an MP4 was collected; Playable: the judges could decode and score it. Check pass: the collector’s container-level check (App. C ); it wrongly fails the ten paper explainers for every system, which affects no score. Median s: elapsed budget time. Calls, tool calls and output tokens: per-run medians. Reas.: reasoning share of output tokens.
Primary renderer (% of runs)
Also used (% of runs)
System
cairo
PIL/numpy
Browser 2D
three.js
Blender
Other
ffmpeg
Synth. audio
GPT-6 Astra
25
16
23
29
7
0
100
93
Claude Opus 5.5
54
5
15
20
4
2
100
96
DeepSeek-V4.1-Flash
24
12
36
24
4
0
100
95
Claude Sonnet 5.5
46
10
22
19
2
1
100
89
Kimi K3
3
45
26
23
3
0
97
85
Appendix
Table F.2 : Primary renderer per system (% of runs) and, below, outcomes pooled by primary renderer. Other: Manim, MoviePy or unidentified. Synth. audio: runs that synthesize music or sound effects in code.
Self-inspection statistics
Mean delivered score of playable runs by # images viewed
System
≥ 1 view
Median
Mean
p90
Extract/run
ffprobe/run
Runs ffprobe
Last 25%
0
1–4
5–14
15–39
40+
GPT-6 Astra
229/239
3
3.4
6
3.1
2.1
202
22%
4.27
4.13
4.08
–
–
Claude Opus 5.5
238/239
7
7.9
12
4.6
3.9
237
11%
(3.47)
3.97
4.09
4.24
(3.80)
DeepSeek-V4.1-Flash
239/239
35
36.0
56
6.9
8.0
238
10%
–
(4.27)
3.53
3.91
4.07
Claude Sonnet 5.5
229/239
10
14.3
28
4.7
4.7
232
24%
3.27
3.77
3.94
4.20
4.24
Kimi K3
239/239
17
18.8
31
1.9
2.7
214
38%
–
3.29
3.84
3.90
4.05
Appendix
Table F.3 : Self-inspection. An image view is a tool call that returns an image to the model. Extract/run: shell commands pulling stills or contact sheets from a rendered video. Last 25%: share of views in the final quarter of the budget. Right: mean delivered score of playable runs by number of image views (“–”: no runs; parentheses: fewer than 5 runs).
Cause or failure mode (runs)
GPT-6 Astra
Opus 5.5
DeepSeek
Sonnet 5.5
Kimi K3
GPT-5.6 Sol
Gem. 4 Argon
Gem. 3.8 Flash
GLM-5.3-Flash
Total
Time management
1
0
2
7
25
25
31
22
56
169
Final render started late or ran too slowly
1
0
2
5
14
18
20
19
25
104
Final render never started
0
0
0
2
11
7
11
3
31
65
Stalled or cut-off render
0
0
0
0
0
0
21
24
0
45
ffmpeg pipe deadlock (stderr never drained)
0
0
0
0
0
0
19
14
0
33
Turn ended, background render cut off
0
0
0
0
0
0
2
10
0
12
Appendix
Table F.4 : Why videos are not delivered. Top: main cause of each of the 224 runs without a playable video, assigned by reading each run’s records. Bottom: state at the end of these runs.
System
Mean
Median
p90
Max
Total (239)
Calls/task
Cache hit (%)
$/playable video
Overall
Overall per $
GPT-6 Astra
8.19
7.88
11.88
15.61
1,957
46
97
8.22
4.109
0.50
Claude Opus 5.5
15.38
10.19
31.22
47.08
3,676
51
65
15.38
4.074
0.26
DeepSeek-V4.1-Flash
0.47
0.45
0.67
1.30
113
117
97
0.48
3.911
8.25
Claude Sonnet 5.5
8.26
7.09
13.07
34.99
1,974
63
81
8.51
3.880
0.47
Kimi K3
3.53
3.53
4.94
7.25
843
61
97
3.96
3.440
0.98
GPT-5.6 Sol
11.84
11.49
19.59
28.39
2,831
106
81
13.29
3.317
0.28
Appendix
Table G.1 : API cost per task (USD, list-price equivalents of the logged usage of the counting run of every task; 239 tasks, 163,197 API calls). Calls/task: requests with usage records per task. Overall: headline three-panel score (no video =0 ).
System
Win rate [Wilson 95%]
BT-Elo [95% CI]
Rank CI
Overall, delivered (rank)
GPT-6 Astra
0.783 [0.644, 0.877]
1242 [1134, 1387]
1–3
4.126 (1)
Claude Opus 5.5
0.741 [0.616, 0.837]
1203 [1111, 1327]
1–3
4.074 (2)
Claude Sonnet 5.5
0.676 [0.508, 0.809]
1121 [1016, 1257]
1–5
3.999 (3)
Gemini 4 Argon
0.607 [0.476, 0.724]
1081 [1001, 1176]
2–6
3.989 (4)
DeepSeek-V4.1-Flash
0.511 [0.370, 0.650]
1022 [900, 1154]
3–7
3.961 (5)
Kimi K3
0.449 [0.319, 0.587]
997 [873, 1107]
4–7
3.864 (6)
Appendix
Table H.1 : Human Bradley–Terry ratings (decisive votes; Elo scale; 95% bootstrap CI and rank CI). Right: the system’s delivered-only overall and its rank (Tab. E.1 ).
Metric
Spearman
Kendall
Overall, delivered only
0.98
0.94
L2 pass rate (delivered)
0.92
0.78
L2 strict success
0.92
0.78
Overall, no video =0
0.77
0.61
Playable-delivery rate
0.66
0.47
Appendix
Table H.2 : Human Elo vs. leaderboard metrics (system level, n=9 ): rank correlation with each three-panel metric.
Figure I.1 : Judge stress test (J1; 15 videos per defect). Left: share of reports that name the defect (checked by hand). Right: change of the most related L3 dimension against the clean re-encode of the same video (95% bootstrap CI).
Figure I.2 : Grounded quiz on the 34 lessons and papers: share of questions answered correctly (mean over tasks) vs. the three-panel overall on the same tasks (no video =0 ).
Figure I.3 : Time budget (DeepSeek-V4.1-Flash, 20 tasks, J2, no video =0 ; 95% CI; differences and p against 60 minutes).
Figure I.4 : CPU budget (6 systems × 20 tasks; three-panel mean, no video =0 ). Left: mean score per system and setting, with playable videos of all six systems on top. Right: pooled paired difference with 95% CI and paired- tp .
Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.
Video generation models are promising foundations for embodied world models, but only if their generated rollouts are physically valid and faithful to task instructions. Multimodal large language models (MLLMs) are increasingly used to judge these properties, yet their agreement with human judgments on embodied videos has not been systematically evaluated. To address this gap, we introduce RoboJudge-Bench, a benchmark for evaluating MLLMs as judges of embodied video generation. The benchmark measures two fundamental dimensions, Physical Adherence and Instruction Alignment, across 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands. Experiments reveal uneven alignment between existing judges and human ratings: general-purpose MLLMs show lower agreement on physical adherence, while specialized video judges designed for general-domain physical reasoning do not consistently transfer to embodied scenarios. Motivated by these findings, we construct RoboJudge-Train, comprising 12K videos annotated with 25K human-verified scores and model-assisted rationales for Physical Adherence and Instruction Alignment. We fine-tune a 9B MLLM on this dataset with supervised learning followed by reinforcement learning, yielding RoboJudge-9B. Among the model judges evaluated on RoboJudge-Bench, RoboJudge-9B achieves the highest overall agreement, reaching a Pearson correlation of 0.719 with a statistically significant 0.089 gain over Gemini-3.7-Flash, the best-performing proprietary baseline in our evaluation. Training on RoboJudge-Train also yields gains on external benchmarks, including VideoPhy2 and Physion-Eval. We further use RoboJudge-9B as a downstream evaluator to benchmark video generation models according to Physical Adherence and Instruction Alignment.
Siyuan Ma, Yu Qi, Cheng Ye +8
The Chinese University of Hong Kong · Northeastern University · University of California, San Diego +3
Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools. Project page: https://andyca111.github.io/VideoGen_Agent/
Binxu Li, Haoyi Duan, Yuhui Zhang +9
Princeton University · Stanford University · UC Davis +4