cs.CVOct 26, 2026
SaveVibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation
Abstract
Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a ruleconsistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated with blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos.
Figures & tables
Figure 2: Overview of VibeVideoBench. (a) Requests from four kinds of real-world sources are drafted under a shared contract, given the assets they need (searched for or synthesized by an LLM), and curated by humans, yielding a benchmark that is diverse in content, form, length and sound. (b) Each task gets a three-level rubric traced to its prompt and its input assets; L2 checks quote their basis and carry modality tags; rubrics go through rounds of model QC and human review until no confirmed finding remains. (c) Agents run in native harnesses in identical one-hour sandboxes; programs check L1, and three VLM judge panels—one routing each check by modality—score L2 and L3.
| Benchmark | Output | Pixels from | General agents | Open domain | Audio judged | Input assets | Task-spec. rubric |
| VBench [ 21 ] | T2V clip | diffusion | ✗ | ✓ | ✗ | ✗ | ✗ |
| EvalCrafter [ 30 ] | T2V clip | diffusion | ✗ | ✓ | ✗ | ✗ | ✗ |
| VideoArgus [ 46 ] | T2V clip | diffusion | ✗ | ✓ | ✗ | ✗ | ✓ |
| ArtifactsBench [ 47 ] | web page | LLM code | ✗ | ✓ | ✗ | ✗ | ✓ |
| TheoremExplain [ 27 ] | Manim video | pipeline | ✗ | ✗ | ✗ | ||
| Paper2Video [ 50 ] | talk video | pipeline | ✗ | ✗ | ✓ |
Table 1: Positioning. Among the benchmarks we know of, only VibeVideoBench has general-purpose agents build complete audio-visual videos from scratch with code across open domains, graded with task-specific rubrics. ✓ : yes; ✗ : no; : partially.
Figure 3: Benchmark composition (239 tasks). Top: the inner ring gives the 12 categories, with spans proportional to task counts. The outer ring names 24 of the 34 presentation forms, each shown once, in the category where it is most characteristic (equal angles, not counts); every task can carry several of the 43 subcategory tags, and the 9 knowledge domains are not drawn. The six cards are real deliveries, one per model family. Bottom: (a) prompt length of the 209 English tasks (lessons and papers keep their content in the supplied note or PDF, so their prompts are short); (b) target runtime, 8 s–15 min; (c) L2 checks per task, 13–110; (d) share of tasks with each requirement.
Table 2: Main results on 239 tasks. All judge-based scores (L2, L3) are means over the three judge panels. L1 : playable videos (of 239) and spec compliance (% of tasks passing every program check). Delivered videos : L2 pass rate and L3 overall (1–5) of playable videos, and human Elo from blind pairwise preferences between playable videos (§ 5.4 ); rows follow L3 overall. No video : a missing video fails; L2 strict is the share of tasks with no failed core check (§ 4 ); grey numbers are ranks. Full results in App. E .
Figure 4: Failures and process. (a) Main cause of the 224 runs without a playable video. (b) Median minutes left at the final write of successful runs (core tasks). (c) Primary renderer (% of runs). (d) Images self-inspected per run vs. delivered quality.
Figure 5: Where the money goes (239 tasks). (a) Mean list-price API cost per task, split by token type: the Claude agents spend most on prompt-cache writes, GPT-5.6 Sol on fresh input. (b) Overall score (no video ) per USD of mean task cost (log scale).
| J1–J2 | J1–J3 | J2–J3 | ||
| System Spearman (overall) | 0.933 | 0.950 | 0.950 | |
| Check-level agreement / Cohen’s | 94.6 / 0.56 | 91.3 / 0.52 | 91.7 / 0.44 | |
| Video-level Pearson (overall) | 0.45 | 0.51 | 0.51 | |
| Same-task pair ordering accuracy | 69.3 | 69.7 | 65.7 | |
| J1 | J2 | J3 | Mean | |
| Human vs. judge, system | 0.98 | 0.80 | 0.90 | 0.98 |
Table 3: Judge reliability on the 239 tasks. Top: agreement between judge panels (about 1,920 videos judged by each pair). Bottom: agreement with blind pairwise human preferences; Spearman between human Elo and the panel’s delivered-only overall ( ), and agreement on human-decisive pairs the panel does not tie.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Description | Count |
| Stream (i): web prompts | ||
| Collected | prompts written for code-based video tools (452 social posts, 25 Remotion gallery, 7 other) | 484 |
| Kept after strict review | every prompt read in full; nine removal codes | 152 (31%) |
| Stream (ii): dense captions of real videos | ||
| Scanned | caption records (HD-VILA, Molmo2, LLaVA-Video, ActivityNet sources) | 508,993 |
| Lexical recall | 26 content directions | 6,711 |
Table A.1 : Construction funnel. Counts at each stage; “kept” rates in parentheses.
| Category | Tasks | Most frequent subcategories (tasks; multi-label) |
| C01 Brand & marketing | 9 | Contextual selling point & call to action (7), Brand & product showcase (3), Musical structure & AV choreography (2), Visual metaphor (1) |
| C02 Product & feature demos | 6 | Software & device tutorials (4), Making, assembly & modification (1), Musical structure & AV choreography (1), System simulation & causal modelling (1) |
| C03 Events & personal expression | 7 | Event invitation (4), Personal expression & tributes (3), Statistical charts & comparison (1), Musical structure & AV choreography (1) |
| C04 Concept & pipeline explainers | 57 | Mechanism & causal chain (26), Engineering & machines (14), Physics & astronomy (13), Mathematical derivation & counting (12) |
| C05 Data & fact stories | 24 | History & society (12), Statistical charts & comparison (11), History & chronology (11), Geography & earth science (8) |
| C06 Visual art & form | 26 | Spatial tour & scene experience (9), Character performance & choreography (5), Visual metaphor (5), Abstract form & scale (3) |
Table A.2 : Categories (239 tasks) with their most frequent subcategories.
| ID | Presentation form | Cat. | ID | Presentation form | Cat. | ||
| S09 | Mechanism & causal chain | 51 | 6 | S27 | Abstract form & scale | 6 | 3 |
| S29 | Musical structure & AV choreography | 28 | 7 | S28 | Visual metaphor | 6 | 2 |
| S07 | Making, assembly & modification | 25 | 3 | S38 | Worked examples & problem solving | 6 | 2 |
| S08 | Mathematical derivation & counting | 23 | 4 | S34 | Conceptual frameworks & decision paths | 5 | 1 |
| S14 | Statistical charts & comparison | 18 | 5 | S01 | Software & device tutorials | 4 | 1 |
| S10 | System architecture & information flow | 17 | 3 | S25 | Time-span, identity & emotional change | 4 | 2 |
Table A.3 : Subcategories (multi-label). : tasks carrying the subcategory; Cat.: number of categories it appears in.
| Language | 209 English / 30 Chinese [core 199 / 6; lessons 24 Chinese; papers 10 English] |
| Prompt length | core English prompts: median 1,638 words, IQR 967–2,179, range 503–5,581; lesson and paper prompts are short (median 262 words/characters) and carry their content in the supplied note or PDF |
| Resolution / fps | 1920 1080: 227; other: 12; 30 fps: 228, 60 fps: 11; exact frame count: 54 |
| Target duration | midpoint median 118 s, IQR 55–270 s, range 8–894 s; lessons 180–480 s; 38 tasks exceed 5 minutes; 11.1 hours per pass |
| Audio policy | 1 required channel: 168 (70%) [core 134 (65%)]; speech required: 117 (49%); every lesson and paper requires narration |
| Input assets | 201 tasks [core 167, all 34 lessons and papers]; 2.69 GB |
| Rubric | 12,985 L2 checks [core 10,813; lessons and papers 2,172]; mean 54.3, median 54, range 13–110 |
Table A.4 : Task statistics (239 tasks; core and lesson/paper subsets in brackets).
| ID / tag | Title | Pass | Fail | Observation scope |
| V05 core, visual | Initial small form | Mario is small at departure and does not possess large-form abilities without cause during the initial stage. | Starts in the large form and skips the first growth, or has already grown before acquiring the mushroom. | From departure until before the first mushroom contact |
| V12 core, visual | Large form breaks a brick from below | After becoming large, Mario hits BRICK1 or BRICK2 from below and causes at least one to shatter. | Breaking the brick early in small form, without contact, or hitting only the question block. | After the transformation until the first brick breaks |
| V30 core, visual | Complete jumps | Required jumps have traceable takeoff, airborne movement and landing/contact. | Pose changes impersonate displacement; teleporting to the landing point. | All required jumps |
| V41 core, visual | Order of key events | Q1 mushroom brick G1 stomp shell kicked into G2 hit by G3 lift over pit flagpole and castle. | Reordering events, or montage that hides the actual sequence. | Whole video |
| V50 core, audio-visual | Jump SFX synchronization | A jump cue corresponding to the action is present at each required takeoff. | No cue, or the effect occurs during an unrelated action. | Every required takeoff and the audio track |
Table B.1 : Example task checks from CG01 (Super Mario; 7 core goals, 59 checks).
| System | Lab | Harness (unattended mode) | Effort |
| GPT-6 Astra | OpenAI | Codex CLI 0.162 ( exec --dangerously-bypass-approvals-and-sandbox ) | max |
| GPT-5.6 Sol | OpenAI | Codex CLI (same) | max (xhigh on 35 early runs) |
| Claude Opus 5.5 | Anthropic | Claude Code 2.1.301 ( -p --dangerously-skip-permissions ) | max |
| Claude Sonnet 5.5 | Anthropic | Claude Code 2.1.301 (same) | max |
| Gemini 4 Argon | Antigravity CLI ( --print --dangerously-skip-permissions ) | high (xhigh rejected) | |
| Gemini 3.8 Flash | Antigravity CLI (same) | high |
Table C.1 : Evaluated systems. Each runs in its own native harness in unattended mode at the highest effort the provider accepted.
| Panel | L3 (deliv.) | L2 pass | Unknown | Leniency | Spread nv 0 | Spread deliv. | 1 | 2 | 3 | 4 | 5 | unk. |
| J1 | 3.61 | 90.9% | 0.9% | 0.41 | 1.47 | 0.89 | 0.8 | 6.0 | 28.6 | 57.4 | 6.0 | 1.3 |
| J2 | 4.01 | 95.7% | 0.4% | 0.19 | 1.19 | 0.46 | 0.2 | 2.5 | 9.2 | 69.9 | 17.3 | 0.8 |
| J3 | 4.03 | 88.2% | 6.3% | 0.22 | 1.39 | 0.65 | 2.1 | 2.2 | 16.3 | 46.8 | 31.4 | 1.2 |
Table D.1 : Judge scale use (239 tasks). L3, L2 and unknown: averages over the nine systems (delivered runs). Leniency: mean of the panel’s five-dimension score minus the mean of the other two panels on the same run (1,927 runs). Spread: best minus worst system overall. Right: distribution of all dimension scores (%; unk.: unknown or n/a).
| System | J1 (Gemini 4 Argon) | J2 (Gemini 3.8 Flash) | J3 vision (GPT-5.6 Sol) |
| GPT-6 Astra | [0.11, 0.23] | [ 0.29, 0.17] | [ 0.02, 0.13] |
| Claude Opus 5.5 | [0.15, 0.27] | [ 0.17, 0.04] | [ 0.18, 0.03] |
| DeepSeek-V4.1-Flash | [ 0.10, 0.05] | [ 0.02, 0.13] | [ 0.09, 0.04] |
| Claude Sonnet 5.5 | [0.11, 0.25] | [ 0.15, 0.01] | [ 0.17, 0.03] |
| Kimi K3 | [ 0.04, 0.08] | [ 0.04, 0.10] | [ 0.12, 0.03] |
| GPT-5.6 Sol | [ 0.40, 0.24] | [0.05, 0.20] | [0.12, 0.27] |
Table D.2 : Self-preference. Relative score of each system under each panel (visual dimensions; positive = the panel rates the system higher than the other two panels do). Shaded: the panel’s own model. 95% bootstrap CIs.
Table E.1 : Full leaderboard on 239 tasks (mean of three judge panels). L1: playable-delivery rate and spec compliance (all program checks pass: the checks of Tab. E.6 on the 205 core tasks and the rule-based checks on lessons and papers; a non-delivery fails). L2: check pass rate; strict success = share of tasks whose core checks have no fail and at least one pass. L3: per-dimension and overall quality (1–5). Rows are ordered by delivered-only overall; the grey number is the rank with no video . Human: Bradley–Terry Elo from blind pairwise preferences between playable videos (App. H ). $/task: mean list-price API cost over the 239 tasks (App. G ). Best in bold , second underlined (ties on the shown digits share the mark).
| # | System | L2 pass (%) | Task success (%) | L3 dims (delivered only) | Overall | Unk. | |||||||
| deliv. | nv 0 | all core | strict | VQ | VA | CL | TM | AQ | deliv. | nv 0 | (%) | ||
| J1: Gemini 4 Argon (full video + audio) | |||||||||||||
| 1 | GPT-6 Astra | 98.4 | 98.2 | 76.6 | 83.7 | 4.00 | 4.06 | 4.07 | 3.87 | 3.80 | 3.963 | 3.946 | 0.6 |
| 2 | Claude Opus 5.5 | 98.4 | 98.4 | 76.6 | 83.7 | 3.92 | 4.01 | 4.04 | 3.92 | 3.68 | 3.915 | 3.915 | 0.7 |
| 3 | Claude Sonnet 5.5 | 95.2 | 92.5 | 67.8 | 73.6 | 3.79 | 3.92 | 3.97 | 3.83 | 3.62 | 3.826 | 3.712 | 0.7 |
| 4 | DeepSeek-V4.1-Flash | 91.6 | 90.5 | 49.0 | 54.4 | 3.64 | 3.82 | 3.78 | 3.50 | 3.60 | 3.667 | 3.620 | 0.9 |
Table E.2 : Per-panel results (239 tasks; rows in the panel’s no-video- order). nv 0: no video . Task success: % of tasks whose core checks all pass or are n/a, with at least one pass (“all core”, unknown counts against), and strict success (no core check fails, at least one passes); no video counts as failure. Unk.: unknown share of delivered checks.
| System | Mean | 95% CI | Comparison | W/L/T | |
| GPT-6 Astra | 4.109 | [4.052, 4.156] | Astra Opus | 0.911 | 127/89/23 |
| Claude Opus 5.5 | 4.074 | [4.037, 4.109] | Opus DeepSeek | 1.000 | 133/74/32 |
| DeepSeek-V4.1-Flash | 3.907 | [3.826, 3.978] | DeepSeek Sonnet | 0.697 | 89/121/29 |
| Claude Sonnet 5.5 | 3.878 | [3.769, 3.977] | Sonnet Kimi | 1.000 | 152/68/19 |
| Kimi K3 | 3.441 | [3.267, 3.593] | Kimi GPT-5.6 Sol | 0.860 | 143/86/10 |
| GPT-5.6 Sol | 3.328 | [3.174, 3.479] | GPT-5.6 Sol Argon | 0.980 | 82/142/15 |
Table E.3 : Uncertainty. Left: 95% bootstrap CI over tasks ( ) of the per-task three-panel overall (no video ); the per-task mean differs from the pooled leaderboard value by 0.011. Right: paired bootstrap probability that rank beats rank , with per-task head-to-head wins/losses/ties.
Figure E.1 : Quality profile per system (239 tasks; three-panel mean, no video ). (a) The five L3 dimensions; (b) overall score per category (mean of per-task scores; tasks in parentheses), the top block of Tab. E.4 . Radial axes start at 2.0 in (a) and 1.0 in (b), not at 0; rings every 0.5. Circles: each lab’s flagship; squares and dashed lines: its second model.
| Category | GPT-6 Astra | Opus 5.5 | Sonnet 5.5 | Gem. 4 Argon | DeepSeek | Kimi K3 | GLM-5.3-Flash | GPT-5.6 Sol | Gem. 3.8 Flash | Mean |
| Overall, no video (1–5) | ||||||||||
| Brand & marketing (9) | 4.18 | 4.18 | 3.94 | 2.69 | 4.11 | 3.60 | 2.96 | 3.14 | 2.19 | 3.44 |
| Product & feature demos (6) | 4.20 | 4.24 | 3.51 | 2.73 | 3.50 | 4.04 | 4.00 | 4.02 | 3.31 | 3.73 |
| Events & personal expression (7) | 4.03 | 3.93 | 3.82 | 3.15 | 3.78 | 3.22 | 2.92 | 3.60 | 3.59 | 3.56 |
| Concept & pipeline explainers (57) | 4.19 | 4.15 | 3.94 | 3.13 | 4.03 | 3.46 | 2.53 | 3.17 | 2.92 | 3.50 |
| Data & fact stories (24) | 4.14 | 4.01 | 3.85 | 3.04 | 4.03 | 3.62 | 2.52 | 3.52 | 2.88 | 3.51 |
Table E.4 : Per-category results (239 tasks, 12 categories; three-panel mean of per-task scores).
| Pass rate (%) | Unknown rate (%) | |||||
| System | Visual | Audio | Audio-visual | Visual | Audio | Audio-visual |
| GPT-6 Astra | 96.4 | 99.4 | 98.6 | 2.5 | 1.7 | 1.9 |
| Claude Opus 5.5 | 96.6 | 98.9 | 98.9 | 2.5 | 1.2 | 1.0 |
| Claude Sonnet 5.5 | 92.3 | 96.6 | 97.9 | 3.0 | 2.7 | 1.5 |
| Gemini 4 Argon | 95.7 | 98.8 | 98.2 | 3.1 | 2.9 | 1.2 |
| DeepSeek-V4.1-Flash | 91.5 | 97.1 | 96.8 | 3.1 | 1.7 | 1.6 |
Table E.5 : Check pass and unknown rates by modality tag (delivered runs, mean of three panels). Visual checks are the hardest for every system. J3’s frame-sampled vision judge abstains far more on visual checks (6.8–9.1%) than the full-video judges (0.1–1.5%).
| System | Playable | Resolution | FPS | Duration | Frame count † | Audio ‡ | Project | All pass | |
| GPT-6 Astra | 99.5 | 98.5 | 96.6 | 98.0 | 98.1 | 100.0 | 96.1 | 190/205 | 92.7 |
| Claude Opus 5.5 | 100.0 | 100.0 | 99.5 | 100.0 | 100.0 | 100.0 | 100.0 | 204/205 | 99.5 |
| Claude Sonnet 5.5 | 96.6 | 96.6 | 96.6 | 94.1 | 92.6 | 98.0 | 95.1 | 192/205 | 93.7 |
| Gemini 4 Argon | 75.6 | 75.6 | 75.6 | 75.6 | 79.6 | 74.5 | 75.6 | 155/205 | 75.6 |
| DeepSeek-V4.1-Flash | 98.5 | 98.5 | 97.6 | 97.1 | 96.3 | 99.0 | 96.6 | 192/205 | 93.7 |
| Kimi K3 | 87.8 | 87.8 | 87.8 | 84.9 | 87.0 | 86.7 | 87.3 | 172/205 | 83.9 |
Table E.6 : Program checks on the 205 core tasks (% of runs passing; non-delivery fails all checks). † 54 tasks require an exact frame count. ‡ 98 tasks whose audio policy marks at least one channel required (strict parse); pass = audio stream with peak above dBFS.
| System | Self-ended | Timed out | Error | Video | Playable | Check pass | Median s | Calls | Tool calls | Out. tok (k) | Reas. |
| GPT-6 Astra | 229 | 10 | 0 | 238 | 238 | 220 | 2,464 | 51 | 49 | 60.9 | 49% |
| Claude Opus 5.5 | 238 | 1 | 0 | 239 | 239 | 228 | 2,283 | 51 | 49 | 130.7 | 56% |
| DeepSeek-V4.1-Flash | 217 | 22 | 0 | 239 | 236 | 220 | 3,070 | 117 | 130 | 166.8 | 59% |
| Claude Sonnet 5.5 | 218 | 20 | 1 | 235 | 232 | 215 | 3,065 | 56 | 63 | 254.7 | 75% |
| Kimi K3 | 180 | 58 | 1 | 218 | 213 | 192 | 2,874 | 63 | 80 | 75.6 | 51% |
| GPT-5.6 Sol | 147 | 90 | 2 | 224 | 213 | 187 | 3,370 | 109 | 101 | 129.7 | 57% |
Table F.1 : Run outcomes and effort (239 tasks). Video: an MP4 was collected; Playable: the judges could decode and score it. Check pass: the collector’s container-level check (App. C ); it wrongly fails the ten paper explainers for every system, which affects no score. Median s: elapsed budget time. Calls, tool calls and output tokens: per-run medians. Reas.: reasoning share of output tokens.
| Primary renderer (% of runs) | Also used (% of runs) | |||||||
| System | cairo | PIL/numpy | Browser 2D | three.js | Blender | Other | ffmpeg | Synth. audio |
| GPT-6 Astra | 25 | 16 | 23 | 29 | 7 | 0 | 100 | 93 |
| Claude Opus 5.5 | 54 | 5 | 15 | 20 | 4 | 2 | 100 | 96 |
| DeepSeek-V4.1-Flash | 24 | 12 | 36 | 24 | 4 | 0 | 100 | 95 |
| Claude Sonnet 5.5 | 46 | 10 | 22 | 19 | 2 | 1 | 100 | 89 |
| Kimi K3 | 3 | 45 | 26 | 23 | 3 | 0 | 97 | 85 |
Table F.2 : Primary renderer per system (% of runs) and, below, outcomes pooled by primary renderer. Other: Manim, MoviePy or unidentified. Synth. audio: runs that synthesize music or sound effects in code.
| Self-inspection statistics | Mean delivered score of playable runs by # images viewed | ||||||||||||
| System | 1 view | Median | Mean | p90 | Extract/run | ffprobe/run | Runs ffprobe | Last 25% | 0 | 1–4 | 5–14 | 15–39 | 40+ |
| GPT-6 Astra | 229/239 | 3 | 3.4 | 6 | 3.1 | 2.1 | 202 | 22% | 4.27 | 4.13 | 4.08 | – | – |
| Claude Opus 5.5 | 238/239 | 7 | 7.9 | 12 | 4.6 | 3.9 | 237 | 11% | (3.47) | 3.97 | 4.09 | 4.24 | (3.80) |
| DeepSeek-V4.1-Flash | 239/239 | 35 | 36.0 | 56 | 6.9 | 8.0 | 238 | 10% | – | (4.27) | 3.53 | 3.91 | 4.07 |
| Claude Sonnet 5.5 | 229/239 | 10 | 14.3 | 28 | 4.7 | 4.7 | 232 | 24% | 3.27 | 3.77 | 3.94 | 4.20 | 4.24 |
| Kimi K3 | 239/239 | 17 | 18.8 | 31 | 1.9 | 2.7 | 214 | 38% | – | 3.29 | 3.84 | 3.90 | 4.05 |
Table F.3 : Self-inspection. An image view is a tool call that returns an image to the model. Extract/run: shell commands pulling stills or contact sheets from a rendered video. Last 25%: share of views in the final quarter of the budget. Right: mean delivered score of playable runs by number of image views (“–”: no runs; parentheses: fewer than 5 runs).
| Cause or failure mode (runs) | GPT-6 Astra | Opus 5.5 | DeepSeek | Sonnet 5.5 | Kimi K3 | GPT-5.6 Sol | Gem. 4 Argon | Gem. 3.8 Flash | GLM-5.3-Flash | Total |
| Time management | 1 | 0 | 2 | 7 | 25 | 25 | 31 | 22 | 56 | 169 |
| Final render started late or ran too slowly | 1 | 0 | 2 | 5 | 14 | 18 | 20 | 19 | 25 | 104 |
| Final render never started | 0 | 0 | 0 | 2 | 11 | 7 | 11 | 3 | 31 | 65 |
| Stalled or cut-off render | 0 | 0 | 0 | 0 | 0 | 0 | 21 | 24 | 0 | 45 |
| ffmpeg pipe deadlock (stderr never drained) | 0 | 0 | 0 | 0 | 0 | 0 | 19 | 14 | 0 | 33 |
| Turn ended, background render cut off | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 10 | 0 | 12 |
Table F.4 : Why videos are not delivered. Top: main cause of each of the 224 runs without a playable video, assigned by reading each run’s records. Bottom: state at the end of these runs.
| System | Mean | Median | p90 | Max | Total (239) | Calls/task | Cache hit (%) | $/playable video | Overall | Overall per $ |
| GPT-6 Astra | 8.19 | 7.88 | 11.88 | 15.61 | 1,957 | 46 | 97 | 8.22 | 4.109 | 0.50 |
| Claude Opus 5.5 | 15.38 | 10.19 | 31.22 | 47.08 | 3,676 | 51 | 65 | 15.38 | 4.074 | 0.26 |
| DeepSeek-V4.1-Flash | 0.47 | 0.45 | 0.67 | 1.30 | 113 | 117 | 97 | 0.48 | 3.911 | 8.25 |
| Claude Sonnet 5.5 | 8.26 | 7.09 | 13.07 | 34.99 | 1,974 | 63 | 81 | 8.51 | 3.880 | 0.47 |
| Kimi K3 | 3.53 | 3.53 | 4.94 | 7.25 | 843 | 61 | 97 | 3.96 | 3.440 | 0.98 |
| GPT-5.6 Sol | 11.84 | 11.49 | 19.59 | 28.39 | 2,831 | 106 | 81 | 13.29 | 3.317 | 0.28 |
Table G.1 : API cost per task (USD, list-price equivalents of the logged usage of the counting run of every task; 239 tasks, 163,197 API calls). Calls/task: requests with usage records per task. Overall: headline three-panel score (no video ).
| System | Win rate [Wilson 95%] | BT-Elo [95% CI] | Rank CI | Overall, delivered (rank) |
| GPT-6 Astra | 0.783 [0.644, 0.877] | 1242 [1134, 1387] | 1–3 | 4.126 (1) |
| Claude Opus 5.5 | 0.741 [0.616, 0.837] | 1203 [1111, 1327] | 1–3 | 4.074 (2) |
| Claude Sonnet 5.5 | 0.676 [0.508, 0.809] | 1121 [1016, 1257] | 1–5 | 3.999 (3) |
| Gemini 4 Argon | 0.607 [0.476, 0.724] | 1081 [1001, 1176] | 2–6 | 3.989 (4) |
| DeepSeek-V4.1-Flash | 0.511 [0.370, 0.650] | 1022 [900, 1154] | 3–7 | 3.961 (5) |
| Kimi K3 | 0.449 [0.319, 0.587] | 997 [873, 1107] | 4–7 | 3.864 (6) |
Table H.1 : Human Bradley–Terry ratings (decisive votes; Elo scale; 95% bootstrap CI and rank CI). Right: the system’s delivered-only overall and its rank (Tab. E.1 ).
| Metric | Spearman | Kendall |
| Overall, delivered only | 0.98 | 0.94 |
| L2 pass rate (delivered) | 0.92 | 0.78 |
| L2 strict success | 0.92 | 0.78 |
| Overall, no video | 0.77 | 0.61 |
| Playable-delivery rate | 0.66 | 0.47 |
Table H.2 : Human Elo vs. leaderboard metrics (system level, ): rank correlation with each three-panel metric.
Figure I.1 : Judge stress test (J1; 15 videos per defect). Left: share of reports that name the defect (checked by hand). Right: change of the most related L3 dimension against the clean re-encode of the same video (95% bootstrap CI).
Figure I.2 : Grounded quiz on the 34 lessons and papers: share of questions answered correctly (mean over tasks) vs. the three-panel overall on the same tasks (no video ).
Figure I.3 : Time budget (DeepSeek-V4.1-Flash, 20 tasks, J2, no video ; 95% CI; differences and against 60 minutes).
Figure I.4 : CPU budget (6 systems 20 tasks; three-panel mean, no video ). Left: mean score per system and setting, with playable videos of all six systems on top. Right: pooled paired difference with 95% CI and paired- .
More in this topic
Video Generation Evaluation
Video generation evaluation measures visual quality, prompt adherence, temporal consistency, physical plausibility, and task-specific behavior in synthesized videos.
See all 191 papers