VAHBench: A Harness Benchmark for Agentic Long-Video Generation
Organizations: TokenRhythm Technologies · Peking University
Abstract
AI agents are commonly used for everyday information retrieval and coding. We ask whether they can also interpret human requests and turn them into complete videos. We introduce VAHBench, a benchmark that tests this ability through five short-drama briefs specifying story, characters, dialogue and duration. Agents plan and carry out production using shared generation tools. Evaluation combines video completeness and basic quality with checks of production reliability and reporting accuracy. Across 240 runs covering 48 configurations of general-purpose and video-specific agents, higher quality scores do not necessarily mean more complete delivery. Missing episodes, repeated footage and static filler reveal gaps between producing a video file and fulfilling the brief. Added composition tools help some agents but do not consistently improve performance. VAHBench measures how well agents translate user intent into finished video content and helps identify where that process breaks down.
Figures & tables
| Benchmark | #Tasks | Video per task | #Metrics | #Systems | Agents | No helpers | No LLM judge |
| VBench [ 12 ] | 1,600 | one clip | 16 | 4 | ✗ | – | ✓ |
| DirectorBench [ 5 ] | 80 | 1 min | 40 | 4; 6 | ✓ | ✓ | ✗ |
| DramaChain Bench [ 27 ] | 20 | 3 episodes | 63 | 22 | ✗ | ✗ | ✗ |
| VideoWeaver [ 31 ] | 285 | 1 min | 13 | 5 | ✓ | ✗ | ✗ |
| VAHBench (ours) | 5 | 3–30 min | 8 | 48 | ✓ | ✓ | ✓ |
| Model | B1 | B2 | B3 | B4 | B5 | |||
| General-purpose | ||||||||
| OpenClaw | ||||||||
| DS-F | 15.1 | 1.9 | 13.4 | 3.3 | 3.5 | 0.160 | 0.596 | 7.44 |
| GLM | 24.5 | 24.7 | 11.0 | 5.9 | 5.5 | 0.207 | 0.655 | 14.32 |
| GPT | 16.5 | 16.2 | 6.7 | 3.5 | 3.2 | 0.137 | 0.655 | 9.22 |
| Opus | 39.1 | 41.4 | 23.2 | 13.0 | 12.4 | 0.377 | 0.640 | 25.82 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Design reason | Supporting figure or table |
| Episode inventory | A surviving episode cannot substitute for missing requested episodes. | Denominator control, Table 5 . |
| Genuine-duration account | Repetition cannot improve credited delivery; overlapping defects count once. | Padding controls, Table 6 and Figure 7 a; union control, Table 5 . |
| Specification | Delivered format and decidable obligations have different failure modes. | Two leaf controls, Table 4 ; empirical distributions and omission analysis, Tables 9 and 10 . |
| Consistency | Identity drift and cut defects differ from compliant format. | Leaf control, Table 4 ; matched test records, Table 8 . |
| Plausibility | Physical contact, audio and subtitle timing are separate obligations. | Three leaf controls, Table 4 ; empirical profiles, Table 7 . |
| Process | Task recovery differs from unresolved failure; a verified contradiction differs from missing evidence. | Reliability/honesty controls, Table 4 ; category aggregation, Table 5 . Process evidence is evaluated here through analytical controls. |
| Leaf | Design reason | Controlled change | Score |
| Format | Measure the delivered artifact, not the requested format. | Three predicates pass aspect fails; resolution/audio still pass. | |
| Brief requirements | Keep content obligations separate from container format. | Four eligible clauses pass word cap fails; three still pass. | |
| Identity / cuts | Measure identity drift and cut defects separately. | Identity ratio ; cut-defect fraction stays . | |
| Physical contact | Retain graded evidence among resolved contacts. | Flagged/resolved fraction . | |
| Audio defects | A clean source cannot erase another audible defect. | All clean clipping fails, range shortfall , no long silence. | |
| Subtitle alignment | Preserve continuous timed-word agreement. | Timed-word satisfaction (deficit ). |
| Decision | Explicit input | Result and implication |
| Episode denominator | One exact episode; three required. | ; surviving-only mean would be 1. |
| Interval union | , , repeated . | 15 seconds deducted; duplicates do not add a penalty. |
| Unknown | Two format checks pass; third is ABSTAIN. | Score 1.000; source coverage 2/3. |
| Measured invalid | Two format checks pass; third is VOID. | Score 0.667; source coverage 3/3. |
| Inapplicable | Two format checks pass; third is NOT_APPLICABLE. | Score 1.000; source coverage 2/2. |
| Across episodes | Same clause has deficit and . | Worst observed satisfaction 0.600; mean would be 0.750. |
| Input case | (s) | (s) | Assembled | |||
| Exact delivery | 60 | 60 | yes | 1.000 | 1.000 | 1.000 |
| Within tolerance | 59 | 59 | yes | 0.983 | 1.000 | 1.000 |
| One second of filler | 59 | 60 | yes | 1.000 | 0.983 | 0.983 |
| Honest shortfall | 40 | 40 | yes | 0.667 | 0.417 | 0.417 |
| Same footage, padded | 40 | 59 | yes | 0.983 | 0.316 | 0.316 |
| Ten seconds looped | 10 | 60 | yes | 1.000 | 0.057 | 0.057 |
| Track | spec | cons | plaus | ||||
| Native | 90 | 0.476 | 0.807 | 0.463 | 0.483 | 0.535 | 27.67 |
| OpenMontage | 45 | 0.671 | 0.892 | 0.180 | 0.427 | 0.395 | 30.71 |
| Purpose-built | 45 | 0.253 | 0.509 | 0.265 | 0.505 | 0.380 | 12.14 |
| System | spec | cons | plaus | |||
| Claude Code | 0.933 | 0.928 | 0.274 | 0.473 | 0.494 | 53.2 |
| Codex | 0.244 | 0.715 | 0.933 | 0.319 | 0.597 | 14.6 |
| Claude Code + OpenMontage | 0.953 | 0.895 | 0.100 | 0.420 | 0.334 | 39.5 |
| Facet | Observed | Missing | Distinct | 10th pct. | Median | 90th pct. |
| spec | 240 | 0 | 152 | 0.441 | 0.814 | 0.981 |
| cons | 233 | 7 | 189 | 0.100 | 0.274 | 0.740 |
| plaus | 236 | 4 | 166 | 0.349 | 0.426 | 0.683 |
| Omitted facet | Rank | Median rank shift | Maximum rank shift |
| Specification | 0.876 | 2 | 16 |
| Consistency | 0.286 | 9 | 28 |
| Plausibility | 0.964 | 1 | 9 |
| System | spec | cons | plaus | |||
| General-purpose | ||||||
| OpenClaw | 0.220 | 0.751 | 0.616 | 0.595 | 0.637 | 14.20 |
| Hermes | 0.657 | 0.868 | 0.243 | 0.499 | 0.461 | 35.87 |
| OpenSquilla | 0.462 | 0.684 | 0.223 | 0.485 | 0.412 | 24.25 |
| Claude Code | 0.814 | 0.943 | 0.295 | 0.508 | 0.513 | 47.86 |
| Codex | 0.136 | 0.730 | 0.939 | 0.354 | 0.616 | 8.47 |
| Brief | Requested video | |||||
| B1 | s | 0.647 | 0.441 | 32.01 | 47 | 48 |
| B2 | s | 0.555 | 0.594 | 33.76 | 46 | 48 |
| B3 | s | 0.439 | 0.443 | 22.72 | 46 | 46 |
| B4 | s | 0.317 | 0.390 | 14.63 | 46 | 46 |
| B5 | s | 0.284 | 0.423 | 13.93 | 48 | 48 |
Agentic Video Generation
Agentic video generation uses AI agents to plan, create, evaluate, and refine videos through coordinated tools and feedback.
See all 49 papers