AI agents are commonly used for everyday information retrieval and coding. We ask whether they can also interpret human requests and turn them into complete videos. We introduce VAHBench, a benchmark that tests this ability through five short-drama briefs specifying story, characters, dialogue and duration. Agents plan and carry out production using shared generation tools. Evaluation combines video completeness and basic quality with checks of production reliability and reporting accuracy. Across 240 runs covering 48 configurations of general-purpose and video-specific agents, higher quality scores do not necessarily mean more complete delivery. Missing episodes, repeated footage and static filler reveal gaps between producing a video file and fulfilling the brief. Added composition tools help some agents but do not consistently improve performance. VAHBench measures how well agents translate user intent into finished video content and helps identify where that process breaks down.
Figures & tables
Figure 1: From a user brief to evidence-based evaluation. Agents interpret the brief, plan production and submit videos with execution traces. The video branch measures completeness C and basic quality Q ; the trajectory branch assesses reliability and honesty through process P . Planning artifacts and reference assets, such as shot plans and character references, support cross-checks of production decisions and delivered results. They inform evaluation without receiving separate scores.
Benchmark
#Tasks
Video per task
#Metrics
#Systems
Agents
No helpers
No LLM judge
VBench [ 12 ]
∼ 1,600
one clip
16
4
✗
–
✓
DirectorBench [ 5 ]
80
≈ 1 min
40
4; 6
✓
✓
✗
DramaChain Bench [ 27 ]
20
3 episodes
63
22
✗
✗
✗
VideoWeaver [ 31 ]
285
≈ 1 min
13
5
✓
✗
✗
VAHBench (ours)
5
3–30 min
8
48
✓
✓
✓
Table 1: Comparison with related benchmarks. “No helpers” means no benchmark-supplied composition skills. “No LLM judge” means that no LLM assigns evaluation scores; VAHBench uses LLM-assisted log parsing. Counts follow each benchmark’s protocol.
Figure 3: Composition of the five briefs. (a) Links among genres, briefs and requirements, with equal total flow per brief. (b) Overlapping genre-tag counts. (c) Declared cast sizes, including recorded or synthetic-audio roles (range 4–8; median 6).
Model
B1
B2
B3
B4
B5
Cˉ
Qˉ
Sˉ
General-purpose
OpenClaw
DS-F
15.1
1.9
13.4
3.3
3.5
0.160
0.596
7.44
GLM
24.5
24.7
11.0
5.9
5.5
0.207
0.655
14.32
GPT
16.5
16.2
6.7
3.5
3.2
0.137
0.655
9.22
Opus
39.1
41.4
23.2
13.0
12.4
0.377
0.640
25.82
Table 2: Results for 48 configurations and 240 runs. B1–B5 give run-level S ; barred columns average five briefs. Agent mean averages four models; Track mean averages all runs in that track. DS-F/DS-V = deepseek-v4-flash/vis, GLM = glm-5.2, GPT = gpt-5.6, Opus = claude-opus-5.
Figure 4: System scores by brief. Panels (a–e) show B1–B5. Each bar reports the mean S of one system on one brief; all 60 means are labelled to two decimals, with shared axes and a common blue score scale. OM = OpenMontage. Each mean covers the four-model mix tested for that system.
Figure 5: Completeness and quality across briefs. (a) Forty-eight C values per brief; black lines connect means and horizontal jitter separates overlaps. (b) Facet means over available observations.
Figure 6: Production routes in generated videos. Episode-1 frames show (a) short-brief generated footage, (b) a returning view after 15 seconds, (c) still-image pan-and-zoom and (d) a larger generated-shot plan. Timestamps identify decoded frames. Appendix D.1 documents provenance.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Design reason
Supporting figure or table
Episode inventory
A surviving episode cannot substitute for missing requested episodes.
Table 4: Analytical controls for eight scored leaves. Arrows change only the stated inputs; other eligible inputs remain fixed. Scores are checked against the implementation, not measured from additional videos.
Decision
Explicit input
Result and implication
Episode denominator
One exact episode; three required.
Craw=0.333 ; surviving-only mean would be 1.
Interval union
[10,20] , [15,25] , repeated [10,20] .
15 seconds deducted; duplicates do not add a penalty.
Unknown
Two format checks pass; third is ABSTAIN.
Score 1.000; source coverage 2/3.
Measured invalid
Two format checks pass; third is VOID.
Score 0.667; source coverage 3/3.
Inapplicable
Two format checks pass; third is NOT_APPLICABLE.
Score 1.000; source coverage 2/2.
Across episodes
Same clause has deficit 0.1 and 0.4 .
Worst observed satisfaction 0.600; mean would be 0.750.
Appendix
Table 5: Boundary controls for aggregation and missing evidence. Inputs are specified numerically. Coverage refers to active source checks within the stated leaf, rather than aggregate facet counts.
Input case
g (s)
d (s)
Assembled
D
Craw
C
Exact delivery
60
60
yes
1.000
1.000
1.000
Within tolerance
59
59
yes
0.983
1.000
1.000
One second of filler
59
60
yes
1.000
0.983
0.983
Honest shortfall
40
40
yes
0.667
0.417
0.417
Same footage, padded
40
59
yes
0.983
0.316
0.316
Ten seconds looped
10
60
yes
1.000
0.057
0.057
Appendix
Table 6: Analytical completeness controls. T=60 s and δ=2 s; D=max(0,1−∣d−T∣/T) measures container-duration agreement. Rows are specified inputs, not additional video trials.
Figure 7: Analytical scoring responses. (a) Padding 40 genuine seconds toward a 60-second target. (b) Quality aggregation for (1,1,x) . (c) Completeness gates at fixed joint quality and process factor F=0.8 . (d) Joint quality and process aggregation at Q=0.9 , with weights 3/4 and 1/4 . Curves follow the stated equations; the axes use the floored values C=max(Craw,ϵ) and P=max(Praw,ϵ) .
Track
n
C
spec
cons
plaus
Q
S
Native
90
0.476
0.807
0.463
0.483
0.535
27.67
OpenMontage
45
0.671
0.892
0.180
0.427
0.395
30.71
Purpose-built
45
0.253
0.509
0.265
0.505
0.380
12.14
Appendix
Table 7: Track means on the three shared text models. All systems contribute five briefs per model. Track compositions differ; means describe the tested systems.
Figure 8: Complementary scores on 180 common-model records. (a) Run-level completeness and quality, coloured and shaped by track. (b) Spearman correlations on the same records, using average ranks for ties and no imputation.
System
C
spec
cons
plaus
Q
S
Claude Code
0.933
0.928
0.274
0.473
0.494
53.2
Codex
0.244
0.715
0.933
0.319
0.597
14.6
Claude Code + OpenMontage
0.953
0.895
0.100
0.420
0.334
39.5
Appendix
Table 8: Matched examples on B1 with glm-5.2. The three records illustrate distinct trade-offs between completeness and quality.
Facet
Observed
Missing
Distinct
10th pct.
Median
90th pct.
spec
240
0
152
0.441
0.814
0.981
cons
233
7
189
0.100
0.274
0.740
plaus
236
4
166
0.349
0.426
0.683
Appendix
Table 9: Quality-facet distributions across 240 runs. Quantiles use linear interpolation over observed values; distinct-value counts use the recorded precision.
Omitted facet
Rank ρ
Median rank shift
Maximum rank shift
Specification
0.876
2
16
Consistency
0.286
9
28
Plausibility
0.964
1
9
Appendix
Table 10: Quality-facet omission sensitivity. Ranks compare 36 configuration means (180 records) with the three-facet Q∗ baseline. Each alternative geometrically averages the two remaining facets; reported S is unchanged.
System
C
spec
cons
plaus
Q
S
General-purpose
OpenClaw
0.220
0.751
0.616
0.595
0.637
14.20
Hermes
0.657
0.868
0.243
0.499
0.461
35.87
OpenSquilla
0.462
0.684
0.223
0.485
0.412
24.25
Claude Code
0.814
0.943
0.295
0.508
0.513
47.86
Codex
0.136
0.730
0.939
0.354
0.616
8.47
Appendix
Table 11: System means across four models and five briefs. Each row has 20 runs. Consistency/plausibility use 16/18 observations for UniVA and 17/18 for ViMax; all other means use 20.
Brief
Requested video
C
Q
S
ncons
nplaus
B1
3×60 s
0.647
0.441
32.01
47
48
B2
3×120 s
0.555
0.594
33.76
46
48
B3
5×120 s
0.439
0.443
22.72
46
46
B4
5×240 s
0.317
0.390
14.63
46
46
B5
6×300 s
0.284
0.423
13.93
48
48
Appendix
Table 12: Brief-level means over 48 configurations. ncons and nplaus count observed facets; C , specification, Q and S each have 48 observations per brief.
Figure 9: Selected productions for B1. Eight episode-1 videos, each shown at ten uniformly spaced time points. Row headings identify the system and recorded model; timestamps are in seconds. Frames retain their original content and aspect ratios.