Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Zhenghan Chen; Zekai Shao; Xin Lin; Leting Cheng; Siming Chen

cs.CVOct 20, 2026

Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Authors: Zhenghan Chen, Zekai Shao, Xin Lin, Leting Cheng, Siming Chen

Organizations: Fudan University · Sun Yat-sen University

Abstract

Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.

Figures & tables

Benchmark	#Items	Source	Approach	Task Scope	Input	Output	Duration	Avg. Tokens / Events	Evaluation	#Metrics
Direct Video Generation
VBench [ 22 ]	946	Synthetic	Direct	General	T	Video	$<1$	10 / 1.06	Objective	16
VABench [ 21 ]	1,299	Synthetic	Direct	General	T / I	Video	$<1$	50 / 2.31	Dual	15
T2AV-Compass [ 10 ]	500	Hybrid	Direct	General	T	Video	$<1$	154 / 3.61	Dual	13
LongAV-Compass [ 30 ]	284	Hybrid	Direct	General	T / I / V	Video	$>1$	209 / 6.76	Dual	16
Code-Based Generation

Table 1: Comparison of representative video generation benchmarks. Source denotes Real (derived from existing works), Synthetic (constructed without such works), or Hybrid (both). T, I, and V denote text, image, and video inputs; Duration is the average video length in minutes. Dual combines objective and subjective evaluation. Avg. Tokens uses the Qwen3 tokenizer [ 60 ] , while event counts follow T2AV-Compass [ 10 ] . † indicates statistics from partial datasets; N/A and – denote visual-only input and unavailable data, respectively.

Figure 2: Data construction pipeline. (A) Independent tasks are developed from curated text seeds, reference images, and video clips. (B) Paired T2V, I2V, and V2V tasks are derived from shared complete source videos. (C) Both task sets undergo prompt construction, QA checklist development, and annotation of subjects, temporal events, motion types, and three complexity dimensions, yielding 2,104 tasks.

Figure 3: Distributions of (a) video categories, (b) motion types, (c) subject types, (d) event counts, and (e) target durations across 958 records (573 paired sources + 385 independent tasks). A record may have multiple video categories, motion types, and subject types.

Figure 4: Task complexity and prompt length. C, D, and S denote construction, dynamic animation, and state-dependency complexity. (a) Record counts by category and complexity level (1–3), with multi-label records counted in each assigned category. (b) Prompt token distributions across T2V, I2V, and V2V, as in Tab. 1 .

Proprietary Models
Model	Avg.	Operational		Video Quality			Text Alignment		Instr. Follow.	Holistic Video Quality
	Rank $\downarrow$	ES $\uparrow$	TC $\downarrow$	TQ $\uparrow$	AQ $\uparrow$	MS $\uparrow$	L-TVA $\uparrow$	G-TVA $\uparrow$	IF $\uparrow$	CC $\uparrow$	VD $\uparrow$	MQ $\uparrow$	VE $\uparrow$
Claude Opus 5.5	1.600	0.944	29,400	0.115	3.636	0.933	0.522	0.653	0.915	4.047	3.808	3.828	3.894
Claude Sonnet 5.5	1.900	0.936	48,679	0.116	3.587	0.924	0.521	0.648	0.911	4.039	3.861	3.847	3.900
GPT-6.1 Sol	2.600	0.892	13,022	0.117	3.631	0.884	0.501	0.614	0.873	3.953	3.883	3.767	3.861
GPT-6 Astra	4.700	0.864	16,647	0.112	3.561	0.856	0.482	0.594	0.850	3.806	3.756	3.703	3.747

Table 2: Main results on T2V. Metrics are defined in Sec. 3.5 . Avg. Rank averages per-metric ranks across all 15 models, excluding ES and TC, with ties assigned average ranks. Models are ordered by Avg. Rank within each group, with darker cells indicating better ranks. Best and second-best scores within each group are bold and underlined , respectively, using unrounded values.

\uparrow

\downarrow

indicate higher/lower is better.

Proprietary Models
Model	Avg.	Operational		Video Quality			Text Alignment		Image Fidelity		Instr. Follow.	Holistic Video Quality
	Rank $\downarrow$	ES $\uparrow$	TC $\downarrow$	TQ $\uparrow$	AQ $\uparrow$	MS $\uparrow$	L-TVA $\uparrow$	G-TVA $\uparrow$	PF $\uparrow$	SF $\uparrow$	IF $\uparrow$	CC $\uparrow$	VD $\uparrow$	MQ $\uparrow$	VE $\uparrow$
Claude Opus 5.5	1.083	0.960	19,905	0.112	3.754	0.948	0.517	0.625	0.894	0.658	0.915	3.934	3.784	3.755	3.818
Claude Sonnet 5.5	1.917	0.954	28,926	0.114	3.716	0.942	0.514	0.620	0.890	0.657	0.901	3.853	3.718	3.683	3.749
GPT-6 Astra	4.000	0.873	12,689	0.111	3.603	0.866	0.481	0.574	0.834	0.649	0.855	3.692	3.657	3.467	3.591
Claude Haiku 5.5	4.000	0.948	35,606	0.111	3.665	0.938	0.505	0.605	0.857	0.629	0.838	3.493	3.378	3.288	3.380

Table 3: Main results on I2V. Metrics are defined in Sec. 3.5 . Scoring, ranking, and notation follow Tab. 2 .

Panel of Table 4 — Table 4: Main results on V2V. Metrics are defined in Sec. 3.5 . Scoring, ranking, and notation follow Tab. 2 . indicates frame sequence input; all other models receive native video.

Figure 5: Mean holistic video quality across all evaluated tasks and successfully rendered videos for 15 MLLMs. Bars indicate the difference between the two averages.

Figure 6: Distribution of failure types across nine selected models.

Figure 7: Performance changes between adjacent complexity levels, using the notation in Fig. 4 . Values are normalized by metric scale, with positive values indicating performance declines.

Figure 8: Output tokens (a) and estimated API cost (b) versus mean holistic video quality for all 15 models, with selected models labeled. Dashed lines indicate the Pareto frontier.

Metric	Human Alignment	Repeat Stability
Instruction Following	0.796	0.983
Holistic Video Quality	0.783	0.970

Table 5: Gemini 3.8 Flash judge validation. Human Alignment uses Pearson’s

r

, and Repeat Stability uses ICC(A,3) on 300 outputs.

Multimodal Video Generation

See all 29 papers

Oct 26, 2026

VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation

Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a ruleconsistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated with blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos.

Muxi Diao, Wenbin Xu, Lele Yang +21

Video Generation Evaluation Multimodal Agent Evaluation AI Agent Benchmarks

Ask PDF

Jun 4, 2026

V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation

Video-to-video (V2V) generation is difficult to evaluate because outputs must both follow editing instructions and preserve frame-level correspondence with the source video, which existing T2V and I2V metrics do not capture. We introduce V2V-Bench, a 11-dimension benchmark organized into five categories: temporal alignment, structural fidelity, transformation quality, video quality, and semantic alignment. V2V-Bench pairs diverse source videos with challenging editing tasks and evaluates two commercial models, Grok Imagine and Gemini Veo3, and one open-source model, Open Sora 2. Results show complementary model strengths: Grok performs better on editing fidelity, while Veo3 achieves stronger visual quality. On six V2V-specific dimensions, V2V-Bench reaches a Spearman correlation of 0.905 with human judgments.

Tao Liu, Leela Krishna, Gouti Pavan Kumar +2

Centific Global Solutions Inc.

Video-Reference Video Generation Video Editing Benchmarks Video Generation Instruction-Based Video Editing Temporal Consistency in Video Generation Video Generation Evaluation

Ask PDF

Jul 27, 2026

FilmBench: A Film-Grade Benchmark for Cinematic Video Generation

Progress in video generation keeps narrowing the visual gap between AI-generated and professionally produced footage, yet most benchmarks still draw prompts from web sources or LLM templates and score them with untrained, generic multimodal models. More fundamentally, their evaluation taxonomies remain rudimentary (overall visual quality, coarse text alignment and temporal smoothness) rather than the professional Cinematic Language criteria by which films are actually made and judged, so they assess basic video plausibility rather than film-grade craft. We introduce FilmBench, a text-to-video (T2V) and reference-to-video (R2V) benchmark grounded in the professional Cinematic Language of the film- academy tradition and co-developed with directors and faculty from the Beijing Film Academy and the Hujing Digital Media & Entertainment Group film studio. It rests on three choices. First, prompts are reverse-engineered from clips of award-winning films spanning 20 cinematic genres and chosen by professional directors, so every prompt is anchored to a verified live-action reference; the prompts follow real shot lists, and most script multiple shots (1,056 of the 1,169 prompts are multi-shot), unlike prior single-clip benchmarks. Second, evaluation follows a three-level Cinematic taxonomy of 3 axes, 12 components and 35 (T2V) +3 (R2V-only) sub-metrics. Third, we develop an in-house expert-grade automatic evaluation agent and open-source its core suite of Cinematic Language operators (FilmOps). Benchmarking leading video generation models (9 for T2V, 7 for R2V), the evaluator reproduces the human model ranking at model-level Spearman \r{ho} = 0.95 (T2V) and 0.96 (R2V). Scores fall well below prior web-style benchmarks, with two consistent gaps in dynamic aesthetics and a marked single- to multi-shot performance drop that widens for weaker models.

Shengyi Wang, Niantong Li, Guangzheng Hu +27

Alibaba Group · Moku Lab, Hujing Digital Media & Entertainment Group · Beijing Film Academy

Video-Reference Video Generation T2V Generation Multi-Shot Video Generation Video Generation Evaluation

Ask PDF

Abstract

Multimodal Video Generation

Explore similar work

VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation

V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation

FilmBench: A Film-Grade Benchmark for Cinematic Video Generation