Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Zhenghan Chen; Zekai Shao; Xin Lin; Leting Cheng; Siming Chen

cs.CVOct 20, 2026

Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Authors: Zhenghan Chen, Zekai Shao, Xin Lin, Leting Cheng, Siming Chen

Organizations: Fudan University · Sun Yat-sen University

Abstract

Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.

Figures & tables

Benchmark	#Items	Source	Approach	Task Scope	Input	Output	Duration	Avg. Tokens / Events	Evaluation	#Metrics
Direct Video Generation
VBench [ 22 ]	946	Synthetic	Direct	General	T	Video	$<1$	10 / 1.06	Objective	16
VABench [ 21 ]	1,299	Synthetic	Direct	General	T / I	Video	$<1$	50 / 2.31	Dual	15
T2AV-Compass [ 10 ]	500	Hybrid	Direct	General	T	Video	$<1$	154 / 3.61	Dual	13
LongAV-Compass [ 30 ]	284	Hybrid	Direct	General	T / I / V	Video	$>1$	209 / 6.76	Dual	16
Code-Based Generation

Table 1: Comparison of representative video generation benchmarks. Source denotes Real (derived from existing works), Synthetic (constructed without such works), or Hybrid (both). T, I, and V denote text, image, and video inputs; Duration is the average video length in minutes. Dual combines objective and subjective evaluation. Avg. Tokens uses the Qwen3 tokenizer [ 60 ] , while event counts follow T2AV-Compass [ 10 ] . † indicates statistics from partial datasets; N/A and – denote visual-only input and unavailable data, respectively.

Figure 2: Data construction pipeline. (A) Independent tasks are developed from curated text seeds, reference images, and video clips. (B) Paired T2V, I2V, and V2V tasks are derived from shared complete source videos. (C) Both task sets undergo prompt construction, QA checklist development, and annotation of subjects, temporal events, motion types, and three complexity dimensions, yielding 2,104 tasks.

Figure 3: Distributions of (a) video categories, (b) motion types, (c) subject types, (d) event counts, and (e) target durations across 958 records (573 paired sources + 385 independent tasks). A record may have multiple video categories, motion types, and subject types.

Figure 4: Task complexity and prompt length. C, D, and S denote construction, dynamic animation, and state-dependency complexity. (a) Record counts by category and complexity level (1–3), with multi-label records counted in each assigned category. (b) Prompt token distributions across T2V, I2V, and V2V, as in Tab. 1 .

Proprietary Models
Model	Avg.	Operational		Video Quality			Text Alignment		Instr. Follow.	Holistic Video Quality
	Rank $\downarrow$	ES $\uparrow$	TC $\downarrow$	TQ $\uparrow$	AQ $\uparrow$	MS $\uparrow$	L-TVA $\uparrow$	G-TVA $\uparrow$	IF $\uparrow$	CC $\uparrow$	VD $\uparrow$	MQ $\uparrow$	VE $\uparrow$
Claude Opus 5.5	1.600	0.944	29,400	0.115	3.636	0.933	0.522	0.653	0.915	4.047	3.808	3.828	3.894
Claude Sonnet 5.5	1.900	0.936	48,679	0.116	3.587	0.924	0.521	0.648	0.911	4.039	3.861	3.847	3.900
GPT-6.1 Sol	2.600	0.892	13,022	0.117	3.631	0.884	0.501	0.614	0.873	3.953	3.883	3.767	3.861
GPT-6 Astra	4.700	0.864	16,647	0.112	3.561	0.856	0.482	0.594	0.850	3.806	3.756	3.703	3.747

Table 2: Main results on T2V. Metrics are defined in Sec. 3.5 . Avg. Rank averages per-metric ranks across all 15 models, excluding ES and TC, with ties assigned average ranks. Models are ordered by Avg. Rank within each group, with darker cells indicating better ranks. Best and second-best scores within each group are bold and underlined , respectively, using unrounded values.

\uparrow

\downarrow

indicate higher/lower is better.

Proprietary Models
Model	Avg.	Operational		Video Quality			Text Alignment		Image Fidelity		Instr. Follow.	Holistic Video Quality
	Rank $\downarrow$	ES $\uparrow$	TC $\downarrow$	TQ $\uparrow$	AQ $\uparrow$	MS $\uparrow$	L-TVA $\uparrow$	G-TVA $\uparrow$	PF $\uparrow$	SF $\uparrow$	IF $\uparrow$	CC $\uparrow$	VD $\uparrow$	MQ $\uparrow$	VE $\uparrow$
Claude Opus 5.5	1.083	0.960	19,905	0.112	3.754	0.948	0.517	0.625	0.894	0.658	0.915	3.934	3.784	3.755	3.818
Claude Sonnet 5.5	1.917	0.954	28,926	0.114	3.716	0.942	0.514	0.620	0.890	0.657	0.901	3.853	3.718	3.683	3.749
GPT-6 Astra	4.000	0.873	12,689	0.111	3.603	0.866	0.481	0.574	0.834	0.649	0.855	3.692	3.657	3.467	3.591
Claude Haiku 5.5	4.000	0.948	35,606	0.111	3.665	0.938	0.505	0.605	0.857	0.629	0.838	3.493	3.378	3.288	3.380

Table 3: Main results on I2V. Metrics are defined in Sec. 3.5 . Scoring, ranking, and notation follow Tab. 2 .

Panel of Table 4 — Table 4: Main results on V2V. Metrics are defined in Sec. 3.5 . Scoring, ranking, and notation follow Tab. 2 . indicates frame sequence input; all other models receive native video.

Figure 5: Mean holistic video quality across all evaluated tasks and successfully rendered videos for 15 MLLMs. Bars indicate the difference between the two averages.

Figure 6: Distribution of failure types across nine selected models.

Figure 7: Performance changes between adjacent complexity levels, using the notation in Fig. 4 . Values are normalized by metric scale, with positive values indicating performance declines.

Figure 8: Output tokens (a) and estimated API cost (b) versus mean holistic video quality for all 15 models, with selected models labeled. Dashed lines indicate the Pareto frontier.

Metric	Human Alignment	Repeat Stability
Instruction Following	0.796	0.983
Holistic Video Quality	0.783	0.970

Table 5: Gemini 3.8 Flash judge validation. Human Alignment uses Pearson’s

r

, and Repeat Stability uses ICC(A,3) on 300 outputs.

Multimodal Video Generation

See all 29 papers

Jun 4, 2026

V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation

Video-to-video (V2V) generation is difficult to evaluate because outputs must both follow editing instructions and preserve frame-level correspondence with the source video, which existing T2V and I2V metrics do not capture. We introduce V2V-Bench, a 11-dimension benchmark organized into five categories: temporal alignment, structural fidelity, transformation quality, video quality, and semantic alignment. V2V-Bench pairs diverse source videos with challenging editing tasks and evaluates two commercial models, Grok Imagine and Gemini Veo3, and one open-source model, Open Sora 2. Results show complementary model strengths: Grok performs better on editing fidelity, while Veo3 achieves stronger visual quality. On six V2V-specific dimensions, V2V-Bench reaches a Spearman correlation of 0.905 with human judgments.

Tao Liu, Leela Krishna, Gouti Pavan Kumar +2

Centific Global Solutions Inc.

Video-Reference Video Generation Video Editing Benchmarks Video Generation Instruction-Based Video Editing Temporal Consistency in Video Generation Video Generation Evaluation

Ask PDF

Jul 27, 2026

FilmBench: A Film-Grade Benchmark for Cinematic Video Generation

Progress in video generation keeps narrowing the visual gap between AI-generated and professionally produced footage, yet most benchmarks still draw prompts from web sources or LLM templates and score them with untrained, generic multimodal models. More fundamentally, their evaluation taxonomies remain rudimentary (overall visual quality, coarse text alignment and temporal smoothness) rather than the professional Cinematic Language criteria by which films are actually made and judged, so they assess basic video plausibility rather than film-grade craft. We introduce FilmBench, a text-to-video (T2V) and reference-to-video (R2V) benchmark grounded in the professional Cinematic Language of the film- academy tradition and co-developed with directors and faculty from the Beijing Film Academy and the Hujing Digital Media & Entertainment Group film studio. It rests on three choices. First, prompts are reverse-engineered from clips of award-winning films spanning 20 cinematic genres and chosen by professional directors, so every prompt is anchored to a verified live-action reference; the prompts follow real shot lists, and most script multiple shots (1,056 of the 1,169 prompts are multi-shot), unlike prior single-clip benchmarks. Second, evaluation follows a three-level Cinematic taxonomy of 3 axes, 12 components and 35 (T2V) +3 (R2V-only) sub-metrics. Third, we develop an in-house expert-grade automatic evaluation agent and open-source its core suite of Cinematic Language operators (FilmOps). Benchmarking leading video generation models (9 for T2V, 7 for R2V), the evaluator reproduces the human model ranking at model-level Spearman \r{ho} = 0.95 (T2V) and 0.96 (R2V). Scores fall well below prior web-style benchmarks, with two consistent gaps in dynamic aesthetics and a marked single- to multi-shot performance drop that widens for weaker models.

Shengyi Wang, Niantong Li, Guangzheng Hu +27

Alibaba Group · Moku Lab, Hujing Digital Media & Entertainment Group · Beijing Film Academy

Video-Reference Video Generation T2V Generation Multi-Shot Video Generation Video Generation Evaluation

Ask PDF

Oct 1, 2026

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standards. Existing evaluation benchmarks, however, predominantly assess visual quality, aesthetic appeal and physical plausibility, while paying limited attention to text, an essential medium for conveying information in everyday scenes. A generated video may appear visually compelling and feature lifelike subjects, yet still render the text within the scene incorrectly. To address this overlooked dimension, we introduce VTR-Bench, a systematic benchmark for evaluating the Visual Text Rendering capabilities of video generation models. VTR-Bench situates text within concrete application scenarios, such as advertisements and scientific videos, with 300 carefully constructed prompts spanning five scenario categories. We develop an automated evaluation pipeline with human alignments that separately assesses text fidelity through carrier-specific transcription and scene and motion requirements through a prompt-specific chain of query. Beyond evaluation, we introduce a Keyframe-Guided Agentic Framework in which a Director agent coordinates image and video generation with visual evaluation, guiding iterative refinement and candidate selection through visual feedback. Experiments on 11 state-of-the-art models reveal widespread difficulties in accurately rendering scene text, with the best-performing model recording an overall word error rate (WER) of 0.250. We further analyze text rendering failures to characterize the challenges faced by current video generation models. These findings highlight visual text rendering as a key challenge for video generation and demonstrate a practical path toward improvement. Code is available at https://github.com/hardenyu21/VTR-Bench.

Yu Huang, Jungang Li, Zhiyuan Wang +8

Department of Data Science, City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Westlake University +3

Agentic Video Generation Video Generation Video Generation Evaluation

Ask PDF

Abstract

Multimodal Video Generation

Explore similar work

V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation

FilmBench: A Film-Grade Benchmark for Cinematic Video Generation

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation