SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

Keyu Tu; Zhuowei Chen; Mengqi Huang; Yuxin Wang; Jiahao Zhu; Zhendong Mao; Yongdong Zhang

cs.CVAug 18, 2026

SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

Authors: Keyu Tu, Zhuowei Chen, Mengqi Huang, Yuxin Wang, Jiahao Zhu, Zhendong Mao, Yongdong Zhang

Organizations: University of Science and Technology of China · Sun Yat-sen University

Abstract

We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a vision-language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.

Figures & tables

Figure 1: Demonstration of Semantic Task Completion Video Generation, which focuses not on the transformation process but solely on outcome achievement and semantic grounding.

Figure 2: Representative SemComp-Data instances. Each example shows a reference frame, its outcome-centric video clip, and the brief instruction for compact presentation; every instance also includes a detailed instruction.

Figure 3: Overview of the four-stage SemComp-Data curation pipeline. Panels (a)–(d) show Candidate Filtering, State Mining, Video Extension, and Instruction Structuring, respectively. The right column shows the corresponding stage outputs, and the bottom row presents the final data triplet.

Figure 4: Statistics of SemComp-Data: (a) distribution of alignment types across domains, with distinct textures denoting different types; (b) high-frequency words in the instructions; and (c) distribution of evaluation instances across the six domains.

Models	$A_{\mathrm{or}}$	$A_{\mathrm{sg}}$	$A_{\mathrm{gec}}$	$A_{\mathrm{gvc}}$	OA Score
Seedance 2.0 [ 5 ]	0.839	0.744	0.444	0.594	20.0%
Wan2.2-TI2V-5B [ 28 ]	0.589	0.400	0.689	0.922	23.3%
Wan2.2-I2V-A14B [ 28 ]	0.800	0.528	0.628	0.789	28.3%
CogVideoX1.5-5B-I2V [ 30 ]	0.550	0.389	0.506	0.744	14.4%
SkyReels-V2-I2V-14B-720P [ 7 ]	0.733	0.489	0.522	0.772	22.8%
HY † -1.5-720P-I2V	0.878	0.706	0.583	0.794	37.8%

Table 1: SemComp-Core Outcome Achievement with detailed instructions.

A_{\mathrm{or}}

A_{\mathrm{sg}}

A_{\mathrm{gec}}

, and

A_{\mathrm{gvc}}

denote pass rates for outcome realization, semantic grounding, grounded entity consistency, and global visual continuity, respectively. The OA Score is the joint pass rate across four criteria. † HY denotes HunyuanVideo [ 27 ] . Bold and underlined values indicate the best and second-best values, respectively.

Models	$G_{\mathrm{pp}}$	$G_{\mathrm{vc}}$	$G_{\mathrm{afr}}$	$G_{\mathrm{wsc}}$	$G_{\mathrm{ti}}$	GR Score
Seedance 2.0 [ 5 ]	0.883	0.994	0.994	0.739	0.978	91.8%
Wan2.2-TI2V-5B [ 28 ]	0.794	0.978	0.889	0.722	0.889	85.4%
Wan2.2-I2V-A14B [ 28 ]	0.911	0.972	0.967	0.672	0.928	89.0%
CogVideoX1.5-5B-I2V [ 30 ]	0.944	0.811	0.772	0.583	0.828	78.8%
SkyReels-V2-I2V-14B-720P [ 7 ]	0.778	0.922	0.739	0.328	0.789	71.1%
HY † -1.5-720P-I2V	0.800	0.939	0.883	0.472	0.861	79.1%

Table 2: SemComp-Core Generation Reliability with detailed instructions. Each

G_{c}

denotes the pass rate for a reliability criterion, and the GR Score is the mean across five criteria. † HY denotes HunyuanVideo [ 27 ] . Bold and underlined values indicate the best and second-best values.

Model	Modality	Instruction Type	$A_{\mathrm{or}}$	$A_{\mathrm{sg}}$	$A_{\mathrm{gec}}$	$A_{\mathrm{gvc}}$	OA Score
Wan2.2-A14B [ 28 ]	I2V	Detailed	0.800	0.528	0.628	0.789	28.3%
	T2V	Detailed	0.889	0.522	0.317	0.117	4.4%
	T2V	Brief	0.389	0.111	0.806	0.250	0.6%
CogVideoX1.5-5B [ 30 ]	I2V	Detailed	0.550	0.389	0.506	0.744	14.4%
	T2V	Detailed	0.567	0.272	0.361	0.161	5.0%
	T2V	Brief	0.567	0.150	0.678	0.161	0.6%

Table 3: Outcome Achievement on SemComp-Core across modalities and instruction settings. Within each model family, the I2V and T2V variants use checkpoints of the same parameter scale. Each

A_{c}

denotes a criterion-level pass rate, and the OA Score is the joint pass rate. † HY denotes HunyuanVideo [ 27 ] .

Figure 5: Generated-video examples. The outcome-centric video clip and the reference image are extracted from the same full-context video.

Configuration Group	Keyword Strings
news_keywords (13)	News; Briefing; Coverage; Current Affairs; Deep Dive; Documentary; Exclusive; Headline; Interview; Live; Live Update; Press Conference; Report.
movie_keywords (13)	Behind the scenes; Blooper; Casting; Movie Clip; Playback; Preview; Review; Spoiler; Teaser; TV Series; Talk; Talks; TalkShow.
entertainment_keywords (19)	ASMR; Celebrity; Challenge; Concert; Gaming; Highlights; Music Video; Podcast; Prank; Reaction; Reacts; Travel; Unboxing; Variety Show; Vlog; HBO; SportsCenter; NFL; NBA.

Table S1: Complete title-filtering keyword configuration.

Domain	Category	Definition
Food and Cooking	Dish Making	A dish or food product is completed from ingredients.
	Food Transformation	The same food item undergoes a visible cooking-related state change.
	Food Plating	Food elements are arranged into a complete presentation.
Beauty and Fashion	Styling	The appearance of the same person is visibly restyled.
	Tool Cleaning	A beauty or fashion tool is cleaned to a reusable condition.
	Try-on	An external appearance reference is applied to a subject.

Table S2: Domain–category taxonomy and category definitions used in SemComp-Data.

Category	Reference State	Outcome State
Dish Making	Ingredients or an incomplete food preparation are visible.	A recognizable completed dish or food product is visible.
Food Transformation	A food item is visible before a cooking-related state change.	The same food item exhibits the intended visible state change.
Food Plating	Food components are present before final arrangement.	The food components form a complete plated presentation.
Styling	A person is visible before the target styling change.	The same person visibly exhibits the completed styling result.
Tool Cleaning	A beauty or fashion tool is visibly soiled or not ready for reuse.	The same tool is visibly clean and reusable.
Try-on	The subject and an external appearance reference are identifiable.	The subject visibly exhibits the referenced garment, accessory, or appearance.

Table S3: Category-specific reference and outcome state definitions used for State Mining.

Alignment Type	Operational Definition	Typical Relevant Attributes
Object Element	Grounds the outcome in task-relevant object categories, components, ingredients, materials, quantities, or colors appearing in the reference, without requiring the exact appearance of a specific object instance.	Object category, composition, count, material, and color palette.
Identity	Requires the depicted person to remain identifiable across the reference and generated outcome.	Person identity, face appearance, body appearance, and pose.
Object Appearance	Grounds the outcome in the instance-level visual appearance of a specific reference object.	Object category, color palette, material, shape structure, surface appearance, and graphic details.
Scene	Grounds the outcome in the organization or context of the reference scene.	Scene type, scene layout, spatial relation, background context, and viewpoint.
Complete preservation-attribute vocabulary (17): object_category , object_composition , object_count , material , color_palette , shape_structure , surface_appearance , graphic_details , person_identity , face_appearance , body_appearance , pose , scene_type , scene_layout , spatial_relation , background_context , and viewpoint .

Table S4: Operational definitions of the four alignment types and the complete preservation-attribute vocabulary. Attribute selection remains instance-specific.

Figure S1 : Common system prompt used for SemComp-Bench evaluation.

Figure S2 : Outcome Achievement prompt for Outcome Realization (

Q_{\mathrm{OA}}^{1}

) and Semantic Grounding (

Q_{\mathrm{OA}}^{2}

Figure S3 : Outcome Achievement prompt for Grounded Entity Consistency (

Q_{\mathrm{OA}}^{3}

) and Global Visual Continuity (

Q_{\mathrm{OA}}^{4}

Figure S4 : Generation Reliability prompt for Physical Plausibility (

Q_{\mathrm{GR}}^{1}

Figure S5 : Generation Reliability prompt for Visual Clarity, Artifact-Free Rendering, Within-Scene Spatiotemporal Coherence, and Text and Interface Integrity (

Q_{\mathrm{GR}}^{2}

–

Q_{\mathrm{GR}}^{5}

Figure S7 : Additional generated-video comparisons on four SemComp-Core instances. Outputs from all seven I2V models in each block were generated using the shared reference condition and detailed instruction; the paired brief instruction is displayed only for compact presentation. Temporally ordered output frames illustrate differences in both outcome achievement and generation reliability.

Model	$\sigma(A_{\mathrm{or}})$	$\sigma(A_{\mathrm{sg}})$	$\sigma(A_{\mathrm{gec}})$	$\sigma(A_{\mathrm{gvc}})$	$\sigma(\text{OA Score})$
Seedance 2.0	0.96	0.96	2.55	7.52	4.41
Wan2.2-TI2V-5B	4.19	4.41	3.85	0.96	3.33
Wan2.2-I2V-A14B	1.67	2.55	5.09	3.47	2.89
CogVideoX1.5-5B-I2V	3.33	4.19	2.55	0.96	0.96
SkyReels-V2-I2V-14B-720P	1.67	3.47	3.47	2.55	0.96
HY † -1.5-720P-I2V	2.55	2.55	2.89	1.92	3.47

Table S6 : Sample standard deviations for SemComp-Core Outcome Achievement under detailed instructions (pp; three runs). † HY denotes HunyuanVideo.

Model	$\sigma(G_{\mathrm{pp}})$	$\sigma(G_{\mathrm{vc}})$	$\sigma(G_{\mathrm{afr}})$	$\sigma(G_{\mathrm{wsc}})$	$\sigma(G_{\mathrm{ti}})$	$\sigma(\text{GR Score})$
Seedance 2.0	3.33	0.96	0.96	1.92	0.96	1.58
Wan2.2-TI2V-5B	0.96	2.55	6.74	12.95	0.96	3.95
Wan2.2-I2V-A14B	5.85	0.96	1.67	1.92	4.19	1.53
CogVideoX1.5-5B-I2V	2.55	13.47	6.31	8.82	8.55	7.12
SkyReels-V2-I2V-14B-720P	0.96	3.47	6.74	11.10	3.85	4.53
HY † -1.5-720P-I2V	8.82	1.92	0.00	6.94	0.96	1.35

Table S7 : Sample standard deviations across three evaluation runs for the SemComp-Core Generation Reliability results under detailed instructions. All values are in percentage points. † HY denotes HunyuanVideo.

Model	Modality	Instruction	$\sigma(A_{\mathrm{or}})$	$\sigma(A_{\mathrm{sg}})$	$\sigma(A_{\mathrm{gec}})$	$\sigma(A_{\mathrm{gvc}})$	$\sigma(\text{OA Score})$
Wan2.2-A14B	I2V	Detailed	1.67	2.55	5.09	3.47	2.89
	T2V	Detailed	2.55	4.19	1.67	4.41	2.55
	T2V	Brief	0.96	0.96	2.55	7.26	0.96
CogVideoX1.5-5B	I2V	Detailed	3.33	4.19	2.55	0.96	0.96
	T2V	Detailed	3.33	0.96	2.55	0.96	1.67
	T2V	Brief	2.89	1.67	0.96	3.47	0.96

Table S8 : Sample standard deviations across three evaluation runs for the Outcome Achievement conditioning study. All values are in percentage points. † HY denotes HunyuanVideo.

Video Generation

Video generation studies models that create and edit moving images from text, images, audio, and structured controls.

See all 364 papers

Aug 20, 2026

VGI-Bench: Probing Visual Intelligence in Video Generation Models

Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of visual reasoning capabilities of video generation models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our analysis further explore the output failure modes, input condition sensitivity, performance transfer boundary from synthetic fine-tuning, and internal denoising perspective revealing limited self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-bench will help stimulate the development of next-generation video generation models. Website: https://hexuan21.github.io/VGI-Bench/

Xuan He, Cong Wei, Yuhao Cheng +20

University of Illinois Urbana Champaign · University of Waterloo · Vector Institute +6

Video Generation Evaluation

Ask PDF

Jun 4, 2026

V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation

Video-to-video (V2V) generation is difficult to evaluate because outputs must both follow editing instructions and preserve frame-level correspondence with the source video, which existing T2V and I2V metrics do not capture. We introduce V2V-Bench, a 11-dimension benchmark organized into five categories: temporal alignment, structural fidelity, transformation quality, video quality, and semantic alignment. V2V-Bench pairs diverse source videos with challenging editing tasks and evaluates two commercial models, Grok Imagine and Gemini Veo3, and one open-source model, Open Sora 2. Results show complementary model strengths: Grok performs better on editing fidelity, while Veo3 achieves stronger visual quality. On six V2V-specific dimensions, V2V-Bench reaches a Spearman correlation of 0.905 with human judgments.

Tao Liu, Leela Krishna, Gouti Pavan Kumar +2

Centific Global Solutions Inc.

Video-Reference Video Generation Video Editing Benchmarks Video Generation Instruction-Based Video Editing Temporal Consistency in Video Generation Video Generation Evaluation

Ask PDF

Oct 20, 2026

Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.

Zhenghan Chen, Zekai Shao, Xin Lin +2

Fudan University · Sun Yat-sen University

Multimodal Video Generation Video Generation Evaluation T2V Generation Image-to-Video Generation Video-Reference Video Generation

Ask PDF

Abstract

Video Generation

Explore similar work

VGI-Bench: Probing Visual Intelligence in Video Generation Models

V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation

Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V