cs.CVFeb 25, 2026

UniVBench: Towards Unified Evaluation for Video Foundation Models

Authors: Jianhui Wei, Xiaotian Zhang, Yichen Li, Yuan Wang, Yan Zhang, Ziyi Chen, Zhihang Tang, Wei Xu, 

Organizations: Zhejiang University · ByteDance · Zhejiang Lab

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Video foundation models aim to integrate video understanding, generation, editing, and instruction following within a single framework, making them a central direction for next-generation multimodal systems. However, existing evaluation benchmarks remain fragmented and limited in scope, as they each target a single task, rely on task-specific metrics, and typically use short or simple video clips. As a result, they do not capture the unified capabilities that these models are designed to deliver. To address this gap, we introduce UniVBench, a benchmark purpose-built for evaluating video foundation models across four core abilities: video understanding, video generation, video editing, and a newly proposed task, video reconstruction, which assesses how faithfully a model can reproduce video content it has encountered. Our benchmark substantially expands the complexity of evaluation by incorporating 200 high-quality, diverse and multi-shot videos, each paired with detailed captions, multi-format editing instructions, and reference images. All videos are human-created and carefully validated, offering richer cinematic information than prior benchmarks. In addition, we develop a unified agentic evaluation system (UniV-Eval) that standardizes prompting, instruction parsing, and scoring across all tasks, enabling fair, scalable, and reproducible comparisons of unified video models. By grounding evaluation in instruction-based multi-shot video tasks, UniVBench provides the first framework for measuring the integrated capabilities that video foundation models aim to achieve. Extensive human annotations ensure our evaluation aligns with human judgment, enabling rigorous assessment and accelerating progress toward robust video intelligence.

Figures & tables

Figure 1: Overview of the UniVBench evaluation setting across 8 Dimensions, 21 Sub-Dimensions, and 6 Tasks. Given a source video, T2V synthesizes a video using its ground-truth caption, while V2V reconstructs the video based solely on the model’s self-generated understanding text, enabling a direct diagnosis of perception–generation coupling. UniVBench supports six unified tasks—video captioning ( V2T ), text-to-video generation ( T2V ), reference-image video generation ( R2V ), text-instruction video editing ( TV2V ), reference-image video editing ( RV2V ), and video reconstruction ( V2V ).
Tasks : ①: V2T ②: T2V ③: R2V ④: TV2V ⑤: RV2V ⑥: V2V
BenchmarkMulti-taskMulti-shotCopyright Issue
Benchmarks for Video Understanding
M-VAD [ 46 ]①✗Yes
MPII-MD [ 49 ]①✗Yes
MSR-VTT [ 63 ]①✓Potential
Charades [ 38 ]①✗Yes
Table 1: Comparison of benchmarks for video understanding, generation and editing. Multi-task shows the applicable tasks of the benchmark. Multi-shot indicates that whether the video source and text annotations have multi-shot content. Copyright Issue indicates whether the video sources in the dataset are editable without copyright issue.
BenchmarksStyleSubjectActionBackg.LightingColorSpatial RelationshipCamera
AuroraCap✗✓✗✓✗✗✗✓
VGenEval✓✓✓✓✓✗✓✓
ShotBench✗✗✗✗✓✗✗✓
FETV✗✓✓✓✓✗✗✗
VBench✗✓✓✓✗✓✓✗
VBench2.0✗✓✓✓✗✗✓✓
Table 2: Comparison of cinematic dimensions across different video evaluation benchmarks.
Tasks : ①: V2T ②: T2V ③: R2V ④: TV2V ⑤: RV2V ⑥: V2V
Evaluation MethodFine-grainedMulti-shotMulti-dimensionTask
BLEU [ 34 ]✗✓✗①
CLIPScore [ 17 ]✗✗✓②
CIDEr [ 48 ]✗✓✗①
FVD [ 47 ]✗✗✓②
CLIPSIM [ 22 ]✗✗✓②
Table 3: Comparison of core capabilities across existing evaluation metrics and our proposed agent-based evaluation system. “-” indicates the metric is not applicable to this dimension.
Figure 2: Workflow of UniV-Eval. The system accepts arbitrary inputs within a task setting and performs dynamic evaluation after planning and decomposition. The final results are delivered as a fine-grained checklist, providing traceable feedback for training optimization.
TaskModelsSubjectBackgroundActionCameraColorLightingVideo StyleRelative PositionAverage
Understanding (V2T)Gemini 2.5 Pro ‡54.4%57.8%27.0%54.1%65.8%63.8%65.4%44.4%54.1%
Seed 1.6 ‡46.3%46.9%22.3%42.8%54.6%49.7%45.9%33.8%42.8%
Qwen3-VL-30B §15.8%14.6%6.4%17.6%25.3%27.1%24.1%9.8%17.6%
AuroraCap §16.7%14.4%6.9%17.8%23.9%26.4%25.2%10.8%17.8%
Tarsier2 §34.3%25.0%32.7%20.9%7.7%4.4%20.1%21.9%21.9%
Showo-2 §25.9%22.2%10.6%16.3%13.7%11.3%18.3%12.3%16.3%
Table 4: Performance comparison of different baselines on UniVBench, summarizing results over six tasks, across eight dimensions.
Figure 3: Case Study Analysis of UniVBench in T2V and Reconstruction Task. T2V generation uses the ground truth text of the video, while V2V reconstruction relies on model’s understanidng text. The generated videos are selected from OmniVideo
Figure 4: An example of evaluation using different metrics, where the blue-highlighted part shows that UniV-Eval provides more detailed, traceable validation and assessment.
Figure 5: Human expert annotations used to validate the reliability of UniV-Eval.
Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Examples of T2V generation results across different baselines
Figure A2 : Examples of T2V generation results across different baselines
Figure A3 : Examples of R2V generation results of Seedance-Lite
Figure B4 : The meta-data distribution of video content.
Figure B5 : The script format used to generate the coherent video captions. Red font indicates the content model needs to fill in. Green font indicates the explanation of each field.
Figure B6 : Evaluation case of LLM as judge and human
Figure B7 : Captioning prompts used to generate detailed video captions.
Figure B8 : Captioning prompts used to generate detailed video captions.
Figure B9 : Captioning prompts used to generate detailed video captions.
Figure B10 : Captioning prompts used to generate detailed video captions.
Figure B11 : Captioning prompts used to generate detailed video captions.
Figure B12 : Captioning prompts used to generate detailed video captions.
Figure B13 : Captioning prompts used to generate detailed video captions.
 V2VTV2VR2VRV2VT2VV2T
I/O total tokens25104258981674327534195671413
Times (s)456244554927
Appendix
Table D1: The cost of evaluation
Figure F14 : Captioning prompts used to generate detailed video captions.
Figure F15 : Evaluation prompts used for V2T task.
Figure F16 : Evaluation prompts used for R2V task.
Figure F17 : Evaluation prompts used for TV2V task.
Figure F18 : Evaluation prompts used for RV2V task.
Figure F19 : Evaluation prompts used for V2V task.
Figure F20 : Evaluation Json template used for V2T task.
Figure F21 : Evaluation Json template used for V2T task.
More in this topic

Video Foundation Models

See all 16 papers

Explore similar work

CardsList
Feb 27, 2026

MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation

The evolution of video generation toward complex, multi-shot narratives has exposed a critical deficit in current evaluation methods. Existing benchmarks remain anchored to single-shot paradigms, lacking the comprehensive story assets and cross-shot metrics required to assess long-form coherence and appeal. To bridge this gap, we introduce MSVBench, the first comprehensive benchmark featuring hierarchical scripts and reference images tailored for Multi-Shot Video generation. We propose a hybrid evaluation framework that synergizes the high-level semantic reasoning of Large Multimodal Models (LMMs) with the fine-grained perceptual rigor of domain-specific expert models. Evaluating 20 video generation methods across diverse paradigms, we find that current models--despite strong visual fidelity--primarily behave as visual interpolators rather than true world models. We further validate the reliability of our benchmark by demonstrating a state-of-the-art Spearman's rank correlation of 94.4% with human judgments. Finally, MSVBench extends beyond evaluation by providing a scalable supervisory signal. Fine-tuning a lightweight model on its pipeline-refined reasoning traces yields human-aligned performance comparable to commercial models like Gemini-2.5-Flash.

Haoyuan Shi, Yunxin Li, Nanhao Deng +5

Harbin Institute of Technology (Shenzhen) · Alibaba International Digital Commerce

Video Generation EvaluationMulti-Shot Video GenerationTemporal Consistency in Video Generation
AskPDF
Mar 24, 2026

MVPBench: A Multi-Video Perception Evaluation Benchmark for Multi-Modal Video Understanding

The rapid progress of Large Language Models (LLMs) has spurred growing interest in Multi-modal LLMs (MLLMs) and motivated the development of benchmarks to evaluate their perceptual and comprehension abilities. Existing benchmarks, however, are limited to static images or single videos, overlooking the complex interactions across multiple videos. To address this gap, we introduce the Multi-Video Perception Evaluation Benchmark (MVPBench), a new benchmark featuring 14 subtasks across diverse visual domains designed to evaluate models on extracting relevant information from video sequences to make informed decisions. MVPBench includes 5K question-answering tests involving 2.7K video clips sourced from existing datasets and manually annotated clips. Extensive evaluations reveal that current models struggle to process multi-video inputs effectively, underscoring substantial limitations in their multi-video comprehension. We anticipate MVPBench will drive advancements in multi-video perception.

Purui Bai, Tao Wu, Jiayang Sun +3

MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China · SIST, ShanghaiTech University, Shanghai, China

Video UnderstandingVideo-Language Model EvaluationMultimodal Model EvaluationVideo QA
AskPDF
Apr 17, 2026

UniEditBench: A Unified and Cost-Effective Benchmark for Image and Video Editing via Distilled MLLMs

The evaluation of visual editing models remains fragmented across methods and modalities. Existing benchmarks are often tailored to specific paradigms, making fair cross-paradigm comparisons difficult, while video editing lacks reliable evaluation benchmarks. Furthermore, common automatic metrics often misalign with human preference, yet directly deploying large multimodal models (MLLMs) as evaluators incurs prohibitive computational and financial costs. We present UniEditBench, a unified benchmark for image and video editing that supports reconstruction-based and instruction-driven methods under a shared protocol. UniEditBench includes a structured taxonomy of nine image operations (Add, Remove, Replace, Change, Stroke-based, Extract, Adjust, Count, Reorder) and eight video operations, with coverage of challenging compositional tasks such as counting and spatial reordering. To enable scalable evaluation, we distill a high-capacity MLLM judge (Qwen3-VL-235B-A22B Instruct) into lightweight 4B/8B evaluators that provide multi-dimensional scoring over structural fidelity, text alignment, background consistency, naturalness, and temporal-spatial consistency (for videos). Experiments show that the distilled evaluators maintain strong agreement with human judgments and substantially reduce deployment cost relative to the teacher model. UniEditBench provides a practical and reproducible protocol for benchmarking modern visual editing methods. Our benchmark and the associated reward models are publicly available at https://github.com/wesar1/UniEditBench.

Lifan Jiang, Tianrun Wu, Yuhang Pei +3

Zhejiang University Hangzhou, China

Video Editing BenchmarksImage Editing EvaluationVideo EditingImage Editing
AskPDF