cs.CVMar 24, 2026
SaveMVPBench: A Multi-Video Perception Evaluation Benchmark for Multi-Modal Video Understanding
Organizations: MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China · SIST, ShanghaiTech University, Shanghai, China
Abstract
The rapid progress of Large Language Models (LLMs) has spurred growing interest in Multi-modal LLMs (MLLMs) and motivated the development of benchmarks to evaluate their perceptual and comprehension abilities. Existing benchmarks, however, are limited to static images or single videos, overlooking the complex interactions across multiple videos. To address this gap, we introduce the Multi-Video Perception Evaluation Benchmark (MVPBench), a new benchmark featuring 14 subtasks across diverse visual domains designed to evaluate models on extracting relevant information from video sequences to make informed decisions. MVPBench includes 5K question-answering tests involving 2.7K video clips sourced from existing datasets and manually annotated clips. Extensive evaluations reveal that current models struggle to process multi-video inputs effectively, underscoring substantial limitations in their multi-video comprehension. We anticipate MVPBench will drive advancements in multi-video perception.
Figures & tables
Fig. 1 : Challenges in managing Multi-Video inputs. Previous benchmarks are primarily designed to address straightforward visual text problems, such as describing the content and associations of static images. While video description provides additional information to enhance the detail of content descriptions, it remains challenging to effectively manage comparisons between multiple related videos.
Fig. 2 : Design principles of MVPBench. A key feature of MVPBench is its ability to consider both the multiplicity of evaluation inputs and the temporal characteristics of evaluation videos.
| Subdomain | Subtask | Source | Prompt |
|---|---|---|---|
| Temporal Segment | Tem w/ | YouCook2 [ 11 ] | Sort these video clips according to the recipe’s textual description. Recipe Description: |
| Cap | step0: add water to a pan. step1: stir in the pack of sauce. step2: stir and cook until the soup boils | ||
| Splicing | Tem w/o | Manual | Sort the given video clips. |
| Cap | Collection | According to their content or logical sequence. | |
| Content Assessment | Daily life | EPIC-Skills [ 12 ] | Your task is to compare these two clips. |
| action evaluation | Determine which one has better task and action completion. |
TABLE I : Prompt examples of MVPBench.
Fig. 3 : Statistics of MVPBench. The benchmark includes 14 tasks in 5 domains, ranging from low-level pattern comparison to mid-level temporal logic reasoning, and extending to high-level visual content understanding.
| Benchmarks | #Videos | #QA Pairs | Open. | Ego. | Com. | Multi. | Gen. | Data source |
|---|---|---|---|---|---|---|---|---|
| MVBench [ 9 ] | 3,641 | 4,000 | ✓ | ✓ | ✓ | ✗ | ✗ | existing datasets |
| Video-Bench [ 27 ] | 5,917 | 17,036 | ✓ | ✗ | ✓ | ✗ | ✗ | existing datasets |
| EgoSchema [ 24 ] | 5,063 | 5,063 | ✗ | ✓ | ✗ | ✗ | ✗ | ego-centric video |
| AutoEval-Video [ 28 ] | 327 | 327 | ✓ | ✗ | ✗ | ✗ | ✗ | Youtube |
| TempCompass [ 29 ] | 410 | 7,540 | ✓ | ✗ | ✓ | ✗ | ✗ | ShutterStock |
| Video-MME [ 10 ] | 900 | 2,700 | ✓ | ✗ | ✓ | ✗ | ✗ | Youtube |
TABLE II : The comparison of various benchmarks: whether the benchmark incorporates videos sourced from diverse open domains (Open.), whether the benchmark incorporates egocentric videos(Ego.), whether the benchmark incorporates complex tasks spanning multiple hierarchical levels of perception(Com.), whether the benchmark incorporates multi-video comprehension category (Multi.) and whether the benchmark incorporates videos synthesized by the generative model(Gen.)
Fig. 4 : Task introduction of MVPBench . Zoom in for better view.
| Model | Tem w/ Cap | Tem w/o Cap | Daily-Action | Pro-Action | Weather | Clarity | Brightness | Forensic |
| Human | 95.16 | 80.01 | 80.96 | 85.72 | 83.34 | 90.00 | 95.71 | 97.00 |
| Closed-source Multimodal LLMs | ||||||||
| Gemini-2.5-Flash [ 33 ] | 30.11 | 11.54 | 43.08 | 28.74 | 18.79 | 35.45 | 13.00 | 40.00 |
| GPT-4o [ 30 ] | 28.47 | 10.21 | 61.90 | 37.14 | 12.51 | 40.00 | 10.00 | 52.00 |
| Open-source Multimodal LLMs | ||||||||
| MiniCPM-V 4.5 [ 34 ] | 4.23 | -9.03 | -9.52 | -2.86 | -15.37 | -10.01 | 12.69 | 31.00 |
TABLE III : Results of different models on MVPBench. The best performance in each task is in-bold . Notably, the table presents normalized proficiency scores for each model rather than observed accuracy . Consequently, negative scores may occur, indicating model performance below random expectation levels for specific tasks. The ”Overall” metric denotes a weighted average across models, with weights proportional to the information gain potential of each respective task.
| Baseline | Tem w/ Cap | Tem w/o Cap | Gait | Cinematographic | Olympic | Dance |
|---|---|---|---|---|---|---|
| Model | 21.13% | 28.30% | 49.00% | 68.00% | 78.00% | 46.25% |
| Random Expectation | 10.56% | 16.67% | 33.33% | 33.33% | 50.00% | 33.33% |
TABLE IV : The result of first-option frequency evaluated on Qwen2.5-VL-72B and its random expectation. The numbers represent MLLM-Human Agreement. The model selects the first video option at a frequency 1.5–2 times the random expectation threshold, empirically validating the presence of Positional Bias in our experimental framework.
TABLE V : Ablation study across the single-video style benchmark experiments on all evaluation tasks. All experimental values represent the mean MLLM-Human Agreement scores [ 41 ] , averaged across models with native multi-video processing capabilities (Gemini-2.5-Flash, GPT-4o, and LLaVA-OneVision-Qwen2).
| Subtask | Prompt example |
|---|---|
| Caption-guided temporal segment splicing | You are tasked with sorting video clips that correspond to steps in a recipe. Each video is numbered sequentially based on the order of input, such as 1, 2, 3, and so on. The videos are currently in a shuffled order, and your goal is to arrange them according to the correct sequence described in the recipe. The final output should be a sequence of numbers representing the correct order of the videos without any additional information. Here is the information provided: Video 1: <video>Video 2: <video>Video 3: <video> Recipe Description: Step 0: Put foil over the pan. Step 1: Put kalbi strips in the pan. Step 2: Put in the broiler. Please determine and provide the correct sequence of video numbers. |
| Caption-free temporal segment splicing | You are tasked with sorting video clips based on their content or logical sequence. Each video is numbered sequentially based on the order of input, such as 1, 2, 3, and so on. The videos are currently in a shuffled order, and your goal is to determine the correct order of these video clips. The final output should be a sequence of numbers representing the correct order of the videos without any additional information. Here are the video clips: Video 1: <video>Video 2: <video>Video 3: <video> Please determine and provide the correct sequence of video numbers. |
| Daily life action evaluation | Please watch the two video clips provided, labeled as Video 1 and Video 2. Each clip contains similar content and tasks being performed. Your task is to compare these clips and evaluate which one demonstrates better task and action completion. After reviewing both videos, select the number corresponding to the video that performed the task more effectively. Your response should be either the number 1 or 2, without any additional commentary or explanation. Here are the video clips: Video 1: <video>Video 2: <video> |
| Professional action evaluation | Please watch the two video clips provided, labeled as Video 1 and Video 2. Each clip contains similar content and tasks being performed. Your task is to compare these clips and evaluate which one demonstrates better task and action completion. After reviewing both videos, select the number corresponding to the video that performed the task more effectively. Your response should be either the number 1 or 2, without any additional commentary or explanation. Here are the video clips: Video 1: <video>Video 2: <video> |
| Weather condition evaluation | Please evaluate the three provided video clips, labeled as Video 1, Video 2, and Video 3. Each clip depicts the same type of weather, but with varying intensity levels: severe, moderate, and mild. Your task is to examine the weather conditions in each video to identify which one exhibits the most severe weather. After assessing the videos, provide a single digit corresponding to the video number that shows the most severe weather conditions. Your response should be concise, consisting solely of the digit without any additional commentary or explanation. Here are the video clips: Video 1: <video>Video 2: <video>Video 3: <video> |
TABLE VI : Prompt examples of Temporal Segment Splicing tasks and Content Assessment tasks.
| Subtask | Prompt example |
|---|---|
| Video clarity evaluation | Please evaluate the three provided video clips, labeled as Video 1, Video 2, and Video 3, focusing on their video quality, specifically in terms of clarity and smoothness. Your task is to rank these videos from highest to lowest quality based solely on these criteria. Indicate your ranking by listing only the corresponding numbers of each video in the order of quality, such as ”3, 2, 1” if Video 3 is the highest quality, followed by Video 2, and then Video 1. Ensure your response includes only the ranking numbers without any additional explanation or commentary. Here are the video clips: Video 1: <video>Video 2: <video>Video 3: <video> |
| Video brightness evaluation | Please analyze the brightness levels of the three provided video clips, labeled as Video 1, Video 2, and Video 3, each with distinct lighting and shooting conditions. Your task is to determine which video clip exhibits the lowest brightness and was filmed under the darkest conditions. After your assessment, provide your answer as a single digit representing the video number with the lowest brightness. Your response should be concise, consisting solely of the digit without any additional commentary or explanation. Here are the video clips: Video 1: <video>Video 2: <video>Video 3: <video> |
| Video forensic detection | Please evaluate the three provided video clips, labeled as Video 1, Video 2, and Video 3. Each clip contains similar content, but two are AI-generated, and one is authentic. Your task is to compare these videos and determine which one is real. After reviewing all three, respond with the number of the video you believe is genuine. Your response should be a single digit, representing the video number, without any additional commentary or explanation. Here are the video clips: Video 1: <video>Video 2: <video>Video 3: <video> |
| Multiview visual perception pairing | Please analyze the given reference video, Ref Video: <video> Compare it with the three candidate videos provided: Video 1: <video>Video 2: <video>Video 3: <video> The task is to identify the candidate video that corresponds to the reference video. The reference video is presented from a first-person perspective, while the correct candidate video is shown from a second-person perspective of the same scene. The other two candidate videos are considered distractors and do not match the reference video. Please examine each video closely and identify the candidate video that accurately matches the reference video. Provide your answer solely as a single digit that corresponds to the number of the correct candidate video, without any additional explanation or information. |
| Common sense judgment of physical laws | Please evaluate the three provided video clips, labeled as Video 1, Video 2, and Video 3. Among these clips, two contain elements that clearly violate natural physical laws, while one depicts a scenario consistent with typical physical principles. Your task is to identify the video that aligns with normal physical laws. After reviewing all three, respond with the number of the video indicating the number of the video that follows the laws of physics. Your response should be a single digit, representing the video number, without any additional commentary or explanation. Here are the video clips: Video 1: <video>Video 2: <video>Video 3: <video> |
TABLE VII : Prompt examples of Video Quality Assessment tasks and Video Logic Inference tasks.
| Subtask | Prompt example |
|---|---|
| Gait recognition matching | Please analyze the given reference video, Ref Video: <video> Compare it with the three candidate videos provided: Video 1: <video>Video 2: <video>Video 3: <video> The task is to identify which candidate video features the same individual as the reference video, based on their gait, which should be identical. The other two candidate videos are distractors, featuring different individuals with different gaits. Please examine each video closely and identify the candidate video that accurately matches the reference video. Provide your answer solely as a single digit that corresponds to the number of the correct candidate video, without any additional explanation or information. |
| Shooting style matching | Please analyze the given reference video, Ref Video: <video> Compare it with the three candidate videos provided: Video 1: <video>Video 2: <video>Video 3: <video> Your task is to identify which candidate video shares the same filming style as the reference video, focusing specifically on camera movement and video processing techniques. Among the three candidate videos, only one will match the reference video in these aspects, while the other two will not. Please examine each video closely and identify the candidate video that accurately matches the reference video. Provide your answer solely as a single digit that corresponds to the number of the correct candidate video, without any additional explanation or information. |
| Olympic sports matching | Please analyze the given reference video, Ref Video: <video> Compare it with the three candidate videos provided: Video 1: <video>Video 2: <video>Video 3: <video> Your task is to determine which of the candidate videos depicts the same sports activity as the reference video. Note that one candidate video matches the reference video while the others serves as a distractor. Please examine each video closely and identify the candidate video that accurately matches the reference video. Provide your answer solely as a single digit that corresponds to the number of the correct candidate video, without any additional explanation or information. |
| Dance style matching | Please analyze the given reference video, Ref Video: <video> Compare it with the three candidate videos provided: Video 1: <video>Video 2: <video>Video 3: <video> Your task is to determine which of the candidate videos features the same dance type as the reference video. Note that one candidate video matches the reference video while the others serves as a distractor. Please examine each video closely and identify the candidate video that accurately matches the reference video. Provide your answer solely as a single digit that corresponds to the number of the correct candidate video, without any additional explanation or information. |
TABLE VIII : Prompt examples of Similar Video Pairing tasks.
Fig. 5 : More visualization of data in MVPBench. Examples of Temporal Segment Splicing tasks and Content Assessment tasks.
Fig. 6 : More visualization of data in MVPBench. Examples of Video Quality Assessment tasks and Video Logic Inference tasks.
Fig. 7 : More visualization of data in MVPBench. Examples of Similar Video Pairing tasks.