cs.CVAug 26, 2026

Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios

Authors: Hongbo Liu, Peixian Chen, Sihan Liu, Peiyuan Zhang, Kai Zou, Dian Zheng, Xiaoxing Hu, Yuhao Dong, 

Organizations: TJU · Tencent Youtu Lab · SJTU · Tencent Hunyuan · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal Large Language Models (MLLMs) have shown strong performance in video understanding. However, their ability to follow instructions in this domain remains under-explored. Real-world video understanding requires models not only to interpret video content correctly, but also to satisfy diverse user-specified constraints. Existing benchmarks focus primarily on task accuracy rather than instruction adherence, leaving this capability insufficiently evaluated. To address this gap, we introduce Video-IFBench, a comprehensive benchmark for evaluating instruction following in video understanding, where models must satisfy diverse user-specified constraints, including those grounded in visual and audio content. We develop an instruction taxonomy with four templates, including single-task, multi-task, selection, and nested instructions, covering 32 task types and 39 manually designed constraint categories spanning both semantic and format requirements. To reduce annotation cost, we build a semi-automatic data construction pipeline that combines MLLMs, programmatic processing, and human verification, resulting in 1.5K samples. We conduct a large-scale evaluation of more than 20 recent MLLMs and show that video instruction following remains challenging for current models, especially for instructions with many constraints, semantic constraints, or complex conditional structures that require selecting the correct branch or path based on video content. We hope our work will facilitate future research on instruction following in video understanding scenarios.

Figures & tables

Figure 1: The data construction pipeline of Video-IFBench. Each curated video is annotated with a global description and fine-grained timestamped facts, which are used to generate Single/Multi instructions and condition-based Selection/Nested instructions. Each instruction is paired with a checklist for LLM-based or programmatic verification. More examples are provided in Appendix D .
Figure 2: Dataset-level statistics of Video-IFBench.
Figure 3: Instruction and constraint statistics of Video-IFBench.
ModelSingleMultiSelectionNestedOverall
TCSRTISRTCSRTISRTCSRTISRTCSRTISRTCSRTISR
Proprietary Models
Gemini-3-Pro Google DeepMind (2025)79.652.388.658.868.559.253.74676.554.5
Gemini-3-Flash Google DeepMind (2025)76.749.287.656.463.954.839.930.772.249.5
Doubao-Seed-2.0-Pro-260215 ByteDance Seed Team (2026)76.744.687.151.146.538.217.814.465.740.8
GPT-5.4 Singh et al. (2025)72.834.782.443.221.116.912.17.657.430.0
Table 1: Main results on Video-IFBench. We report TCSR and TISR for each instruction type and the overall benchmark. Best results within each model group are in bold.
Figure 4: Performance distribution on 1515 most challenging constraints. Colored bars show the performance range of different model groups, and points denote individual models.
Figure 5: Model scaling and semantic-format constraint performance.
Table 2: Scaling analysis of task and constraint complexity. The left table reports task pass rate across task-count ranges, and the right table reports TISR across constraint-count ranges.
Figure 6: Depth sensitivity on Nested instructions.
Table 3: Task pass rate on Selection instructions using the 3-condition-plus-else subset.
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

ModelSingleMultiSelectionNestedOverall
TCSRTISRTCSRTISRTCSRTISRTCSRTISRTCSRTISR
Proprietary Models
Gemini-3-Pro [ 12 ]86.664.187.753.373.163.057.851.580.1 (+3.6)58.7 (+4.2)
Gemini-3-Flash [ 12 ]83.055.887.853.467.157.343.835.375.7 (+3.5)52.2 (+2.7)
Doubao-Seed-2.0-Pro-260215 [ 2 ]84.154.385.948.747.741.220.817.868.6 (+2.9)44.7 (+3.9)
GPT-5.4 [ 37 ]78.042.580.140.324.720.413.59.959.6 (+2.2)33.0 (+3.0)
Appendix
Table 4: Main results evaluated with Qwen3.5-35B-A3B as the judge. Best results within each model group are in bold. Parenthesized values in the Overall columns denote changes from the main results.
Figure 7: Effect of thinking mode on semantic and format constraints.
CategoryTask TypeDescriptionExample Question
Perception & RecognitionEntity RecognitionTo describe specific objects, individuals, animals, or elements present within the video frames.What type of vehicle is parked in front of the house at 0:15?
Perception & RecognitionAction RecognitionTo classify the specific movements, gestures, or activities being performed by entities in the video.What is the man in the blue shirt doing when the black car stops?
Perception & RecognitionAttribute RecognitionTo describe specific properties or characteristics of an entity, such as its color, size, shape, or material.What color is the backpack the child is carrying?
Perception & RecognitionScene DescriptionTo recognize and describe the global environment, setting, or background where the video takes place.Count the number of scenes appeared in the video and describe each of them in detail.
Perception & RecognitionText Recognition (OCR)To detect, extract, and read visible text within the video, such as street signs, subtitles, documents, or labels.What does the neon sign above the shop door say?
Perception & RecognitionDialogue RecognitionTo recognize and describe spoken content, speaker turns, or dialogue exchanged in the video.What are the exact words the news anchor says right after the loud crash is heard?
Appendix
Table 5: Video understanding task taxonomy in Video-IFBench.
GroupConstraint TypeDescriptionExample
SemanticTemporal ScopeRestrict the task to evidence, entities, events, or judgments within a specified absolute or segment-defined time window of the video, such as a timestamp range, beginning/middle/end portion, or last N seconds.Use only the visual events that occur in the last 10 seconds of the video in your answer.
SemanticTemporal AnchorConstrain a target relative to one explicit reference event, action, state, or timestamp, such as before, after, during, or at the moment of that anchor.Mention only the video content before the door opens.
SemanticTemporal Order RelationRequire the response to determine or preserve a temporal ordering relation among two or more already-specified events or targets, such as whether one happens before, after, between, or immediately adjacent to another.Indicate the temporal order of all mentioned visual events in your answer.
SemanticEvent StageRestrict the target to an intrinsic phase of a single continuous event or process, such as onset, peak, hold, completion, or recovery, rather than an arbitrary external time window.Do not mention any video content after the man leaves the room in your answer.
SemanticDuration ConstraintRequire the target event, state, relation, or occurrence to satisfy a duration condition such as minimum length, maximum length, longest, shortest, brief, or sustained.Mention only actions lasting more than 2 seconds.
SemanticTemporal GranularityConstrain the temporal unit used in the answer to a specified granularity, such as seconds only, minutes only, or shot indices only.Use shot numbers for every mentioned visual segment in your answer; do not use other timestamps.
Appendix
Table 6: Constraint taxonomy in Video-IFBench.
Figure 8: A Single instruction example with 6 constraints. Gemma-4-31B-it and Qwen3-Omni-30B-A3B-Instruct fail to avoid mentioning words that denote colors.
Figure 9: A Single instruction example with 7 constraints. Qwen3-Omni-30B-A3B-Instruct provides inaccurate temporal spans, mentions colors, and includes turtle content outside the requested time window of 01:08 to 02:12. Qwen2.5-Omni-7B describes dogs, cats, and rabbits from outside the requested time window, uses non-uppercase on-screen text, and mentions colors.
Figure 10: A Multi instruction example with 12 constraints. Gemini-3-Pro mentions an action that lasts less than 2 seconds in the video, violating the constraint in red .
Figure 11: A Multi instruction example 4 constraints. Qwen2.5-Omni-7B uses nested bullet points despite the requirement of exactly five bullet points and explicitly states that the candle holder is lit, violating the instruction to avoid mentioning whether the candles or candle holders are lit.
Figure 12: A Selection instruction example with 7 branches. Qwen3.5-397B-A17B-Think selects the wrong branch.
Figure 13: A Nested instruction example with a maximum depth of 15. Gemini-3-Pro follows the wrong path.
More in this topic

Video-Language Model Evaluation

Video-language model evaluation measures whether multimodal systems understand, ground, and reliably use visual and audio evidence over time.

See all 103 papers

Explore similar work

CardsList

  1. VCIFBench: Evaluating Complex Instruction Following for Video Understanding

    Jun 3, 2026Huangchen Xu, Yuan Wu, Yi ChangVLM EvaluationVideo Understanding

    School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University · 3International Center of Future Science, Jilin University

  2. AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents

    Sep 18, 2026Seoyeon An, Hyeonseo Jang, Minsu Kim +3Video QAAI Agent Benchmarks

    KRAFTON

  3. VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding

    Aug 12, 2026Fan Zhang, Guangming Yao, Jinyang Wu +6AI Agent BenchmarksVideo QA

    The Chinese University of Hong Kong · Ant Group · Tsinghua University +2