AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents

Seoyeon An; Hyeonseo Jang; Minsu Kim; Chanho Lee; Younghan Park; Kangwook Lee

cs.CVSep 18, 2026

AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents

Authors: Seoyeon An, Hyeonseo Jang, Minsu Kim, Chanho Lee, Younghan Park, Kangwook Lee

Organizations: KRAFTON

Abstract

Comprehensive video understanding is crucial for advancing artificial intelligence toward the intricate dynamics of the physical world. While recent advances in Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in video understanding, existing benchmarks remain confined to simple scene-level queries or global summaries that require only single-step inference. Real-world video understanding involves more challenging tasks that require multi-hop multimodal reasoning, and there is a critical absence of video benchmarks equipped to rigorously evaluate these agentic capabilities. To bridge this gap, we introduce AgentVidBench, a multi-hop video question answering benchmark focused on evaluating the spatial, temporal, and causal reasoning capabilities of MLLM agents. Beyond standard question-answer pairs, AgentVidBench provides step-by-step solution traces to support trajectory evaluation that assesses whether agents explicitly acquire the evidence needed to justify their answers. Experiments with 12 proprietary and open-source MLLMs show that single-turn performance remains limited on AgentVidBench, while integrating these models into state-of-the-art agentic workflows generally improves performance with respect to both accuracy and trajectory scores. We further present a simple yet effective agentic strategy that serves as a competitive baseline on AgentVidBench, establishing our benchmark as a holistic testbed for future research on agentic video understanding. Code and datasets are available at https://github.com/krafton-ai/agentvidbench and https://huggingface.co/datasets/agentvidbench/agentvidbench.

Figures & tables

Figure 1: An illustrative example from AgentVidBench, requiring spatial, temporal, and causal reasoning. The example shows the question, answer choices (ground-truth answer highlighted), a full-video filmstrip overview, and a summarized human-curated trajectory.

Figure 2: Overview of AgentVidBench. (a) Benchmark construction (Section 3.2 ): After video collection, questions are authored with consensus-based answers, distractors are generated to form multiple-choice options, and tool-agnostic milestones are extracted from reasoning notes. (b) Evaluation (Section 3.4 ): MLLM agent produces a reasoning trajectory and a final answer, which are assessed by a trajectory score and accuracy, respectively.

Panel of Figure 3 — Figure 3: AgentVidBench at a glance. Panels (a) and (b) summarize video-level statistics, including genre composition and duration distribution. Panels (c) and (d) summarize question-level statistics, including required skills and difficulty. Questions can have multiple skill tags, and the colors in (c) indicate the corresponding top-level capability groups.

Proprietary MLLMs
Models	Single-Turn		STAR [ 8 ]		DVD [ 65 ]		AVP [ 48 ]		Video-ReAct (Ours)
Models	Acc.	Traj.	Acc.	Traj.	Acc.	Traj.	Acc.	Traj.	Acc.	Traj.
GPT-5	0.29	0.55	0.14	0.33	0.34	0.60	0.41	0.65	0.49 $\uparrow$ 0.20	0.76 $\uparrow$ 0.21
Claude Opus 4.7	0.30	0.44	0.19	0.37	0.43	0.63	0.38	0.64	0.54 $\uparrow$ 0.24	0.79 $\uparrow$ 0.35
Gemini 2.5 Flash	0.46	0.66	0.10	0.29	0.27	0.47	0.46	0.65	0.45 $\downarrow$ 0.01	0.71 $\uparrow$ 0.05
Gemini 2.5 Pro	0.51	0.71	0.13	0.35	0.30	0.47	0.56	0.71	0.58 $\uparrow$ 0.07	0.77 $\uparrow$ 0.05

Table 1: Performance comparison of proprietary and open-source MLLMs across different agentic workflows on the AgentVidBench dataset. We report both Accuracy (Acc.) and Trajectory (Traj.) scores for each configuration. Colored arrows next to Video-ReAct (Ours) indicate absolute changes relative to the corresponding Single-Turn setting for the same model and metric.

Figure 4: Trajectory score profiles of vanilla MLLMs and agentic frameworks. Each radar plot shows the average scores across five trajectory evaluation dimensions: Task Understanding (TU), Evidence Coverage (ECov), Evidence Grounding (EG), Evidence Completeness (ECom), and Reasoning Faithfulness (RF). The dashed outline indicates the human reference score, where larger filled regions represent stronger trajectory quality.

	Adaptive Controls			AVP		Ours
Settings	FPS	Res.	Int.	Acc.	Traj.	Acc.	Traj.
W/o FPS	1	✓	✓	0.50	0.69	0.50	0.73
W/o Res.	✓	low	✓	0.49	0.68	0.56	0.73
W/o Res.	✓	med	✓	0.43	0.66	0.55	0.77
W/o Int.	✓	✓	full	0.46	0.68	0.50	0.72
Full	✓	✓	✓	0.56	0.71	0.58	0.77

Table 2: Ablation study on adaptive video-inspection controls using Gemini 2.5 Pro.

Methods	Acc.	MTok.	Eff. (%)
Single-Turn	0.51	15.16	3.36
AVP	0.56	10.60	5.28
Ours	0.58	15.44	3.76

Table 3: Accuracy vs. token efficiency. Eff. denotes token efficiency, computed as accuracy (Acc.) divided by million tokens (MTok.).

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmarks	Video QA	Reasoning Trajectory	Tool-aware Agent Trajectory
VideoMME	✓	$\times$	$\times$
LongVideoBench	✓	$\times$	$\times$
MINERVA	✓	✓	$\times$
AgentVidBench	✓	✓	✓

Appendix

Table 4: Comparison of video benchmarks.

Source platform	# Videos
YouTube	44
Wikimedia Commons	10
NASA	8
Self Recorded	4
DVIDS (US DoD)	2
Current Biology	1

Appendix

Table 5: Distribution of video sources in AgentVidBench.

License	# Videos
YouTube CC BY (reuse allowed)	43
Public Domain	14
CC BY-NC 4.0	4
CC BY-SA 4.0	4
CC BY 4.0	3
CC BY-SA 3.0	2

Appendix

Table 6: Distribution of video licenses in AgentVidBench.

Path-specific cue	Path-specific reference	Tool-agnostic milestone
Tool name	Use the OCR tool to read the model code.	Read the model code on the bezel as CF-19 .
Specific timestamp	At 02:13, inspect the laptop screen.	Locate the close-up moment where the laptop screen and bezel text are clearly visible.
Fixed time range	The relevant evidence is in the 00:20–00:50 segment.	Locate the driving segment where the relevant event is visible.
Timestamp list	Check the moments at 2:49, 2:50, 3:36, and 3:51.	Identify all candidate moments showing the target action.
Exact arithmetic chain	Sum the counts as 4+2+1+7+10+2+5+3+3 .	Sum the extracted counts to obtain the final total.

Appendix

Table 7: Examples of path-specific cues and their tool-agnostic milestone counterparts.

# Milestones	# Questions	Ratio (%)
3	14	14.0
4	32	32.0
5	40	40.0
6	12	12.0
7	2	2.0

Appendix

Table 8: Number of evidence milestones per question in AgentVidBench.

Milestone type	Occurrences	Ratio (%)
Temporal localization	113	24.8
Visual recognition	69	15.1
Exclusion check	59	12.9
Counting	54	11.8
Context identification	43	9.4
OCR/text reading	41	9.0

Appendix

Table 9: Distribution of evidence milestone types. Percentages are computed over 456 milestones.

Process-quality axis	Pearson $r$
Task understanding	0.656
Ground-truth evidence coverage	0.756
Evidence grounding	0.814
Evidence completeness	0.746
Reasoning faithfulness	0.837
Mean	0.762

Appendix

Table 10: Correlation between human-reference and milestone-reference judge scores.

Reference input	Affected Qs	Tool-agnostic rate
Human answer explanation	85/100	15.0%
Evidence milestones	1/100	99.0%

Appendix

Table 11: Path-specific cues detected in human explanations and milestone references.

Level	Axis	# Labels	Description
Video	Genre	12	Content genre, grouped into 4 super-categories.
	Duration	3	Short, medium, or long duration bin.
	Production style	14	Camera, editing, and presentation style.
	Audio type	5	Dominant audio characteristics.
Question	Required skills	20	Multi-label skills, grouped into 6 capabilities.
	Difficulty	4	Question difficulty label.

Appendix

Table 12: Overview of taxonomy axes in AgentVidBench .

Super-category	Description
Daily Lifestyle	Everyday activity-oriented videos. Genres: Travel/Lifestyle Vlog, Cooking/Food, Fitness/Workout.
Knowledge & Documentary	Informational, educational, or documentary-style videos. Genres: Science/Nature Documentary, News/Interview, Tutorial/How-to.
Performance	Event- or performance-centered videos. Genres: Sports Competition, Live Performance, Music Clip.
Media Arts	Screen-based, artistic, or contextual visual content. Genres: Screen Recording, Art/Animation, Stock/B-roll.

Appendix

Table 13: Video genre taxonomy.

Taxonomy	Labels
Production style	scene-cut edit, static camera, handheld camera, single continuous shot, drone/aerial, tracking camera, animation, screen capture, subtitle present, split screen, timelapse, text-overlay heavy, multicam switch, slow motion.
Audio type	speech, music, ambient sound, sound effects, silent.

Appendix

Table 14: Production-style and audio-type labels. Labels are multi-label at the video level.

Capability	Description	Fine-grained skills
Temporal	Reasoning about time, sequence, or scene structure of the video.	event localization, event ordering, scene segmentation, change detection.
Perception	Frame-level visual recognition of entities, attributes, or activities.	object recognition, action recognition, face detection / re-identification, attribute/state recognition, spatial relation, spatial search.
Counting	Quantifying occurrences across the video, possibly under conditions or with arithmetic over multiple counts.	simple counting, conditional counting, counting arithmetic.
Agentic reasoning	Higher-order skills: planning multi-step investigations, filtering candidate answers, or aggregating evidence across observations.	multi-step planning, candidate filtering, evidence aggregation.
OCR/Text	Reading textual or numeric content rendered visually in the frame.	OCR/text reading, numeric value extraction.
Audio-visual	Tasks that require integrating audio cues with visual content.	speech grounding, audio event detection.

Appendix

Table 15: Required-skill taxonomy. Labels are multi-label at the question level.

Component	Model	Settings
Milestone extraction	GPT-5	Reasoning effort: medium; maximum completion tokens: 16,384.
Trajectory judge	Claude Opus 4.7	1M context enabled; maximum output tokens: 4,096.

Appendix

Table 16: Model settings used in milestone extraction and trajectory judging.

Figure 5: Qualitative example of Video-ReAct (Ours). Round 1 spends a dense budget on a narrow window (2:05–2:27, 5 FPS, medium) to read the seven station names off the route map; Round 2 inverts the setting (2:27–end, 1 FPS, low) to survey every station visit at once, grounding water-supply artifacts at three stations and clearing the other four.

Figure 6: Trajectory score profiles of proprietary MLLMs under Single-Turn and agentic frameworks on AgentVidBench. Each radar plot shows the average scores across five trajectory evaluation dimensions: Task Understanding (TU), Evidence Coverage (ECov), Evidence Grounding (EG), Evidence Completeness (ECom), and Reasoning Faithfulness (RF). Columns correspond to Single-Turn, STAR, DVD, AVP, and Ours. The dashed outline indicates the human reference score, where larger filled regions represent stronger trajectory quality.

Figure 7: Trajectory score profiles of Qwen-family open-source MLLMs under Single-Turn and agentic frameworks on AgentVidBench. Each row corresponds to a Qwen backbone or its thinking variant, and each column corresponds to Single-Turn, STAR, DVD, AVP, or Ours. Each radar plot shows the average scores across five trajectory evaluation dimensions: Task Understanding (TU), Evidence Coverage (ECov), Evidence Grounding (EG), Evidence Completeness (ECom), and Reasoning Faithfulness (RF). The dashed outline indicates the human reference score, where larger filled regions represent stronger trajectory quality.

Figure 8: Trajectory score profiles of Gemma-family open-source MLLMs under Single-Turn and agentic frameworks on AgentVidBench. Each row corresponds to a Gemma backbone or its thinking variant, and each column corresponds to Single-Turn, STAR, DVD, AVP, or Ours. Each radar plot shows the average scores across five trajectory evaluation dimensions: Task Understanding (TU), Evidence Coverage (ECov), Evidence Grounding (EG), Evidence Completeness (ECom), and Reasoning Faithfulness (RF). The dashed outline indicates the human reference score, where larger filled regions represent stronger trajectory quality.

Proprietary MLLMs
Models	Single-Turn	STAR [ 8 ]	DVD [ 65 ]	AVP [ 48 ]	Ours
GPT-5	0.58	0.29	0.58	0.66	0.75 $\uparrow$ 0.17
Claude Opus 4.7	0.42	0.31	0.63	0.64	0.78 $\uparrow$ 0.36
Gemini 2.5 Flash	0.68	0.25	0.44	0.66	0.71 $\uparrow$ 0.03
Gemini 2.5 Pro	0.73	0.28	0.45	0.74	0.79 $\uparrow$ 0.06
Open-source MLLMs

Appendix

Table 17: Soft milestone coverage rate on the AgentVidBench dataset, reported for each combination of backbone and method. The metric assigns full credit to covered milestones, half credit to partial milestones, and zero credit to incorrect or missing milestones. Blue arrows indicate absolute changes relative to the corresponding Single-Turn setting.

Failure tag	Single-Turn	STAR [ 8 ]	DVD [ 65 ]	AVP [ 48 ]	Ours
unsupported_observation	47.1	70.2	34.5	15.9	7.7
counting_error	37.8	38.6	37.0	31.9	29.3
evidence_answer_conflict	27.3	65.0	32.7	17.6	17.6
insufficient_full_video_scan	27.5	45.0	43.1	22.1	14.7
visual_misrecognition	37.3	34.4	23.6	23.9	23.0
premature_fixation	25.2	23.4	30.9	8.4	15.2

Appendix

Table 18: Failure-tag frequency, in occurrences per 100 questions, by evaluated setting, averaged over the 20 backbone configurations. Tags are ordered by their overall frequency. The lowest frequency in each row is shown in bold.

Milestone type	Single-Turn	STAR [ 8 ]	DVD [ 65 ]	AVP [ 48 ]	Ours
temporal_localization	0.56	0.27	0.53	0.84	0.86
visual_recognition	0.45	0.18	0.42	0.67	0.70
exclusion_check	0.39	0.11	0.29	0.56	0.64
counting	0.26	0.06	0.20	0.43	0.45
context_identification	0.74	0.40	0.70	0.90	0.91
ocr_text_reading	0.39	0.05	0.43	0.67	0.69

Appendix

Table 19: Soft milestone coverage rate by milestone type and evaluated setting, averaged over the 20 backbone configurations. The highest coverage in each row is shown in bold.

Models	STAR [ 8 ]	DVD [ 65 ]	AVP [ 48 ]	Ours
Qwen3.5-27B	$\mathbf{-0.16}$ $[-0.25,-0.07]$	$\mathbf{+0.16}$ $[+0.06,+0.26]$	$\mathbf{+0.24}$ $[+0.15,+0.34]$	$\mathbf{+0.28}$ $[+0.19,+0.37]$
Gemma-4-31B	$\mathbf{-0.29}$ $[-0.39,-0.19]$	$-0.07$ $[-0.17,+0.02]$	$+0.08$ $[-0.01,+0.17]$	$\mathbf{+0.13}$ $[+0.03,+0.23]$
Gemini 2.5 Pro	$\mathbf{-0.38}$ $[-0.49,-0.27]$	$\mathbf{-0.21}$ $[-0.32,-0.10]$	$+0.05$ $[-0.03,+0.13]$	$+0.07$ $[-0.02,+0.16]$

Appendix

Table 20: Paired bootstrap confidence intervals for the accuracy difference between each agentic framework and Single-Turn (

10{,}000

resamples). Point estimates whose interval does not contain zero are shown in bold.

Comparison	Settings resolved	Pooled $p$
Ours vs. Single-Turn	15 / 16	$<0.001$
AVP [ 48 ] vs. Single-Turn	15 / 16	$<0.001$
DVD [ 65 ] vs. Single-Turn	0 10 / 16	$<0.001$
STAR [ 8 ] vs. Single-Turn	0 12 / 16	$<0.001$

Appendix

Table 21: McNemar’s test comparing each agentic framework against Single-Turn, pooled over the 16 open-weight settings in Table 1 .

Models	95% rank interval	P(rank 1)	P(top 3)
Gemma-4-31B +Think	$[1,3]$	71.2%	98.1%
Gemma-4-31B	$[1,5]$	14.3%	79.0%
Gemma-4-26B-A4B +Think	$[1,6]$	11.5%	72.9%
Qwen3.5-27B +Think	$[2,6]$	0 2.3%	35.0%
Remaining 10 models	$[2,16]$	$\leq$ 0.5%	$\leq$ 9.0%

Appendix

Table 22: Bootstrap stability of the model ranking induced by AgentVidBench. A model’s score is its mean accuracy over the five methods in Table 1 , and all 16 models are re-ranked on every bootstrap sample.

Models	1 fps		50 frames
Models	Acc.	Traj.	Acc.	Traj.
Gemini 2.5 Pro	0.51	0.71	0.40 $\downarrow$ 0.11	0.68 $\downarrow$ 0.03
Qwen3.5-27B	0.22	0.53	0.29 $\uparrow$ 0.07	0.57 $\uparrow$ 0.04
+Think	0.25	0.57	0.23 $\downarrow$ 0.02	0.47 $\downarrow$ 0.10
Gemma-4-31B	0.38	0.59	0.26 $\downarrow$ 0.12	0.51 $\downarrow$ 0.08
+Think	0.37	0.61	0.24 $\downarrow$ 0.13	0.50 $\downarrow$ 0.11

Appendix

Table 23: Single-Turn results under the original 1 fps setting and under a matched 50-frame setting. Arrows indicate the change relative to the 1 fps setting.

Qwen3.5-27B
Method	Acc.	Traj.	Tok./Q
Single-Turn	0.29	0.57	14.2K
STAR + Gemini 2.5 Pro	0.08 $\downarrow$ 0.01	0.43 $\uparrow$ 0.17	40.3K
DVD + Gemini 2.5 Pro	0.36 $\downarrow$ 0.01	0.64 $\uparrow$ 0.06	149.3K
Ours	0.49	0.78	158.8K
Gemma-4-31B

Appendix

Table 24: Results after aligning the video recognition tools of DVD and STAR with the Gemini 2.5 Pro model used by our method. Single-Turn rows use the matched 50-frame setting of Table 23 ; arrows indicate the change relative to Table 1 . Tok./Q denotes the average number of tokens per question.

Explore similar work

Aug 12, 2026

VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding

Video understanding is a fundamental task for evaluating the capabilities of multimodal large language models (MLLMs). However, existing leading models have already achieved approximately 90% accuracy on the Video-MME leaderboard, suggesting that conventional single-turn video understanding tasks are becoming increasingly saturated and insufficient for assessing the intelligence of advanced MLLMs. Towards this end, we introduce VideoGAIA, an agentic video understanding benchmark for general artificial intelligence (AI) assistants. Moving beyond one-shot video question answering, VideoGAIA formulates video understanding as a multi-turn, tool-augmented interaction process, where models must iteratively perceive videos, invoke external tools, gather complementary information, and integrate multimodal evidence across turns. VideoGAIA contains 271 model-human co-designed tasks covering diverse and complex real-world scenarios. Each video-question-answer instance is independently verified by three human experts to ensure both correctness and appropriate difficulty. All evaluated MLLMs, including frontier models such as GPT-5.5 and Kimi-K3, achieve less than 60% accuracy on VideoGAIA, highlighting its value as a high-quality and timely benchmark for evaluating next-generation MLLMs. We hope that VideoGAIA will facilitate the transition from conventional video understanding toward agentic video understanding.

Fan Zhang, Guangming Yao, Jinyang Wu +6

The Chinese University of Hong Kong · Ant Group · Tsinghua University +2

AI Agent Benchmarks Video QA Tool Use in VLMs

Ask PDF

Sep 21, 2026

Agentic Video Understanding: A Survey

As large language models (LLMs) become capable of processing increasingly diverse modalities and longer temporal contexts, an emerging line of work is moving beyond fixed video-language inference toward agentic systems that actively decide what information to inspect, retain, verify, and act upon. This survey reviews video understanding agents: systems that use video as the primary information source and solve understanding tasks through adaptive state construction and action selection. We first formalize an agent loop for video understanding, then address a central question: why do agents matter for video understanding? To answer this, we organize the literature through a challenge-to-design taxonomy, linking context bottlenecks to hierarchical evidence memory, evidence sparsity to active evidence acquisition, temporal causality to state and process tracking, and multimodal ambiguity to role-specialized coordination. We further review state space paradigms, learning paradigms, supervision signals, benchmarks, and evaluation protocols. Finally, we identify open directions toward agentic-native temporal modeling and video-native agents. Project page: https://github.com/DXY0711/Awesome-Agentic-Video-Understanding

Xinyu Deng, Siwen Luo, Daochang Liu

The University of Western Australia

Temporal Video Understanding Memory-Augmented Video Understanding

Ask PDF

Mar 20, 2026

VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking

Video agentic models have advanced challenging video-language tasks. However, most agentic approaches still heavily rely on greedy parsing over densely sampled video frames, resulting in high computational cost. We present VideoSeek, a long-horizon video agent that leverages video logic flow to actively seek answer-critical evidence instead of exhaustively parsing the full video. This insight allows the model to use far fewer frames while maintaining, or even improving, its video understanding capability. VideoSeek operates in a think-act-observe loop with a well-designed toolkit for collecting multi-granular video observations. This design enables query-aware exploration over accumulated observations and supports practical video understanding and reasoning. Experiments on four challenging video understanding and reasoning benchmarks demonstrate that VideoSeek achieves strong accuracy while using far fewer frames than prior video agents and standalone LMMs. Notably, VideoSeek achieves a 10.2 absolute points improvement on LVBench over its base model, GPT-5, while using 93% fewer frames. Further analysis highlights the significance of leveraging video logic flow, strong reasoning capability, and the complementary roles of toolkit design.

Jingyang Lin, Jialian Wu, Jiang Liu +6

AMD · University of Rochester

Long-Video Understanding Video Reasoning Query-Aware Video Frame Selection

Ask PDF