Beyond Watching and Searching: Benchmarking Agents for Video-centered Deep Research Reports

Jiahao Meng; Qize Yang; Yuxuan Wang; Yunfei Chu; Wensi Dai; Jin Xu; Yunhai Tong

cs.CVOct 22, 2026

Beyond Watching and Searching: Benchmarking Agents for Video-centered Deep Research Reports

Authors: Jiahao Meng, Qize Yang, Yuxuan Wang, Yunfei Chu, Wensi Dai, Jin Xu, Yunhai Tong

Organizations: Peking University · Alibaba Token Hub, Alibaba Group · Independent Researcher

Abstract

Deep research agents synthesize open-web evidence into long reports, but current benchmarks do not evaluate research centered on a source video. We introduce Video-Centered Deep Research Report Generation. Given a video and user request, an agent identifies research questions, retrieves external multimodal evidence, and produces an interactive report linked to relevant video moments. This task has direct practical value because it helps users locate relevant moments, verify claims, reproduce demonstrated procedures, and make informed decisions from video content. We present a skill-based workflow and OmniDR-Bench for this task. For evaluation, we propose Omni Atomic Rubric Trees (Omni-ARTs). Each tree contains three dimensions, 23 shared directions, and task-specific atomic binary criteria. The directions test audiovisual grounding, multimodal evidence use, and interactive presentation. Audited Rubric Construction (ARC) builds these trees through independent evidence collection, automated checks, behavioral calibration, and iterative human review. OmniDR-Bench contains 50 Chinese and English tasks across 19 realworld subcategories. It includes 6,414 criteria. Across 18 agent configurations, the best system scores 63.3 out of 100. Current agents still need broader content coverage, stronger causal and logical analysis across modalities, and more accurate citation support. We further analyze models, harnesses, skill configurations, video categories, and research trajectories. The code and data are released at https://github.com/marinero4972/OmniDR-Bench.

Figures & tables

Figure 1: Video-Centered Deep Research and representative OmniDR-Bench results. Left: successful-report scores and task success. Right: the task integrates source-video audiovisual context with external text, image, and video evidence in an interactive, cited report.

Figure 2: Seven-stage reference workflow. Research anchors connect source-video moments to external webpage, image, video, and document evidence. The agent synthesizes these records into an interactive report and repairs failed checks.

Setting	Source video	Open web	Video retrieval	Long report	Time links	Primary objective
Video summary	Yes	No	No	No	No	Condense one video
Text deep research	No	Yes	No	Yes	No	Research a standalone topic
Open-web video search	No	Yes	Yes	No	No	Find video evidence
Video deep research QA	Yes	Yes	No	No	No	Answer one hard question
Text-image research report	No	Yes	No	Yes	No	Synthesize text and image evidence
Ours	Yes	Yes	Yes	Yes	Yes	Research and audit one video

Table 1: Task boundaries. Ours uses one given video as organizing evidence for an auditable report.

Figure 3: OmniDR-Bench statistics and one task that connects source-video evidence to external multimodal research. Selected content, citation, and presentation criteria show the evaluation scope. Appendix Tables 12 and 25 give more tasks, while Appendix Figures 5 and 6 show the full Omni-ART hierarchy and one complete report.

Figure 4: Query curation and Audited Rubric Construction (ARC). Curation retains source-video-dependent tasks that require external evidence. ARC independently gathers audiovisual and open-web multimodal evidence, derives an Omni Atomic Rubric Tree (Omni-ART), and validates both facts and criteria.

Dimension	Directions
Content	Instruction following, fact coverage, video faithfulness, segmentation, causal analysis, timeliness, comparison, cross-source linkage, logical consistency, ethics and safety
Citation	Source count, citation closure between inline citations and source records, source diversity, source authority, claim support, source freshness, multimodal material
Presentation	Video-note alignment, interactivity, visual quality, image-text match, readability, format use

Table 2: Evaluation dimensions. Each direction contains task-specific binary criteria.

Model	Avg.	95% CI	Success	Avg. succ.	Max	Min	Content	Presentation	Citation
Claude Opus 5	63.3	[57.8, 68.3]	96.0%	66.0	88.6	26.5	59.9	90.3	59.9
Claude Opus 4.8	60.7	[56.8, 64.2]	98.0%	62.0	81.8	37.2	57.4	86.0	53.5
Claude Opus 4.7	58.4	[54.1, 62.1]	96.0%	60.9	77.7	44.5	58.7	84.8	48.5
GPT-5.6 Sol	54.9	[52.9, 57.1]	100.0%	54.9	74.3	39.6	42.0	85.7	56.0
GPT-5.5	50.4	[48.0, 52.9]	100.0%	50.4	74.0	28.7	38.7	80.9	49.6
Gemini-3.8-Flash	52.6	[50.2, 55.1]	100.0%	52.6	73.2	28.3	43.6	79.6	49.7

Table 3: Results on 50 tasks with task-bootstrap 95% confidence intervals for Avg. Avg. counts failures as zero, while other quality scores use successful reports. Bold and underline mark the top two values except Success.

Family	Overall	Content	Citation	Presentation
Knowledge and viewpoints	54.8	48.9	46.9	81.2
Consumer and business	52.8	41.5	51.3	83.3
Physical procedures	53.5	43.7	50.2	83.1
Creative and digital	53.4	47.0	45.5	81.4

Table 4: Successful-report means by major video family on a

0

100

scale.

	Provided capability					Evaluation result
Variant	UI	Video	Web	Multi.	VDR	Avg.	Success	Avg. succ.	Content	Pres.	Citation	Steps	Time (h)	Tokens (M)
(i)	$\checkmark$					28.0	66.7%	42.0	18.9	50.5	28.2	95.5	0.6	1.2
(ii)	$\checkmark$	$\checkmark$				31.8	73.3%	43.4	26.6	54.3	25.4	94.0	0.8	1.4
(iii)	$\checkmark$	$\checkmark$	$\checkmark$			45.5	100.0%	45.5	34.9	80.3	40.0	104.0	0.6	2.0
(iv)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$		46.6	100.0%	46.6	40.1	80.7	34.7	103.0	0.7	2.3
(v)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	53.9	100.0%	53.9	40.9	83.9	55.4	167.0	1.6	4.7

Table 5: Cumulative skill ablation on 15 matched tasks with GPT-5.6 Sol and Claude Code. Columns abbreviate interface (UI), multimodal retrieval (Multi.), video-research orchestration (VDR), and presentation (Pres.). Scores include failures except Avg. succ.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Role	Released setting
$\bm{\alpha}^{(v)}$	Video-view query ranking weights	$(0.35,0.35,0.20,0.10)$
$\bm{\alpha}^{(u)}$	User-demand query ranking weights	$(0.30,0.30,0.20,0.20)$
$T_{\mathrm{win}}$	Source-video analysis window	$300$ seconds
$B_{\mathrm{web}},B_{\mathrm{video}}$	Minimum external-source and related-video budgets	$25,5$
$B_{\mathrm{doc}},B_{\mathrm{anchor}}$	Minimum document and per-anchor evidence budgets	$3,3$
$\tau_{d}$	Saturation for each high-volume coverage direction	$0.18$

Appendix

Table 6: Symbols and current settings for construction, execution, and evaluation.

Figure 5: Omni-ART hierarchy. Three dimensions and 23 directions test audiovisual grounding and cross-modal reasoning, source use and claim support, and interactive presentation. Task-specific binary leaves use a language-model judge (J), a deterministic program (M), or both, while outlined directions are gates.

Round	Correctness	Unanimity
1	64.0%	77.0%
2	87.0%	81.0%

Appendix

Table 7: Two-round human verification during task-package curation. Percentages are rounded to one decimal.

Measure	Reports	Result
Human unanimity on atoms	30	93.0%
Machine-human atom agreement	30	91.0%
Opus 4.8 Omni-ART score	50	60.7/100
GPT-5.6 Sol Omni-ART score	50	54.9/100
Opus 4.8 holistic rating	15	3.73/5
GPT-5.6 Sol holistic rating	15	3.51/5

Appendix

Table 8: Human validation on 30 randomly sampled reports. Holistic ratings use a 1–5 scale, and the reported correlation uses report-level scores.

Family	Subcategory	Typical research need
Knowledge and viewpoints	Academic frontier	Recover prerequisites, trace papers, check newer methods, and assess reproducibility
	Macro economy	Verify statistics, update policy context, and trace effects to industries or firms
	Law and traffic	Connect visible events to rules, precedent, evidence requirements, and uncertainty
	Controversy and history	Check claims against records and compare competing accounts
	Hardcore science	Explain mechanisms, test boundary conditions, and identify safe reproduction settings
Consumer and business	Product review	Verify product claims, compare alternatives, and assess long-term failure evidence

Appendix

Table 9: Full scenario taxonomy and typical research needs. One video may support multiple queries.

Test	Acceptance condition
Video-only	The request cannot be completed from the source video alone and requires external information
Web-only	Generic web research cannot replace the given video because replacing the source changes the investigation and at least one important anchor requires visual or temporal evidence
Answerability	Public evidence is sufficient within a practical research budget

Appendix

Table 10: Acceptance tests used during task selection.

Form	Count	Intended scope
General	7	A complete video-centered research report
Video-view	17	A broad research direction grounded in the video
User-demand	26	A concrete decision, diagnosis, reproduction, or application need

Appendix

Table 11: Query forms in the fixed 50-task set.

Form	Query	Why the source video is necessary
General	Conduct a general, in-depth, video-centered research analysis of this video.	The complete timeline supplies the candidate claims, demonstrations, and visual anchors
Video-view	Study the trade-off between under-display-camera image quality and display quality, centered on the device shown in the video.	The shown device, tests, and visual artifacts define the object and comparison boundary
Video-view	Study the distinction between the equivalence point and the reaction endpoint in the titration curve explained by the video.	The demonstrated curve, terminology, and procedure determine what must be explained and checked
User-demand	Assess whether the video’s claim about low debt-to-asset ratios among high-income households supports taking a leveraged home loan under current policy and market conditions.	The claim and its original measurement frame come from the video, while the decision needs current policy and external data

Appendix

Table 12: Representative query forms and the evidence that keeps each query video-centered.

Stage	Operation	Main output	Quality control
S0	Specify the task	Task scope and search plan	Source-video dependence, research focus, language, metadata, and route coverage
S1	Build the fact library	Timestamped video evidence, external evidence, and reconciled facts	Isolated collection, provenance, conflict preservation, relevance filtering, and semantic deduplication
S2	Generate candidate criteria	Evidence-bound atomic questions	Binding to fact records or executable report properties
S3	Screen the criteria	Validated criteria with difficulty labels	Grounding, atomicity, objectivity, judgeability, uniqueness, and coverage
S4	Assemble Omni-ART	A task-specific rubric tree	Three dimensions, 23 directions, and per-direction caps
S5	Validate the task package	Static, behavioral, stability, and human audit results	Checks of facts and criteria plus package-level issue rates

Appendix

Table 13: The seven ARC stages, their artifacts, and their quality controls.

Field	Meaning
fid	Stable fact identifier
claim	One checkable statement
type	Numeric, entity, date, place, quote, mechanism, opinion, or structural
origin	Video, external, or both
video_ref	Source-video time interval when applicable
sources	URL, quotation, date, and provenance tier such as primary, official, academic, media, or secondary

Appendix

Table 14: Evidence-record schema for generating and auditing source-bound Omni-ART criteria.

Setting	$(\lambda_{C},\lambda_{R},\lambda_{P})$	$\tau_{d}$	$\kappa_{d}$	Top three systems with scores	Max. rank change
Released	$(0.50,0.30,0.20)$	0.18	15	Opus 5 63.3, GLM-5.3 62.0, Opus 4.8 60.7	0
Equal weights	$(0.33,0.33,0.33)$	0.18	15	Opus 5 67.2, GLM-5.3 65.3, Opus 4.8 64.3	1
Content heavy	$(0.60,0.25,0.15)$	0.18	15	Opus 5 61.9, GLM-5.3 61.0, Opus 4.8 59.5	1
Citation heavy	$(0.40,0.40,0.20)$	0.18	15	Opus 5 63.3, GLM-5.3 61.7, Opus 4.8 60.3	1
Lower saturation	$(0.50,0.30,0.20)$	0.12	15	Opus 5 65.9, GLM-5.3 65.1, Opus 4.8 63.6	0
Higher saturation	$(0.50,0.30,0.20)$	0.24	15	Opus 5 60.5, GLM-5.3 59.4, Opus 4.8 57.9	1

Appendix

Table 15: Sensitivity of the all-task ranking to score aggregation parameters. The final column is the largest absolute rank change from the released setting.

		Evidence level				Content directions					Citation
Harness	Model	L0	L1	L2	L3	FC	CA	CMP	VF	LC	CS
Claude Code	Claude Opus 5	88.7	11.8	15.1	12.0	12.3	10.7	13.1	97.6	34.4	20.9
Claude Code	Claude Opus 4.8	85.7	11.4	12.2	7.5	10.5	12.8	6.9	89.6	23.0	22.1
Claude Code	Claude Opus 4.7	83.2	12.6	14.0	8.7	12.3	14.2	6.1	94.3	24.8	23.1
Claude Code	GPT-5.6 Sol	84.8	7.1	6.9	2.7	5.7	6.9	1.5	90.4	20.2	23.9
Claude Code	GPT-5.5	80.0	6.6	4.9	1.5	4.5	5.2	1.5	89.0	16.9	22.9

Appendix

Table 16: Atomic pass rates for all 18 configurations and their arithmetic mean. FC, CA, CMP, VF, LC, and CS denote fact coverage, causal analysis, comparison, video faithfulness, logical consistency, and claim support.

Model	Source count	Multimodal material	Claim support
GPT-5.6 Sol	98.0%	93.5%	23.9%
Gemini-3.8-Flash	24.0%	93.0%	13.9%

Appendix

Table 17: Selected citation-related atomic pass rates on the final task set.

Query form	Tasks	Avg.
General	7	41.3
Video-view	17	52.5
User-demand	26	50.4

Appendix

Table 18: Results by query form over all 18 systems.

Harness	Model	K	C	P	D
Claude Code	Claude Opus 5	68.1	69.9	64.3	62.6
Claude Code	Claude Opus 4.8	62.0	60.8	64.0	60.7
Claude Code	Claude Opus 4.7	63.6	60.0	63.0	56.8
Claude Code	GPT-5.6 Sol	57.1	54.1	53.1	55.0
Claude Code	GPT-5.5	51.1	49.2	50.5	50.3
Claude Code	Gemini-3.8-Flash	53.9	51.7	50.9	53.5

Appendix

Table 19: Successful-report means by model and major video family. K, C, P, and D denote knowledge, consumer, physical, and digital videos.

Family	Subcategory	Tasks	Overall	Content	Citation	Presentation
Knowledge	Hardcore science	2	46.0	28.3	51.8	81.4
Physical	Fitness and rehabilitation	3	46.0	27.5	51.5	84.1
Physical	Agricultural machinery	1	48.7	33.2	51.0	83.9
Consumer	Livestream commerce	3	50.2	37.4	49.8	82.8
Consumer	B2B software	2	50.6	38.3	51.0	80.7
Digital	UI and dashboards	4	50.6	40.1	48.3	80.3

Appendix

Table 20: Successful-report means for all video subcategories on a

0

100

scale.

Comparison	Mean gain	Wins	95% CI
Full vs. all atomic skills	7.3	12/15	[3.4, 11.1]

Appendix

Table 21: Paired effect of adding the video-research workflow to all atomic skills.

Harness	Avg.	Success	Steps	Tool calls	Tool errors	Time	Tokens
Claude Code	48.4	96.0%	156.0	105.5	7.0	1.3	4.3
Codex	48.4	99.3%	134.0	114.5	0.0	0.7	6.2
OpenClaw	39.4	84.7%	173.0	91.0	5.0	0.5	2.7

Appendix

Table 22: Controlled harness comparison over the three shared base models. Steps, tool calls, and errors are counts, time is in hours, and tokens are in millions.

Harness	Model	Steps (#)	Tool calls (#)	Tool errors (#)	Time (h)	Tokens (M)
Claude Code	Claude Opus 5	260.5	148.5	5.5	3.9	28.7
Claude Code	Claude Opus 4.8	286.0	131.0	2.0	1.9	21.5
Claude Code	Claude Opus 4.7	221.0	149.0	2.0	2.1	15.1
Claude Code	GPT-5.6 Sol	167.0	114.5	11.5	1.3	4.6
Claude Code	GPT-5.5	137.5	91.0	7.0	0.7	3.9
Claude Code	Gemini-3.8-Flash	237.0	125.0	9.0	3.1	10.9

Appendix

Table 23: Median process statistics over available task traces for all 18 configurations. Steps count logged trajectory entries, and tool errors count failed invocations. The # columns are counts, time is in hours, and tokens are in millions.

Harness	Model	Traces (#)	First write pos. (%)	Last search pos. (%)	Post-write search (%)	Write blocks (#)
Claude Code	Claude Opus 5	50	41.5	85.4	61.5	11.0
Claude Code	Claude Opus 4.8	50	57.4	87.0	38.0	7.0
Claude Code	Claude Opus 4.7	49	53.7	86.2	50.0	10.0
Claude Code	GPT-5.6 Sol	50	58.8	79.9	33.3	6.0
Claude Code	GPT-5.5	50	60.8	79.4	25.0	4.5
Claude Code	Gemini-3.8-Flash	50	69.0	87.7	18.5	6.0

Appendix

Table 24: Median phase-order statistics over available task traces for all 18 configurations. Positions and search shares are percentages.

Example	User focus	Video evidence	External research needed
A	Decide whether a household should take additional mortgage leverage	The speaker’s debt-to-asset claim and its implied measurement frame	Current mortgage rules, interest rates, household leverage data, price-to-income indicators, and decision uncertainty
B	Study the display and imaging trade-off of the shown under-display camera	Device identity, test conditions, and visible display or camera artifacts	Device specifications, optical mechanisms, independent tests, competing designs, and later product updates
C	Explain the pH jump and the distinction between stoichiometric and observed titration endpoints	The demonstrated curve, terminology, apparatus, and procedure	Textbook definitions, indicator ranges, measurement error, instrument guidance, and comparable demonstrations
D	Assess whether the shown renter-friendly plug-in pendant is safe and code compliant	Shown cord socket, PVC coupling, globe bulb, assembly steps, and visible product markings	Electrical code, listing standards, heat and strain risks, rental rules, safer alternatives, and current product evidence
E	Set alert and action limits for cleanroom viable colony-forming-unit (CFU) and particle data	The video’s particle-count chart, its approximate thresholds, and the demonstrated monitoring context	Cleanroom standards, viable and nonviable monitoring guidance, historical process-control methods, and current regulatory requirements

Appendix

Table 25: Representative tasks linking source-video and external evidence.

Figure 6: Four views of an interactive report for the pendant-light safety task. The interface aligns the audiovisual timeline with research findings, multimodal media, claim-level citations, and source records.

Deep Research Agents

Deep research agents plan long-horizon searches, synthesize cited evidence, and produce reports for complex open-ended questions.

See all 115 papers

Oct 4, 2026

VideoResearchAgent: Grounded Task Synthesis and Sim-to-Real RL for Open-Web Video Research

Existing deep research agents are designed primarily for text- and image-based web sources, while video reasoning systems typically assume that relevant videos are provided in advance. We study open-web video research, where an agent must autonomously discover relevant videos, navigate their temporal content, and ground answers in visual evidence. Training such agents at scale is challenging as live video interaction is slow and unreliable, whereas fixed local simulation can induce retrieval-specific shortcuts that fail to transfer to the open web. We introduce VideoResearchAgent, a scalable training framework to address these challenges. First, we introduce controllable task synthesis pipeline to synthesize multi-hop research tasks from timestamped visual evidence while filtering text-only shortcuts. Second, we build a field-aligned local video simulator that preserves deployment-facing search and watch interactions while accelerating video search by a factor of 34.5-64.6. Third, we introduce Retrieval-Domain-Randomized GRPO (RDR-GRPO), which diversifies candidate rankings, distractors, metadata, and result structure during training to reduce overfitting to simulated retrieval. On Video-BrowseComp, the VideoResearchAgent trained using Qwen3.5-4B achieves 40.48% accuracy, comparable to Gemini-3-Flash-Preview, while reducing cumulative API-token consumption by 74.9% relative to the untrained model. Together, these results establish an accurate and efficient training recipe for open-web video research.

Yuhang Zhou, Fei Li, Yuxi Wu +2

Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University

Autonomous Research Agents Video Reasoning Sim-to-Real Transfer Temporal Video Understanding

Ask PDF

Aug 4, 2026

Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

We introduce Video-DeepResearch (Video-DR), extending multimodal agents from static images to continuous video streams, a setting that demands dense spatiotemporal grounding coupled with open-web exploration. Preliminary evaluations reveal two critical bottlenecks in current models: (1) modality bias, where agents bypass visual tools in favor of textual search, and (2) parametric knowledge leakage, where models rely on internal memory rather than genuine tool-augmented execution. To address these challenges, we propose Video-DR, featuring a decoupled perception-exploration pipeline with stage-wise tool unlocking that compels exhaustive cross-frame visual grounding prior to web retrieval. Our framework adopts a two-stage training recipe: supervised fine-tuning followed by Group Relative Policy Optimization (GRPO), enabling autonomous exploration that breaks the imitation-learning ceiling. Furthermore, we curate Video-DR-Bench, a human-AI collaborative benchmark comprising 200 complex, multi-hop VQA instances. Empirical results demonstrate that our Video-DeepResearch-35B-A3B establishes a new state-of-the-art of 64.0% average accuracy, surpassing proprietary Claude-4.5-Sonnet (59.0%) by 5.0 points and significantly outperforming GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%). The 30B-A3B variant achieves 59.3%, competitive with Claude-4.5-Sonnet and demonstrating the effectiveness of our training paradigm even at compact scale. Code: https://github.com/Osilly/Vision-DeepResearch.

Zhen Fang, Yu Zeng, Wenxuan Huang +17

Deep Research Agents Video QA AI Agent Benchmarks Tool-Using Vision-Language Agents Autonomous Research Agents

Ask PDF

Aug 24, 2026

Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agents

Open-world video understanding often requires a model to locate sparse visual evidence and acquire external knowledge that is absent from the video and its parametric memory. While Thinking-with-Videos enables active temporal perception and Deep Research supports multi-step information seeking, the two capabilities are typically developed in isolation. We introduce VideoRover, a unified Video Deep Research framework that iteratively coordinates video cropping, multimodal search, and webpage browsing. Given a video-question pair, VideoRover uses each tool result to select the next action, so localized video clips guide external retrieval and retrieved evidence triggers further video inspection and verification. To develop this capability, we construct an automated data curation pipeline, producing 26K verified SFT trajectories and 3K challenging RL instances. We also introduce VideoRover-Bench, a benchmark stratified by video duration and research difficulty. Experiments on VideoDR and VideoRover-Bench show that our VideoRover-8B-RL achieves performance comparable to proprietary models in the direct-answer setting without tool use while outperforming larger open-source models equipped with the same tool suite. Ablation studies and training dynamics further validate the complementary roles of active video grounding, external retrieval, and long-horizon reinforcement learning.

Wenqi Liu, Shijie Ma, Yunxiao Wang +21

Shandong University · Institute of Automation, Chinese Academy of Sciences · Beihang University +4

Tool-Using Agents Video QA Video Reasoning RL for Tool Use

Ask PDF

Abstract

Deep Research Agents

Explore similar work

VideoResearchAgent: Grounded Task Synthesis and Sim-to-Real RL for Open-Web Video Research

Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agents