Video-Browser: Towards Open-web Agentic Video Browsing

Zhengyang Liang; Yan Shu; Xiangrui Liu; Minghao Qin; Nicu Sebe; Jiaxin Mao; Zheng Liu; Lizi Liao

cs.CVDec 28, 2025

Video-Browser: Towards Open-web Agentic Video Browsing

Authors: Zhengyang Liang, Yan Shu, Xiangrui Liu, Minghao Qin, Nicu Sebe, Jiaxin Mao, Zheng Liu, Lizi Liao

Organizations: Singapore Management University · University of Trento · Beijing Academy of Artificial Intelligence · Hong Kong Polytechnic University

Abstract

Autonomous agents are shifting information seeking from passive retrieval toward proactive, open-ended research, yet this progress has largely bypassed video, one of the web's richest sources of evidence. We formalize Agentic Video Browsing, in which an agent must query video search engines, decide which returned videos to watch, and answer questions grounded in their visual content. To study this setting reproducibly, we introduce Video-BrowseComp, a benchmark and infrastructure suite with 660 human-annotated questions at three difficulty levels, paired online and offline evaluation tracks, and a fixed video search stack of roughly 12,000 videos spanning over 2,000 hours for evaluation and training-data construction. Our analysis reveals that the main bottleneck lies before video understanding: agents often fail to find the answer-bearing video at all. This bottleneck stems from two coupled challenges. Search engines index textual metadata such as titles and tags, whereas questions often target implicit visual or cross-source evidence absent from these surface forms; and exhaustively inspecting the many returned videos is prohibitively expensive. We propose Video-Browser, an evidence-surfacing agent that improves both sides of search. A Translation Memory helps the Planner phrase queries by analogy to past question-to-query transformations that successfully surfaced their target videos, and a Watcher performs sparse visual screening over returned candidates, retaining only videos that plausibly contain answer-bearing evidence for later dense verification. On the offline track, Video-Browser raises overall accuracy from 11.6% to 27.3% with Qwen3-VL-8B and from 39.8% to 45.1% with Gemini-3.1-Pro over the corresponding ReAct search agents, showing that better query formulation and post-search candidate selection improve end-to-end browsing performance.

Figures & tables

Figure 1: Statistics and example from Video-BrowseComp. Left: category distribution of evaluation set. Right: an example showing the question, seed video frames, answer, and evidence chain.

Search Method	Query form	R@1	R@5	R@10	R@20	R@50	R@100
BM25	raw question	3.1	6.0	7.2	9.9	11.8	15.0
BM25	+ Qwen3-VL-8B	6.9	13.5	15.5	20.4	25.7	29.6
BM25	+ GPT-5.4	10.7	20.7	23.9	28.0	35.7	42.4
BM25	+ Gemini-3.1-Pro	15.0	24.5	29.9	33.5	39.6	44.9
BGE-M3	raw question	9.3	18.8	23.2	29.0	39.2	49.0
BGE-M3	+ Qwen3-VL-8B	8.6	16.1	19.2	25.7	32.6	39.6

Table 1: Search recall across three backbones and four query forms. Bold marks best performance.

VLM	L1	L2	L3	All
Qwen3-VL-8B	64.4	59.8	57.4	61.6
GPT-5.4	80.5	70.7	77.0	76.0
Gemini-3.1-Pro	87.3	80.4	73.8	82.7

Table 2: Reasoning accuracy when ground-truth videos are provided.

Figure 2: Overview of Video-Browser. The top panel constructs Translation Memory by selecting search-validated queries from training examples, while the bottom panel shows the online browsing loop, where the Planner generates searchable queries, the Watcher selects promising videos, and the Analyst performs dense visual verification.

Model	OA	Recall	Difficulty			CE	Genre Accuracy (%)
Model	(%)	(%)	L1	L2	L3	(%)	Doc	Edu	Film	Game	Music	Sport	TV	Vid
ReAct Search Agent
Qwen3-VL-8B	11.6	23.1	14.6	8.2	11.5	69.7	16.2	7.8	12.0	1.8	21.5	9.0	11.3	13.1
GPT-5.4	27.8	41.8	34.6	20.7	26.2	21.7	35.1	21.6	30.0	8.8	43.1	25.4	32.3	26.2
Gemini-3.1-Pro	39.8	43.6	46.3	35.9	29.5	21.3	48.6	47.1	36.0	26.3	61.5	25.4	40.3	36.1
Video Agent

Table 3: Offline-track performance comparison across agent baselines. Recall denotes cumulative recall over candidates retrieved during the browsing loop. We report overall, difficulty-level, and genre-wise accuracy, with calibration error. Bold marks the best result.

Method	Top-1 Hit	Cond. Top-1	Cond. MRR@10	L1 Top-1	L2 Top-1	L3 Top-1
Search Engine	6.0	–	–	7.3	2.2	13.1
LLM Reranker	12.0	57.4	73.5	11.2	7.1	29.5
Watcher	15.6 (+3.6)	75.3 (+17.9)	83.6 (+10.1)	15.6 (+4.4)	9.8 (+2.7)	32.8 (+3.3)

Table 5: Post-search reranking on BM25 Top-10 results using Qwen3-VL-8B. Top-1 Hit is measured over all questions; Cond. Top-1 and Cond. MRR@10 are computed only when the seed video is present in the BM25 Top-10 pool. Bold marks the best result.

Setting	OA	CE	Recall	Watched
Baseline	22.2	66.9	42.9	34.2
$+$ Memory	25.1	64.3	46.2	34.4
$+$ Watcher	27.3	56.8	46.2	36.2
w/o Transcript	19.8	65.1	45.3	34.9
w/o Visual	9.8	71.4	46.0	34.9

Table 6: Component and modality ablations.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Genre	Eval Offline (450)			Eval Online (210)			Train (1000)		Eval All	Train All
Genre	L1	L2	L3	L1	L2	L3	L1	L2	Eval All	Train All
Documentary	18	9	10	2	8	0	53	45	47	98
Education & Knowledge	19	26	6	23	2	0	68	74	76	142
Film	21	17	12	46	0	15	67	56	111	123
Games & Esports	21	22	14	13	9	0	60	46	79	106
Music & Variety	32	28	5	5	5	0	35	53	75	88

Appendix

Table 7: Per-genre and per-difficulty composition of Video-BrowseComp. The eval set is split into offline (450) and online (210) tracks, the training set (1,000) is auto-generated.

Length bucket	# videos	% of pool
$<$ 1 minute	5,413	43.5%
1–5 minutes	3,195	25.7%
5–10 minutes	1,138	9.1%
10–30 minutes	1,486	11.9%
30–60 minutes	682	5.5%
$>$ 1 hour	541	4.3%

Appendix

Table 8: Video duration distribution over the 12,455 videos in the offline pool.

Stage	Candidates	Filter applied
1. Decoy search (4 rewriters $\times$ 450 q.)	19,895	—
2. Prefilter (dedup, drop pool, drop GT)	17,168	remove existing pool / any-question GT
3. Layer-1 text verify (Qwen3-VL-8B)	8,883	keep No with High / Mid relevance
4. Layer-2 visual verify (707 shortlisted)	549	keep No with High / Mid relevance
5. Final selection (top-3 merged into pool)	522	per-question cap and selection ranking

Appendix

Table 9: Hard-negative pipeline funnel. Candidate counts denote the number surviving each filtering stage. Layer-2 evaluates 707 candidates shortlisted from the 8,883 Layer-1 survivors, of which 549 pass visual verification.

Model	OA	Difficulty			CE	Genre Accuracy (%)
Model	(%)	L1	L2	L3	(%)	Doc	Edu	Film	Game	Music	Sport	TV	Vid
Close Source Search Agent
Gemini-2.5-Flash*	20.95	32.80	4.84	0.00	35.98	0.00	36.00	29.51	4.55	10.00	11.11	36.84	16.67
Gemini-2.5-Pro*	23.81	37.60	4.84	0.00	31.45	20.00	32.00	31.15	9.09	10.00	8.89	57.89	16.67
GPT-5.1*	15.24	21.60	6.45	4.35	30.20	0.00	8.00	21.31	18.18	10.00	8.89	21.05	22.22
o4-mini-deep-research	22.86	30.40	12.90	8.70	42.55	10.00	28.00	29.51	27.27	10.00	17.78	21.05	16.67

Appendix

Table 10: Comprehensive results with a heatmap on Genre Accuracy. Bold indicates best performance. * means model has web search ability.

Backbone	Phase	Mean per question	Tokens / Q (k)	Tokens / unit (k)	% of total
Qwen3-VL-8B-Instruct	Browse	18.8 candidates browsed	38.3	2.0	38%
	Verify	3.0 videos verified	62.6	20.9	62%
	Total	—	100.9	—	100%
Gemini-3.1-Pro	Browse	18.8 candidates browsed	73.1	3.9	15%
	Verify	3.0 videos verified	427.6	142.5	85%
	Total	—	500.7	—	100%

Appendix

Table 11: Per-question token decomposition for Video-Browser, averaged over 450 offline-track questions. “Browse” aggregates Planner, Searcher, Stage-1 visual selector, and Checker calls across the

\mathcal{T}_{\max}{=}3

loops; “Verify” is the single Analyst call at termination. “Tokens / unit” is per browsed candidate or per verified video. The asymmetry is the intended sparse-vs-dense split of the pipeline.

Figure 12: Information Deficit (Kyle Lowry).

Figure 13: Action-to-Entity Mismatch.

Figure 14: Semantic Distractor (Icelandic Food).

Figure 15: Fine-grained Visual Hallucination.

Multimodal Search Agents

Multimodal search agents retrieve and reason across text, images, video, and other sources to answer complex questions.

See all 47 papers

VideoResearchAgent: Grounded Task Synthesis and Sim-to-Real RL for Open-Web Video Research
Oct 4, 2026Yuhang Zhou, Fei Li, Yuxi Wu +2Autonomous Research Agents Video Reasoning
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University
AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents
Sep 18, 2026Seoyeon An, Hyeonseo Jang, Minsu Kim +3Video QA AI Agent Benchmarks
KRAFTON
VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning
Jul 3, 2026Zhenkun Gao, Yicheng Bao, Jinlong Peng +13Multimodal RAG Video QA
East China Normal University · Tencent Youtu Lab · HFIPS, Chinese Academy of Sciences +3

Oct 4, 2026J/K move · Enter open · S save

VideoResearchAgent: Grounded Task Synthesis and Sim-to-Real RL for Open-Web Video Research

Yuhang Zhou, Fei Li, Yuxi Wu, Bin Zhu+1

Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University

Autonomous Research Agents Video Reasoning Sim-to-Real Transfer Temporal Video Understanding

Open paper AskPDF arXiv

Existing deep research agents are designed primarily for text- and image-based web sources, while video reasoning systems typically assume that relevant videos are provided in advance. We study open-web video research, where an agent must autonomously discover relevant videos, navigate their temporal content, and ground answers in visual evidence. Training such agents at scale is challenging as live video interaction is slow and unreliable, whereas fixed local simulation can induce retrieval-specific shortcuts that fail to transfer to the open web. We introduce VideoResearchAgent, a scalable training framework to address these challenges. First, we introduce controllable task synthesis pipeline to synthesize multi-hop research tasks from timestamped visual evidence while filtering text-only shortcuts. Second, we build a field-aligned local video simulator that preserves deployment-facing search and watch interactions while accelerating video search by a factor of 34.5-64.6. Third, we introduce Retrieval-Domain-Randomized GRPO (RDR-GRPO), which diversifies candidate rankings, distractors, metadata, and result structure during training to reduce overfitting to simulated retrieval. On Video-BrowseComp, the VideoResearchAgent trained using Qwen3.5-4B achieves 40.48% accuracy, comparable to Gemini-3-Flash-Preview, while reducing cumulative API-token consumption by 74.9% relative to the untrained model. Together, these results establish an accurate and efficient training recipe for open-web video research.

Abstract

Multimodal Search Agents

Explore similar work

VideoResearchAgent: Grounded Task Synthesis and Sim-to-Real RL for Open-Web Video Research

AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents

VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning