AdaVDR: Adaptive Tool Use and Reflection for Video Deep Research

Xintong Zhang; Xiaomeng Fan; Shilin Yan; Ekko He; Zicheng Liu; Zijian Zou; Guannan Zhang; Yuwei Wu

cs.CVAug 26, 2026

AdaVDR: Adaptive Tool Use and Reflection for Video Deep Research

Authors: Xintong Zhang, Xiaomeng Fan, Shilin Yan, Ekko He, Zicheng Liu, Zijian Zou, Guannan Zhang, Yuwei Wu,

Organizations: Accio Team, Alibaba Group · Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology

Abstract

Video deep research aims to answer questions by jointly understanding video content and retrieving external knowledge from the open Web. However, these tasks often require diverse and multi-step tool-use trajectories, which may lead to unnecessary video tool calls, incurring additional costs and failure risk. In this paper, we propose AdaVDR, an adaptive video deep research agent with adaptive tool use and reflection, which selects tools based on task and model capabilities while backtracking when needed. To this end, we develop a data construction pipeline that builds adaptive tool-use trajectories according to task types and model capabilities. Specifically, we construct QA pairs that require both video and external information, and organize the information acquisition processes into task-specific tool-use trajectories. We then apply model-conditioned tool necessity filtering to remove tools that can be bypassed using video understanding and internal knowledge of the model. Based on this pipeline, we construct a training dataset with adaptive trajectories and perform SFT on them to initialize these capabilities of AdaVDR. To further improve adaptive capabilities, we propose NASA-GRPO, a necessity-aware segment-guided credit assignment method, which evaluates segment-guided evidence contribution and tool necessity to construct normalized tool-level rewards. We also construct VDR-EE, a benchmark built using the proposed pipeline. Experiments show that AdaVDR outperforms the base model by 15.20 and 29.00 percentage points on VDR-EE and the existing VideoDR, respectively, demonstrating its superiority.

Figures & tables

Figure 1: Adaptive tool use and reflection in AdaVDR. AdaVDR adaptively selects tools and reflects on unreliable evidence, reducing redundant calls and error propagation.

Figure 2: Overview of the data construction pipeline, including QA and trajectory generation, trajectory refinement, and tool-necessary filtering.

Figure 4: Examples of event-oriented and entity-oriented questions in VDR-EE.

Figure 5: Overview of NASA-GRPO, a necessity-aware segment-guided credit assignment RL.

Proprietary
Model	VideoDR	VDR-EE								Avg.
	Avg.	Entity			Event				Avg.
		Single	Multiple	Overall	Short	Medium	Long	Overall
	Workflow	Direct
Gemini-3.1-Pro ( Team et al., 2023 )	–	56.58	28.57	42.48	54.29	55.88	50.00	53.61	46.80	–
Gemini-3-Flash ( Team et al., 2023 )	59.00	42.11	22.08	32.03	51.43	61.76	46.43	53.61	40.40	49.70

Table 2: Accuracy (%) on benchmkarks. VDR-EE uses the Direct and Agentic settings, while VideoDR uses the Workflow and Agentic settings. AdaVDR results are averaged over three runs.

VideoDR
Model	Video Grounding			External Retrieval			Trajectory
	Temporal	Timestamp	Spatial	Image Search	Web Search	Visit Page	Avg. Turns
Qwen3.5-9B	0.46	0.24	0.07	0.12	2.56	0.67	5.12
AdaVDR-9B-SFT	1.55	1.73	0.44	0.93	4.07	0.46	10.18
AdaVDR-9B-RL	1.09	1.14	0.02	1.09	4.81	2.52	11.67
VDR-EE

Table 3: Average tool calls and turns per question across training stages on VideoDR and VDR-EE.

SFT data composition ( $\Delta$ vs. Base Model)
	VideoDR		VDR-EE
Configuration	Acc.	$\Delta$	Acc.	$\Delta$
Base Model	37.00	–	40.00	–
Base Data	50.00	+13.00	47.20	+7.20
Adaptive Data	48.00	+11.00	46.00	+6.00
Reflection Data	51.00	+14.00	47.60	+7.60

Table 4: Ablation of SFT data and RL rewards. Accuracy is reported in %;

\Delta

is the gain in percentage points over the base model for SFT and over Reflection + Adaptive Data for RL.

Figure 6: Training rollout rewards for (a) GRPO, (b) the variant without the necessity weight, and (c) NASA-GRPO. Pale and dark lines show raw and smoothed rewards. The curves in (b) are approximately digitized from the training-dashboard screenshot.

Figure 7: Qualitative examples of AdaVDR on entity-oriented and event-oriented questions.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Compute and parallelism
Parameter	Value
Backend	Megatron
GPUs	2 nodes $\times$ 8 GPUs (16 total)
Training tensor parallelism	4
Rollout tensor parallelism	1
Pipeline parallelism	1

Appendix

Table 5: RL training configuration for NASA-GRPO.

Direct
Model	Domains							Avg.
	Culture	Entertain ment	Indus trial	News	Scene understanding	Sports	Science
Proprietary
Gemini-3.1-Pro [ Team et al., 2023 ]	38.89	54.29	27.78	37.14	63.89	50.00	55.56	46.80
Gemini-3-Flash [ Team et al., 2023 ]	44.44	40.00	16.67	22.86	66.67	44.44	47.22	40.40
GPT-5.4 [ Singh et al., 2025 ]	41.67	14.29	16.67	22.86	50.00	36.11	41.67	32.00

Appendix

Table 6: Domain-wise accuracy (%) on VDR-EE under the Direct and Agentic settings.

Direct
Model	Video Duration			Avg.
	Short $\leq 5$ min	Medium $5<t<10$ min	Long $\geq 10$ min
Proprietary
Gemini-3.1-Pro [ Team et al., 2023 ]	50.65	52.50	38.71	46.80
Gemini-3-Flash [ Team et al., 2023 ]	41.56	48.75	32.26	40.40
GPT-5.4 [ Singh et al., 2025 ]	31.17	36.25	29.03	32.00

Appendix

Table 7: Accuracy (%) across full-video duration groups on VDR-EE under the Direct and Agentic settings.

Workflow
Model	VideoDR Domains
	History	Geography	Culture	Economy	Technology	Daily Life	Avg.
Proprietary
Gemini-3-Pro-Preview [ Team et al., 2023 ]	72.73	70.00	80.00	62.50	64.29	69.70	69.00
GPT-4o [ Hurst et al., 2024 ]	63.64	40.00	33.33	43.75	42.86	39.39	42.00
GPT-5.2 [ Singh et al., 2025 ]	72.73	70.00	80.00	56.25	64.29	72.73	69.00

Appendix

Table 8: Domain-wise accuracy (%) on VideoDR under the Workflow and Agentic settings.

Proprietary
Model	Agentic
	Low	Medium	High	Avg.
#Samples	32	36	32	100
Gemini-3-Pro-Preview [ Team et al., 2023 ]	93.75	69.44	65.62	76.00
GPT-4o [ Hurst et al., 2024 ]	62.50	38.89	28.12	43.00
GPT-5.2 [ Singh et al., 2025 ]	84.38	58.33	65.62	69.00

Appendix

Table 9: Accuracy (%) across VideoDR difficulty levels under the Agentic setting.

Agentic workflow
Model	SimpleVQA	MMSearch	LiveVQA	BC-VL	FVQA	Avg.
GPT-5	67.3	62.7	–	57.6	62.0	–
Gemini-2.5-Pro	74.3	55.7	–	42.3	65.0	–
Gemini-3.1-Pro	–	86.1	76.6	64.1	84.0	–
Claude-4-Sonnet	–	67.2	–	48.6	69.0	–
Claude-4.6-Opus	–	76.2	67.4	48.3	74.5	–

Appendix

Table 10: Accuracy (%) on five image deep-research benchmarks.

Model	VideoMME	MV-Bench	LongVideoBench	Avg.
InternVL3.5-8B	66.0	72.1	62.1	66.7
Keye-VL-1.5-8B	73.0	56.9	66.0	65.3
Eagle2.5-8B	72.4	74.8	66.4	71.2
MiniCPM-V-4.5-8B	67.9	60.5	63.9	64.1
LLaVA-OneVision-2-8B	71.9	66.2	66.9	68.3
Qwen3.5-9B	71.1	68.2	64.6	68.0

Appendix

Table 11: Video question-answering accuracy (%).

Benchmark	Model	Segment	Useful segments (%)	Necessary among useful (%)	False acceptance (%)
VideoDR	SFT	Visual	58.4	68.9	27.6
		Text	64.4	71.4	19.4
	NASA-GRPO	Visual	76.1	84.6	12.8
		Text	77.8	85.2	11.7
VDR-EE	SFT	Visual	60.7	69.7	23.4
		Text	64.6	68.8	21.6

Appendix

Table 12: Tool effectiveness and necessity for visual and text segments on 100 manually reviewed VideoDR questions and VLM-judged VDR-EE questions. Useful segments and false acceptance use all segments as the denominator; necessary segments use useful segments as the denominator.

Figure 8: Tool effectiveness by segment type on VideoDR and VDR-EE. Necessary and useful but bypassable segments partition useful segments; false acceptance and other unsuccessful segments partition the remaining segments.

Figure 9: A VideoDR case illustrating unsupported answering after failed evidence acquisition. SFT retains an incorrect church identity despite the image-search result and answers approximately 2,000 m without supporting elevation evidence. The RL trajectory uses the retrieved church identity and elevation evidence to answer approximately 2,580 m.

Benchmark	Necessity weight	Necessary segments (%)	Necessary calls (%)	Redundant calls (%)
VideoDR	Without	54.9	54.6	45.4
	With	65.9	76.2	23.8
VDR-EE	Without	50.8	53.2	46.8
	With	65.9	74.5	25.5

Appendix

Table 13: Necessity of evidence-acquisition segments and tool calls on VideoDR and VDR-EE. Segment percentages use all segments; call percentages use all tool calls.

Figure 10: Necessary and redundant tool calls on VideoDR and VDR-EE, with and without the model-conditioned necessity weight. Shares are computed over all tool calls.

Rollouts	All tool calls	Credit 1.0 calls	Credit 0.5 calls	Positive-credit calls	Positive-credit share (%)	No-credit share (%)
0–9	3775	589	213	802	21.2	78.8
10–19	3878	652	374	1026	26.5	73.5
20–29	4947	878	362	1240	25.1	74.9
30–39	5059	922	616	1538	30.4	69.6
40–49	4753	923	582	1505	31.7	68.3

Appendix

Table 14: Tool-level process credits during RL training with the model-conditioned necessity weight. Positive-credit calls receive 0.5 or 1.0; no-credit calls receive 0.

Figure 11: Process-credit composition across RL rollout windows. Each bar shows the share of calls assigned credit 0, 0.5, or 1.0, using the counts in Table 14 .

Tool type	Without necessity weight		With necessity weight		$\Delta$ Calls
	Calls	Share (%)	Calls	Share (%)
temporal grounding	62	10.42	65	11.44	+3
timestamp grounding	60	10.08	65	11.44	+5
spatial grounding	6	1.01	4	0.70	-2
image search	54	9.08	70	12.32	+16
web search	344	57.82	200	35.21	-144

Appendix

Table 15: Tool-call distribution in a training snapshot with and without the model-conditioned necessity weight.

Figure 12: Tool-use distribution in a training snapshot with and without the model-conditioned necessity weight. Bars show each tool’s share of all calls.

Bypassable-call weight	VideoDR		VDR-EE
Bypassable-call weight	Acc. (%)	Avg. calls	Acc. (%)	Avg. calls
0.2	61.00	9.10	50.80	10.50
0.3	63.00	9.55	52.40	11.05
0.4	64.00	10.10	54.00	11.65
0.5 (default)	66.00	10.67	55.20	12.26
0.6	62.00	11.75	52.00	13.40

Appendix

Table 16: Ablation of the necessity weight assigned to bypassable tool calls. The default is 0.5.

$\lambda$	VideoDR		VDR-EE
	Acc. (%)	Avg. calls	Acc. (%)	Avg. calls
0.2	62.00	8.40	51.60	10.00
0.3 (default)	66.00	10.67	55.20	12.26
0.4	64.00	12.00	53.60	13.60
0.5	61.00	13.40	51.20	15.00

Appendix

Table 17: Ablation of the process-advantage coefficient

\lambda

. The default is 0.3.

Configuration	VideoDR		VDR-EE
	Acc. (%)	$\Delta$	Acc. (%)	$\Delta$
Full model (all tools)	66.00	–	55.20	–
$-$ temporal grounding	62.00	$-4.00$	52.80	$-2.40$
$-$ timestamp grounding	61.00	$-5.00$	52.00	$-3.20$
$-$ spatial grounding	65.00	$-1.00$	54.80	$-0.40$
$-$ image search	58.00	$-8.00$	48.80	$-6.40$

Appendix

Table 18: Inference-time tool ablation. Each row independently removes only the named tool from the full model; all other tools remain available. Full-model accuracies match Table 2 .

Tool type	Agreement (%)
temporal grounding	96.0
timestamp grounding	94.0
spatial grounding	92.0
image search	98.0
web search	98.0
page visit	96.0

Appendix

Table 19: Agreement between human consensus and training-time tool credits by tool type.

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 326 papers

Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agents
Aug 24, 2026Wenqi Liu, Shijie Ma, Yunxiao Wang +21Tool-Using Agents Video QA
Shandong University · Institute of Automation, Chinese Academy of Sciences · Beihang University +4
Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent
Aug 4, 2026Zhen Fang, Yu Zeng, Wenxuan Huang +17Deep Research Agents Video QA
VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning
Jul 3, 2026Zhenkun Gao, Yicheng Bao, Jinlong Peng +13Multimodal RAG Video QA
East China Normal University · Tencent Youtu Lab · HFIPS, Chinese Academy of Sciences +3

Aug 24, 2026J/K move · Enter open · S save

Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agents

Wenqi Liu, Shijie Ma, Yunxiao Wang, Meng Liu+20

Shandong University · Institute of Automation, Chinese Academy of Sciences · Beihang University+4

Tool-Using Agents Video QA Video Reasoning RL for Tool Use

Open paper AskPDF arXiv

Open-world video understanding often requires a model to locate sparse visual evidence and acquire external knowledge that is absent from the video and its parametric memory. While Thinking-with-Videos enables active temporal perception and Deep Research supports multi-step information seeking, the two capabilities are typically developed in isolation. We introduce VideoRover, a unified Video Deep Research framework that iteratively coordinates video cropping, multimodal search, and webpage browsing. Given a video-question pair, VideoRover uses each tool result to select the next action, so localized video clips guide external retrieval and retrieved evidence triggers further video inspection and verification. To develop this capability, we construct an automated data curation pipeline, producing 26K verified SFT trajectories and 3K challenging RL instances. We also introduce VideoRover-Bench, a benchmark stratified by video duration and research difficulty. Experiments on VideoDR and VideoRover-Bench show that our VideoRover-8B-RL achieves performance comparable to proprietary models in the direct-answer setting without tool use while outperforming larger open-source models equipped with the same tool suite. Ablation studies and training dynamics further validate the complementary roles of active video grounding, external retrieval, and long-horizon reinforcement learning.

Abstract

Autonomous Research Agents

Explore similar work

Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agents

Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning