Organizations: Accio Team, Alibaba Group · Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology
Video deep research aims to answer questions by jointly understanding video content and retrieving external knowledge from the open Web. However, these tasks often require diverse and multi-step tool-use trajectories, which may lead to unnecessary video tool calls, incurring additional costs and failure risk. In this paper, we propose AdaVDR, an adaptive video deep research agent with adaptive tool use and reflection, which selects tools based on task and model capabilities while backtracking when needed. To this end, we develop a data construction pipeline that builds adaptive tool-use trajectories according to task types and model capabilities. Specifically, we construct QA pairs that require both video and external information, and organize the information acquisition processes into task-specific tool-use trajectories. We then apply model-conditioned tool necessity filtering to remove tools that can be bypassed using video understanding and internal knowledge of the model. Based on this pipeline, we construct a training dataset with adaptive trajectories and perform SFT on them to initialize these capabilities of AdaVDR. To further improve adaptive capabilities, we propose NASA-GRPO, a necessity-aware segment-guided credit assignment method, which evaluates segment-guided evidence contribution and tool necessity to construct normalized tool-level rewards. We also construct VDR-EE, a benchmark built using the proposed pipeline. Experiments show that AdaVDR outperforms the base model by 15.20 and 29.00 percentage points on VDR-EE and the existing VideoDR, respectively, demonstrating its superiority.
Figures & tables
Figure 1: Adaptive tool use and reflection in AdaVDR. AdaVDR adaptively selects tools and reflects on unreliable evidence, reducing redundant calls and error propagation.
Figure 2: Overview of the data construction pipeline, including QA and trajectory generation, trajectory refinement, and tool-necessary filtering.
Table 3
Figure 4: Examples of event-oriented and entity-oriented questions in VDR-EE.
Figure 5: Overview of NASA-GRPO, a necessity-aware segment-guided credit assignment RL.
Model
VideoDR
VDR-EE
Avg.
Avg.
Entity
Event
Avg.
Single
Multiple
Overall
Short
Medium
Long
Overall
Workflow
Direct
Proprietary
Gemini-3.1-Pro ( Team et al., 2023 )
–
56.58
28.57
42.48
54.29
55.88
50.00
53.61
46.80
–
Gemini-3-Flash ( Team et al., 2023 )
59.00
42.11
22.08
32.03
51.43
61.76
46.43
53.61
40.40
49.70
Table 2: Accuracy (%) on benchmkarks. VDR-EE uses the Direct and Agentic settings, while VideoDR uses the Workflow and Agentic settings. AdaVDR results are averaged over three runs.
Model
Video Grounding
External Retrieval
Trajectory
Temporal
Timestamp
Spatial
Image Search
Web Search
Visit Page
Avg. Turns
VideoDR
Qwen3.5-9B
0.46
0.24
0.07
0.12
2.56
0.67
5.12
AdaVDR-9B-SFT
1.55
1.73
0.44
0.93
4.07
0.46
10.18
AdaVDR-9B-RL
1.09
1.14
0.02
1.09
4.81
2.52
11.67
VDR-EE
Table 3: Average tool calls and turns per question across training stages on VideoDR and VDR-EE.
VideoDR
VDR-EE
Configuration
Acc.
Δ
Acc.
Δ
Base Model
37.00
–
40.00
–
SFT data composition ( Δ vs. Base Model)
Base Data
50.00
+13.00
47.20
+7.20
Adaptive Data
48.00
+11.00
46.00
+6.00
Reflection Data
51.00
+14.00
47.60
+7.60
Table 4: Ablation of SFT data and RL rewards. Accuracy is reported in %; Δ is the gain in percentage points over the base model for SFT and over Reflection + Adaptive Data for RL.
Figure 6: Training rollout rewards for (a) GRPO, (b) the variant without the necessity weight, and (c) NASA-GRPO. Pale and dark lines show raw and smoothed rewards. The curves in (b) are approximately digitized from the training-dashboard screenshot.
Figure 7: Qualitative examples of AdaVDR on entity-oriented and event-oriented questions.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Compute and parallelism
Backend
Megatron
GPUs
2 nodes × 8 GPUs (16 total)
Training tensor parallelism
4
Rollout tensor parallelism
1
Pipeline parallelism
1
Appendix
Table 5: RL training configuration for NASA-GRPO.
Model
Domains
Avg.
Culture
Entertain ment
Indus trial
News
Scene understanding
Sports
Science
Direct
Proprietary
Gemini-3.1-Pro [ Team et al., 2023 ]
38.89
54.29
27.78
37.14
63.89
50.00
55.56
46.80
Gemini-3-Flash [ Team et al., 2023 ]
44.44
40.00
16.67
22.86
66.67
44.44
47.22
40.40
GPT-5.4 [ Singh et al., 2025 ]
41.67
14.29
16.67
22.86
50.00
36.11
41.67
32.00
Appendix
Table 6: Domain-wise accuracy (%) on VDR-EE under the Direct and Agentic settings.
Model
Video Duration
Avg.
Short ≤5 min
Medium 5<t<10 min
Long ≥10 min
Direct
Proprietary
Gemini-3.1-Pro [ Team et al., 2023 ]
50.65
52.50
38.71
46.80
Gemini-3-Flash [ Team et al., 2023 ]
41.56
48.75
32.26
40.40
GPT-5.4 [ Singh et al., 2025 ]
31.17
36.25
29.03
32.00
Appendix
Table 7: Accuracy (%) across full-video duration groups on VDR-EE under the Direct and Agentic settings.
Model
VideoDR Domains
History
Geography
Culture
Economy
Technology
Daily Life
Avg.
Workflow
Proprietary
Gemini-3-Pro-Preview [ Team et al., 2023 ]
72.73
70.00
80.00
62.50
64.29
69.70
69.00
GPT-4o [ Hurst et al., 2024 ]
63.64
40.00
33.33
43.75
42.86
39.39
42.00
GPT-5.2 [ Singh et al., 2025 ]
72.73
70.00
80.00
56.25
64.29
72.73
69.00
Appendix
Table 8: Domain-wise accuracy (%) on VideoDR under the Workflow and Agentic settings.
Model
Agentic
Low
Medium
High
Avg.
#Samples
32
36
32
100
Proprietary
Gemini-3-Pro-Preview [ Team et al., 2023 ]
93.75
69.44
65.62
76.00
GPT-4o [ Hurst et al., 2024 ]
62.50
38.89
28.12
43.00
GPT-5.2 [ Singh et al., 2025 ]
84.38
58.33
65.62
69.00
Appendix
Table 9: Accuracy (%) across VideoDR difficulty levels under the Agentic setting.
Model
SimpleVQA
MMSearch
LiveVQA
BC-VL
FVQA
Avg.
Agentic workflow
GPT-5
67.3
62.7
–
57.6
62.0
–
Gemini-2.5-Pro
74.3
55.7
–
42.3
65.0
–
Gemini-3.1-Pro
–
86.1
76.6
64.1
84.0
–
Claude-4-Sonnet
–
67.2
–
48.6
69.0
–
Claude-4.6-Opus
–
76.2
67.4
48.3
74.5
–
Appendix
Table 10: Accuracy (%) on five image deep-research benchmarks.
Model
VideoMME
MV-Bench
LongVideoBench
Avg.
InternVL3.5-8B
66.0
72.1
62.1
66.7
Keye-VL-1.5-8B
73.0
56.9
66.0
65.3
Eagle2.5-8B
72.4
74.8
66.4
71.2
MiniCPM-V-4.5-8B
67.9
60.5
63.9
64.1
LLaVA-OneVision-2-8B
71.9
66.2
66.9
68.3
Qwen3.5-9B
71.1
68.2
64.6
68.0
Appendix
Table 11: Video question-answering accuracy (%).
Benchmark
Model
Segment
Useful segments (%)
Necessary among useful (%)
False acceptance (%)
VideoDR
SFT
Visual
58.4
68.9
27.6
Text
64.4
71.4
19.4
NASA-GRPO
Visual
76.1
84.6
12.8
Text
77.8
85.2
11.7
VDR-EE
SFT
Visual
60.7
69.7
23.4
Text
64.6
68.8
21.6
Appendix
Table 12: Tool effectiveness and necessity for visual and text segments on 100 manually reviewed VideoDR questions and VLM-judged VDR-EE questions. Useful segments and false acceptance use all segments as the denominator; necessary segments use useful segments as the denominator.
Figure 8: Tool effectiveness by segment type on VideoDR and VDR-EE. Necessary and useful but bypassable segments partition useful segments; false acceptance and other unsuccessful segments partition the remaining segments.
Figure 9: A VideoDR case illustrating unsupported answering after failed evidence acquisition. SFT retains an incorrect church identity despite the image-search result and answers approximately 2,000 m without supporting elevation evidence. The RL trajectory uses the retrieved church identity and elevation evidence to answer approximately 2,580 m.
Benchmark
Necessity weight
Necessary segments (%)
Necessary calls (%)
Redundant calls (%)
VideoDR
Without
54.9
54.6
45.4
With
65.9
76.2
23.8
VDR-EE
Without
50.8
53.2
46.8
With
65.9
74.5
25.5
Appendix
Table 13: Necessity of evidence-acquisition segments and tool calls on VideoDR and VDR-EE. Segment percentages use all segments; call percentages use all tool calls.
Figure 10: Necessary and redundant tool calls on VideoDR and VDR-EE, with and without the model-conditioned necessity weight. Shares are computed over all tool calls.
Rollouts
All tool calls
Credit 1.0 calls
Credit 0.5 calls
Positive-credit calls
Positive-credit share (%)
No-credit share (%)
0–9
3775
589
213
802
21.2
78.8
10–19
3878
652
374
1026
26.5
73.5
20–29
4947
878
362
1240
25.1
74.9
30–39
5059
922
616
1538
30.4
69.6
40–49
4753
923
582
1505
31.7
68.3
Appendix
Table 14: Tool-level process credits during RL training with the model-conditioned necessity weight. Positive-credit calls receive 0.5 or 1.0; no-credit calls receive 0.
Figure 11: Process-credit composition across RL rollout windows. Each bar shows the share of calls assigned credit 0, 0.5, or 1.0, using the counts in Table 14 .
Tool type
Without necessity weight
With necessity weight
Δ Calls
Calls
Share (%)
Calls
Share (%)
temporal grounding
62
10.42
65
11.44
+3
timestamp grounding
60
10.08
65
11.44
+5
spatial grounding
6
1.01
4
0.70
-2
image search
54
9.08
70
12.32
+16
web search
344
57.82
200
35.21
-144
Appendix
Table 15: Tool-call distribution in a training snapshot with and without the model-conditioned necessity weight.
Figure 12: Tool-use distribution in a training snapshot with and without the model-conditioned necessity weight. Bars show each tool’s share of all calls.
Bypassable-call weight
VideoDR
VDR-EE
Acc. (%)
Avg. calls
Acc. (%)
Avg. calls
0.2
61.00
9.10
50.80
10.50
0.3
63.00
9.55
52.40
11.05
0.4
64.00
10.10
54.00
11.65
0.5 (default)
66.00
10.67
55.20
12.26
0.6
62.00
11.75
52.00
13.40
Appendix
Table 16: Ablation of the necessity weight assigned to bypassable tool calls. The default is 0.5.
λ
VideoDR
VDR-EE
Acc. (%)
Avg. calls
Acc. (%)
Avg. calls
0.2
62.00
8.40
51.60
10.00
0.3 (default)
66.00
10.67
55.20
12.26
0.4
64.00
12.00
53.60
13.60
0.5
61.00
13.40
51.20
15.00
Appendix
Table 17: Ablation of the process-advantage coefficient λ . The default is 0.3.
Configuration
VideoDR
VDR-EE
Acc. (%)
Δ
Acc. (%)
Δ
Full model (all tools)
66.00
–
55.20
–
− temporal grounding
62.00
−4.00
52.80
−2.40
− timestamp grounding
61.00
−5.00
52.00
−3.20
− spatial grounding
65.00
−1.00
54.80
−0.40
− image search
58.00
−8.00
48.80
−6.40
Appendix
Table 18: Inference-time tool ablation. Each row independently removes only the named tool from the full model; all other tools remain available. Full-model accuracies match Table 2 .
Tool type
Agreement (%)
temporal grounding
96.0
timestamp grounding
94.0
spatial grounding
92.0
image search
98.0
web search
98.0
page visit
96.0
Appendix
Table 19: Agreement between human consensus and training-time tool credits by tool type.
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.
Open-world video understanding often requires a model to locate sparse visual evidence and acquire external knowledge that is absent from the video and its parametric memory. While Thinking-with-Videos enables active temporal perception and Deep Research supports multi-step information seeking, the two capabilities are typically developed in isolation. We introduce VideoRover, a unified Video Deep Research framework that iteratively coordinates video cropping, multimodal search, and webpage browsing. Given a video-question pair, VideoRover uses each tool result to select the next action, so localized video clips guide external retrieval and retrieved evidence triggers further video inspection and verification. To develop this capability, we construct an automated data curation pipeline, producing 26K verified SFT trajectories and 3K challenging RL instances. We also introduce VideoRover-Bench, a benchmark stratified by video duration and research difficulty. Experiments on VideoDR and VideoRover-Bench show that our VideoRover-8B-RL achieves performance comparable to proprietary models in the direct-answer setting without tool use while outperforming larger open-source models equipped with the same tool suite. Ablation studies and training dynamics further validate the complementary roles of active video grounding, external retrieval, and long-horizon reinforcement learning.
Wenqi Liu, Shijie Ma, Yunxiao Wang +21
Shandong University · Institute of Automation, Chinese Academy of Sciences · Beihang University +4
We introduce Video-DeepResearch (Video-DR), extending multimodal agents from static images to continuous video streams, a setting that demands dense spatiotemporal grounding coupled with open-web exploration. Preliminary evaluations reveal two critical bottlenecks in current models: (1) modality bias, where agents bypass visual tools in favor of textual search, and (2) parametric knowledge leakage, where models rely on internal memory rather than genuine tool-augmented execution. To address these challenges, we propose Video-DR, featuring a decoupled perception-exploration pipeline with stage-wise tool unlocking that compels exhaustive cross-frame visual grounding prior to web retrieval. Our framework adopts a two-stage training recipe: supervised fine-tuning followed by Group Relative Policy Optimization (GRPO), enabling autonomous exploration that breaks the imitation-learning ceiling. Furthermore, we curate Video-DR-Bench, a human-AI collaborative benchmark comprising 200 complex, multi-hop VQA instances. Empirical results demonstrate that our Video-DeepResearch-35B-A3B establishes a new state-of-the-art of 64.0% average accuracy, surpassing proprietary Claude-4.5-Sonnet (59.0%) by 5.0 points and significantly outperforming GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%). The 30B-A3B variant achieves 59.3%, competitive with Claude-4.5-Sonnet and demonstrating the effectiveness of our training paradigm even at compact scale. Code: https://github.com/Osilly/Vision-DeepResearch.
Video understanding is moving beyond closed-context perception toward open-world evidence exploration, a paradigm formalized as Video Deep Research (VDR). However, existing multimodal search agents primarily target static images, and the current VDR benchmark relies on text-centric retrieval that discards crucial visual information. To address these limitations, we propose VideoSearcher, a closed-loop agentic framework that empowers Vision-Language Models with multi-tool reasoning for VDR. VideoSearcher unifies temporal localization, spatial focusing, and multimodal search within a single reasoning trajectory, enabling agents to progressively ground visual clues, retrieve relevant evidence, and synthesize answers. To optimize knowledge-intensive reasoning trajectories, we propose Bi-branch Sequence Policy Optimization (BiSPO), a reinforcement learning algorithm that decouples tool-invocation optimization from answer-accuracy optimization. This design provides stable learning signals for both evidence-grounded reasoning and purposeful tool use. Furthermore, we construct VideoSearch-QA, the first benchmark designed to evaluate open-world video information grounding and multimodal search-based reasoning. Extensive experiments demonstrate that VideoSearcher significantly outperforms prior open-source agentic baselines across various search-oriented and multimodal understanding benchmarks.
Zhenkun Gao, Yicheng Bao, Jinlong Peng +13
East China Normal University · Tencent Youtu Lab · HFIPS, Chinese Academy of Sciences +3