Organizations: College of Computer Science, Zhejiang University of Technology, Zhejiang, China · Zhejiang Key Laboratory of Visual Information Intelligent Processing, Zhejiang, China · UC Berkeley, CA, USA · College of Computer Science and Technology, East China Normal University, Shanghai, China
Long video understanding is challenging due to dense visual redundancy, long-range temporal dependencies, and the tendency of chain-of-thought and retrieval-based agents to accumulate semantic drift and correlation-driven errors. We argue that long-video reasoning should begin not with reactive retrieval, but with deliberate task formulation: the model must first articulate what must be true in the video for each candidate answer to hold. This thinking-before-finding principle motivates VideoHV-Agent, a framework that reformulates video question answering as a structured hypothesis-verification process. Based on video summaries, a Thinker rewrites answer candidates into testable hypotheses, a Judge derives a discriminative clue specifying what evidence must be checked, a Verifier grounds and tests the clue using localized, fine-grained video content, and an Answer agent integrates validated evidence to produce the final answer. Experiments on three long-video understanding benchmarks show that VideoHV-Agent achieves state-of-the-art accuracy while providing enhanced interpretability, improved logical soundness, and lower computational cost. We make our code publicly available at: https://github.com/Haorane/VideoHV-Agent.
Figures & tables
Figure 1: From correlation-based search to hypothesis verification: prior agents search and aggregate related clips, while VideoHV-Agent verifies testable clues with focused visual evidence.
Figure 2: Overview of the proposed VideoHV-Agent framework. The framework first (a) summarizes the long video captions, then performs (b) two-step reasoning where a Thinker and a Judge agent rewrite options into hypotheses and a discriminative clue, and a Verifier agent grounds this clue to collect visual evidence, finally an Answer agent integrates the evidence to (c) answer the question.
Method
Accuracy (%)
TS-LLAVA [ 26 ]
57.8
HCQA [ 41 ]
58.8
VideoAgent [ 31 ]
60.2
VideoTree [ 35 ]
66.2
LVNet [ 25 ]
68.2
Tarsier [ 29 ]
68.6
Table 1: Zero-Shot Performance on EgoSchema subset compared to the state of the art.
Method
val set
ATP-hard subset
Supervised
ViLA [ 30 ]
74.4
-
VideoChat2 [ 16 ]
79.5
68.2
LLaVA-OV [ 14 ]
80.2
-
LinVT-Qwen2-VL [ 7 ]
85.5
69.1
Zero-shot
Table 2: Results on NextQA dataset compared to the state of the art.
Method
Accuracy (%)
SeViLA [ 39 ]
60.9
IG-VLM [ 12 ]
65.3
VideoTree [ 35 ]
66.9
LLoVi [ 40 ]
67.1
TS-LLAVA [ 26 ]
67.9
ENTER [ 1 ]
71.5
Table 3: Zero-Shot performance on IntentQA compared to the state of the art.
Method
Accuracy(%)
w/o hypothesis
76.0
w/o clue
78.6
w/o verification status
74.0
VideoHV-Agent
81.0
Table 4: Ablation of framework components.
Figure 3: Ablations of different maximum number of loops.
Figure 4: Proportion of samples with different numbers of loops.
Method
Time Cost(s)
Accuracy(%)
VideoAgent [ 31 ]
129.46
60.2
VideoTree [ 35 ]
160.21
66.2
VideoMultiAgents [ 13 ]
134.90
75.4
VideoHV-Agent
123.66
81.0
Table 5: Averaged time cost comparison with other agent-based methods.
Figure 5: Qualitative study of event understanding in long videos. VideoHV-Agent uses hypothesis–verification to locate decisive evidence, highlighting its ability to avoid search purposefully and ground conclusions in explicit visual proof.
Figure 6: Comparison for different types of NextQA questions.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
LLM
Methods
Accuracy (%)
GPT-4o
CoT
46.7
VideoTree
54.2 (+7.5%)
VCA
56.3 (+9.6%)
Ours
60.6 (+13.9%)
Appendix
Table S1: Results on VideoMME-L, with VCA (ICCV’25) .
LLM
Caption Model
Methods
Egoschema
Claude-3-Haiku
-
CoT
60.9
LaViLa
LifelongMemory
64.8 (+3.9%)
LaViLa
Ours
65.4 (+4.5%)
GPT-3.5
-
CoT
60.4
GPT-4o
LVNet
61.0 (+0.6%)
LaViLa
LifelongMemory
64.0 (+3.6%)
Appendix
Table S2 : Extra EgoSchema results under different LLM/captioner.
Long-form video understanding often involves multiple questions about different aspects of the same recording. Yet existing video agents typically process each question through an isolated tool-use trajectory. This repeatedly restarts video exploration and memory construction, missing opportunities to acquire evidence jointly and progressively build a shared understanding that supports the complete question set. We introduce VAMR (Video Agent for Multi-Question Reasoning), which coordinates all questions about a video through one shared tool-use trajectory. At each round, a persistent policy model can invoke tools for one or more unresolved questions and submit answers for questions with sufficient evidence. Question-conditioned visual perception retrieves fine-grained clues for several questions in one call, while layered multi-question memory integrates reusable context into a shared video story and preserves separate evidence for individual questions. After supervised fine-tuning initializes this interaction protocol, we propose question-horizon policy optimization (\qhpo) to optimize shared trajectories in which questions progress and finish at different rounds. Specifically, a question-level critic estimates the value of each active question, while round alignment maps each question advantage to the rounds that directly serve it before the aligned advantages are aggregated to optimize the shared actor. Across LVBench, Video-Holmes, and LongVideoBench, VAMR achieves the highest accuracy overall and the fewest reasoning rounds among iterative methods. On LVBench, it reaches 62.1% accuracy, exceeding VideoARM by 4.3 points while reducing reasoning rounds and processed frames by 85.9% and 61.4%.
Runquan Gui, Hanzhu Chen, Zehao Wang +3
University of Science and Technology of China · Tencent
Long video understanding often behaves like a visual needle-in-a-haystack problem: query-relevant evidence is sparsely distributed across long temporal spans, while packing dense frames into a single VLM context incurs context rot and high cost. Existing video agents often rely on query-agnostic offline preprocessing or ad hoc tool sets, which can miss query-specific details and waste computation. In this work, we present VideoXAgent, a purely online video-agent harness for long video understanding that starts from the given video file and user query, plans and decomposes the task, invokes specialized expert tools on demand, and aggregates multimodal evidence to produce a final answer while resolving conflicts among observations. To support this on-demand invocation, we design a suite of heterogeneous expert tools guided by a data-driven taxonomy of atomic capabilities, spanning scripts, VLMs, and domain models (e.g., detection, OCR, ASR, face recognition). The harness further enforces objective evidence prompting and budget-aware control to curb hallucination and non-termination. Across Video-MME-Long, LongVideoBench-Long, LVBench, and MINERVA, VideoXAgent is competitive with frontier LMMs and video agents under a smaller context footprint---about 50k tokens of agent context per sample, even on hour-long videos. In particular, on complex video-reasoning benchmarks such as MINERVA, it matches this level while using only about 15% of the context of a 1,024-frame dense-packing baseline. Notably, the harness remains effective with a visually weak or even text-only orchestrator, suggesting that strong long-video understanding can emerge from progressive agentic evidence seeking rather than from packing the full video into a single context. Project page: https://go-agent-x.github.io/video_agent_harness/
Recent advances in Video Large Language Models (Video-LLMs) have enabled performance on long-video understanding tasks. However, existing methods still face two key limitations: evidence acquisition often relies on a single search intent, and answer generation lacks an effective visual feedback mechanism. To address these limitations, we propose CoVER, a Comprehensive Visual Evidence and Reflection framework for long-video understanding. CoVER enables Video-LLMs to See More by dynamically gathering query-expanded visual evidence, and Think Deeper by verifying draft answers with effective answer-specific visual feedback. Together, these mechanisms shift long-video understanding from answer-centric generation to evidence-centric and visually verifiable reasoning. Experimental results show that CoVER-7B substantially outperforms models with the same parameter scale and even surpasses state-of-the-art closed-source models on certain metrics.
Shuning Wang, Zhiheng Wu, YiNuo Lu +6
Baidu Inc. · Harbin Institute of Technology · Hong Kong University of Science and Technology