Long video understanding is challenging due to rich and complicated multimodal clues in long temporal range.Current methods adopt reasoning to improve the model's ability to analyze complex video clues in long videos via text-form reasoning.However,the existing literature suffers from the fact that the text-only reasoning under fixed video context may exacerbate hallucinations since detailed crucial clues are often ignored under limited video context length due to the temporal redundancy of long videos.To address this gap,we propose Video-TwG,a curriculum reinforced framework that employs a novel Think-with-Grounding paradigm,enabling video LLMs to actively decide when to perform on-demand grounding during interleaved text-video reasoning, selectively zooming into question-relevant clips only when necessary.Video-TwG can be trained end-to-end in a straightforward manner, without relying on complex auxiliary modules or heavily annotated reasoning tracesIn detail,we design a Two-stage Reinforced Curriculum Strategy, where the model first learns think-with-grounding behavior on a small short-video GQA dataset with grounding labels,and then scales to diverse general QA data with videos of diverse domains to encourage generalization. Further, to handle complex think-with-grounding reasoning for various kinds of data,we propose TwG-GRPO algorithm which features the fine-grained grounding reward, self-confirmed pseudo reward and accuracy-gated mechanism.Finally,we propose to construct a new TwG-51K dataset that facilitates training. Experiments on Video-MME, LongVideoBench, and MLVU show that Video-TwG consistently outperforms strong LVU baselines.Further ablation validates the necessity of our Two-stage Reinforced Curriculum Strategy and shows our TwG-GRPO better leverages diverse unlabeled data to improve grounding quality and reduce redundant groundings without sacrificing QA performance.
Figures & tables
Figure 1. Conceptual level comparison between our Video-TwG and existing video reasoning models. The frame with blue triangle means the important detail for this question. Existing reasoning models reach the wrong answer via reasoning on missed video context, while we address it via leveraging grounding during reasoning to dynamically perceive important clues.
Figure 2. The trajectory of think-with-grounding . Initially, the coarse-grained video and question are given and in each turn, the model gives a thinking process and an action based on the history interaction. If the action is grounding, the grounded video clip is zoomed in with a fine-grained representation and added to the context. If the action is answering, the reasoning process stops.
Figure 3. The illustration of our proposed TwG-GRPO algorithm.
Method
Video-MME
LongVideoBench
MLVU-MC
short
medium
long
overall
(8s,15s]
(15s,60s]
(180s,600s)
(900s,3600s]
overall
Video-LLaVA-7B
45.3
38.0
36.2
39.9
-
-
-
-
39.1
47.3
ShareGPT4Video-8B
48.3
36.3
35.0
39.9
-
-
-
-
39.7
46.4
VideoChat2-7B
-
-
33.2
39.5
-
-
-
-
-
-
LongVA-7B
61.1
50.4
46.2
52.6
-
-
-
-
47.8
56.3
Long-LLaVA-7B
61.9
51.4
45.4
52.9
-
-
-
-
-
-
Table 1. Results of our Video-TwG and baselines on three mainstream long video benchmarks: Video-MME, LongVideoBench, MLVU. For MLVU, we evaluate the multiple choices questions. ∗ means our reproduced results. ⋅↑ is the relative gain against the Qwen2.5-VL-7B. The best and second are marked with bold and underline .
Method
Video-MME
short
medium
long
overall
Base 1
56.2
43.2
40.4
46.6
Base-TwG-CoT 1
56.9
43.2
42.1
47.4
Base-Think 2
59.2
46.4
43.2
49.6
Base-MultiGrained 4
63.1
45.9
42.7
50.6
Base-Think-MultiGrained 5
63.0
46.9
45.3
51.7
Table 2. Ablation experiments of different variants of the base Qwen2.5-VL-7B on Video-MME.
Figure 4. Ablation results on Video-MME of our models with different training stages.
Figure 5. Training curves of several variants of our Video-TwG in stage 1. The curves are smoothed for aesthetics. It’s the same below.
Method
Video-MME
short
medium
long
overall
stage 1 w/o gated mechanism
60.2
48.1
43.6
50.6
stage 1
61.2
46.9
45.1
51.1
Table 3. The ablation results of the accuracy-gated mechanism in stage 1 training.
Figure 6. Training curves of several variants of our Video-TwG in stage 2.
Method
Video-MME
short
medium
long
overall
Video-TwG w/o Rpseudo
63.4
48.0
47.2
52.9
Video-TwG
63.7
49.4
47.7
53.6
Table 4. The ablation results of the self-confirmed pseudo reward in stage 2 training.
Figure 7. The case study on Video-R1, Qwen2.5-VL-7B, Our Video-TwG. The location of the crucial video clues for this question is marked with green boxes.
Method
Video-MME
short
medium
long
overall
Video-TwG w/o Rpseudo
682
687
725
2094
Video-TwG
615
584
566
1765
Table 5. The number of samples that has grounding action on Video-MME benchmark.
Method
Video-MME
short
medium
long
overall
Video-TwG w/o Rpseudo
65.5
55.3
51.4
57.3
Video-TwG
68.1
54.5
53.5
58.9
Table 6. The evaluation results on the grounded video clip on Video-MME by InternVL-3.5-38B for our Video-TwG w/ and w/o Rpseudo .
Long-video understanding remains challenging for Multimodal Large Language Models (MLLMs) due to limited context length. Uniform sampling may miss crucial moments, while agent-based frame video understanding methods often evaluate frames independently, overlooking the temporal organization of videos. Ideally, evidence selection should mimic how humans answer questions about long videos: first locating the relevant segment from the global context, then zooming into local objects and details. We propose Temporal Tree of Thought T^3, a training-free framework for adaptive coarse-to-fine long-video understanding. T^3 constructs a question-agnostic hierarchical temporal tree via recursive temporally constrained clustering, where each node represents a contiguous segment with an informative key frame. During inference, T^3 performs an answer-retrieve-explore loop: it reasons over coarse representative frames, generates a search statement when evidence is insufficient, and expands relevant branches for finer-grained evidence. This process adaptively shifts the search target from temporal regions to specific objects and visual details to help video understanding. Experiments on VideoMME, LongVideoBench, and LVBench show that T^3 improves Qwen2.5-VL-7B by 0.5%, 4.6%, and 4.4%, respectively, under the same frame budget, demonstrating the effectiveness of structured temporal reasoning.
Video temporal grounding (VTG) aims to localize the video interval corresponding to a language query. Recent large vision-language models (LVLMs) show great potential in solving such a multi-modal reasoning task. However, long videos often contain large amounts of redundant information that disturbs LVLMs to mine query-relevant evidence. Instead of dense frame sampling which incurs prohibitive training memory, previous reinforcement learning with verifiable rewards (RLVR) works typically utilize sparse sampling, which makes training feasible but may miss critical evidence. In this paper, we propose a summarize before grounding'' framework (named SumGround'') for long-video temporal grounding. The key of SumGround is to perform query-guided chunk condensation to aggregate and retrieve query-relevant evidence. Specifically, we split the video into several chunks and perform two-level chunk condensation. First, we introduce query-guided latent summaries, which is represented as KV states of query-guided prompts, to compress redundant visual tokens into compact query-relevant chunk summaries. Furthermore, we design an associative summary retrieval scheme to rank and select chunk summaries that are most likely to contain the event interval. Both query-guided latent summary and associative summary retrieval schemes are enabled by RLVR. To reduce memory consumption, we propose a length-aware gradient gating module to selectively stop gradient back-propagated to visual tokens. Extensive experiments demonstrate that SumGround performs favorably against previous state-of-the-art methods across multiple downstream datasets, with remarkable gains on long videos.
Nanxing Hu, Xiaoyue Duan, Qiwei Yan +3
Beihang University · University of the Chinese Academy of Sciences · Institute of automation, Chinese academy of science
Recent advances in Video Large Language Models (Video-LLMs) have enabled performance on long-video understanding tasks. However, existing methods still face two key limitations: evidence acquisition often relies on a single search intent, and answer generation lacks an effective visual feedback mechanism. To address these limitations, we propose CoVER, a Comprehensive Visual Evidence and Reflection framework for long-video understanding. CoVER enables Video-LLMs to See More by dynamically gathering query-expanded visual evidence, and Think Deeper by verifying draft answers with effective answer-specific visual feedback. Together, these mechanisms shift long-video understanding from answer-centric generation to evidence-centric and visually verifiable reasoning. Experimental results show that CoVER-7B substantially outperforms models with the same parameter scale and even surpasses state-of-the-art closed-source models on certain metrics.
Shuning Wang, Zhiheng Wu, YiNuo Lu +6
Baidu Inc. · Harbin Institute of Technology · Hong Kong University of Science and Technology