Enhancing Localized Reasoning for Long Video Understanding via Efficient Segment-to-Video Supervision
Authors: Beibei Zhang, Chao Xu, Jun Lan, Zongyi Li, Lai Wei, Huijia Zhu, Tongwei Ren
Organizations: State Key Laboratory for Novel Software Technology, Nanjing University · Ant Group · School of Computer Science, Shanghai Jiao Tong University
Though Multimodal Large Language Models (MLLMs) have shown impressive potential in video understanding, long video understanding (LVU) remains challenging since distracting noise in complex and lengthy contexts can obscure localized details, misleading MLLMs to produce incorrect answers. Recent works mitigate these issues by incentivizing deep reasoning to include relevant evidence. However, these methods have two main problems: First, the reinforcement fine-tuning framework (RFT) they leveraged incurs substantial training overheads, including high annotation costs and complicated reward designs. Second, the self-reflective and iterative-perception mechanism in some methods causes lengthy outputs and high inference latency. To alleviate these problems, we propose a novel Segment-to-Video Supervision} method (S2V) to efficiently enhance fine-grained reasoning in LVU. Specifically, we generate question answer pairs (VQA) based on localized segments, and then transfer these segment-based VQA back to the whole video for training. Due to focusing on short segments, segment-based VQA can naturally notice details which tend to be overlooked from a whole-video perspective. Training on such data can enforce MLLMs to correctly associate fine-grained details with QA while avoiding distracting noise in the whole video. The S2V training involves just reinforcement learning (RL) with a simple accuracy reward based on only 10K VQA samples and the resulting S2V model predicts answer using a single forward pass with limited output tokens. Experimental results demonstrate that S2V can consistently improve LVU performance across multiple LVU benchmarks, outperforming both general MLLMs and reasoning-based methods not only in LVU accuracy but also in training and inference efficiency.
Figures & tables
Figure 1: Comparison between existing reasoning-based LVU methods vs. our S2V.
Figure 2: Overview of the S2V data synthesis pipeline. Here, n refers to the segment number, which is a dynamic number depending on the segment detection result, NF is the sampled frame number of the whole video and ND is the response repetition number for difficulty filtering.
Figure 3: S2V-10K Statistics and its topic, question type and video duration distribution.
Method
LongVideoBench ( Wu et al. 2024 )
Video-MME ( Fu et al. 2025 )
MLVU ( Zhou et al. 2025 )
(8,15]
(15,60]
(180,600]
(900,3600]
Overall
Short
Medium
Long
Overall
PlotQA
Needle
Ego
Count
Overall
Closed-Source MLLMs
GPT-5 ( Singh et al. 2026 )
76.7
72.7
62.1
56.2
63.1
84.2
72.9
67.0
74.7
80.0
78.0
65.3
38.3
72.9
Gemini-2.5 ( Comanici et al. 2025 )
76.7
83.7
70.1
61.3
69.1
86.9
79.8
72.3
79.7
82.9
79.4
65.1
60.2
78.2
Open-Source MLLMs
Qwen3-VL-4B ( Bai et al. 2025a )
71.4
71.5
55.6
46.8
56.2
76.3
61.3
52.6
63.4
72.7
70.4
60.2
37.4
66.1
Table 1: Performance comparison results of our S2V vs. different general MLLMs on LongVideoBench, Video-MME, and MLVU. Here, best results are in bold , sub-optimal ones are underlined .
Method
#param
Data
Frame
LVB
V-MME
MLVU
Retrieval-based LVU Methods
GoldFish
7B
176K
-
-
28.9
37.3
SALOVA
7B
2.3M
1fps
44.6
53.1
-
GenS
7B
150K
1fps
58.7
-
66.9
Reasoning-based LVU Methods
Video-R1
7B
425K
64
58.5
61.4
62.4
Table 2: Performance comparison results of our S2V vs. different LVU-tailored methods on LongVideoBench, Video-MME, and MLVU. Here, Data refers to training data scale and Frame denotes the input frame number.
Figure 4: Performance and latency comparison results of our S2V vs. both general MLLMs and LVU-reasoning methods on LongVideoBench benchmark.
Figure 5: Example LVU results of MLVU predicted by Qwen3-VL-4B vs. S2V-4B. Here, query-relevant frames are marked with green boxes.
Figure 6: Performance comparison results of our S2V vs. initial MLLMs. Here, Seg-View , Vid-View and w/o Seg-View refer to leveraging the corresponding segment, whole video and the video with the corresponding segment removed to answer the question respectively, Vid-View w/ Is and Vid-View w/ In leverage the whole video with segment position or null position in instruction to answer question respectively, and Δ is the VQA accuracy gap.
Figure 7: Ablation study of taking different numbers of frames as input on LongVideoBench and Video-MME.
Method
VQAV
VQAS
IS
LVB
MLVU
Qwen3-VL-4B
✗
✗
✗
56.2
66.1
V2V-4B
✓
✗
✗
57.7
67.0
S2V-4B w/o loc
✓
✓
✗
60.6
69.3
S2V-4B
✓
✓
✓
62.3
71.9
Table 3: Ablation study of our S2V synthesis pipeline on LongVideoBench and MLVU. Here, VQAV and VQAS refer to VQA annotation based on the whole video and segments, respectively and IS is the segment timestamp included in the VQA instruction.
Figure 8: Ablation study of simple, medium and hard sample numbers in S2V-10K on LongVideoBench and MLVU.
Figure 9: Ablation study of S2V trained with different data scales and model scales on Video-MME and MLVU.
Table 4: Performance comparison results of our S2V vs. different general MLLMs and TVG-tailored methods on TVGBench.
Method
SFT
RL
LongVB
Video-MME
MLVU
Qwen3-VL-4B
✗
✗
56.2
63.4
66.1
S2V-4B
✓
✗
55.6
61.1
67.0
S2V-4B
✗
✓
62.3
65.6
71.9
MiMo-VL-7B
✗
✗
58.5
61.9
64.2
S2V-7B
✓
✗
52.3
56.0
61.6
S2V-7B
✗
✓
62.6
64.4
69.1
Appendix
Table 5: Ablation study of S2V using different training strategies on LongVideoBench (LongVB), Video-MME and MLVU.
Figure 10: The segment-based VQA generation prompt in the segment-based VQA generation step of S2V-10K. This prompt guides the model to generate a multi-choice question answer triplet including question, answer and distracting options based on the provided segment frames and question template.
Figure 11: The segment-question relevance check prompt in the multi-segment VQA generation step of S2V-10K. This prompt guides the model to check if each segment involved in the multi-segment sample is truly relevant to the generated VQA.
Figure 12: A segment-based VQA example and the corresponding converted video-based VQA sample in S2V-10K.
Figure 13: A failed data construction case caused by object recognition error.
Figure 14: A failed data construction case caused by confusing distractors.
Figure 15: A failed data construction case caused by ambiguity in segment.
Figure 16: An S2V prediction output includes localized details which tend to be overlooked.
Figure 17: An S2V prediction output successfully associates localized information with correct answer.
Figure 18: An S2V prediction output correctly comprehends the temporal relations among localized evidence.
Existing Multimodal Large Language Models (MLLMs) often suffer from hallucinations in long video understanding (LVU), primarily due to the imbalance between textual and visual tokens. Observing that MLLMs handle short visual inputs well, recent LVU works alleviate hallucinations by automatically parsing the vast visual data into manageable segments that can be effectively processed by MLLMs. SFT-based tool-calling methods can serve this purpose, but they typically require vast amounts of fine-grained, high-quality data and suffer from constrained tool-calling trajectories. We propose a novel VideoTIR that leverages Reinforcement Learning (RL) to encourage proper usage of comprehensive multi-level toolkits for efficient long video understanding. VideoTIR explores both Zero-RL and SFT cold-starting to enable MLLMs to retrieve and focus on meaningful video segments/images/regions, enhancing long video understanding both accurately and efficiently. To reduce redundant tool-calling, we propose Toolkit Action Grouped Policy Optimization (TAGPO), which enhances the efficiency of the calling process through stepwise reward assignment and reuse of failed rollouts. Additionally, we develop a sandbox-based trajectory synthesis framework to generate high-quality trajectories data. Extensive experiments on three long-video QA benchmarks demonstrate the effectiveness and efficiency of our method.
Zhe Gao, Shiyu Shen, Taifeng Chai +7
Nanjing University, Nanjing, China · Nankai University, Tianjin, China · East China Normal University, Shanghai, China +3
Long-video understanding remains challenging for Multimodal Large Language Models (MLLMs) due to limited context length. Uniform sampling may miss crucial moments, while agent-based frame video understanding methods often evaluate frames independently, overlooking the temporal organization of videos. Ideally, evidence selection should mimic how humans answer questions about long videos: first locating the relevant segment from the global context, then zooming into local objects and details. We propose Temporal Tree of Thought T^3, a training-free framework for adaptive coarse-to-fine long-video understanding. T^3 constructs a question-agnostic hierarchical temporal tree via recursive temporally constrained clustering, where each node represents a contiguous segment with an informative key frame. During inference, T^3 performs an answer-retrieve-explore loop: it reasons over coarse representative frames, generates a search statement when evidence is insufficient, and expands relevant branches for finer-grained evidence. This process adaptively shifts the search target from temporal regions to specific objects and visual details to help video understanding. Experiments on VideoMME, LongVideoBench, and LVBench show that T^3 improves Qwen2.5-VL-7B by 0.5%, 4.6%, and 4.4%, respectively, under the same frame budget, demonstrating the effectiveness of structured temporal reasoning.
Multimodal Large Language Models (MLLMs) have made strong progress in video understanding, yet long videos remain difficult: the visual token budget grows with video length, so temporally sparse evidence is easily lost. Existing methods compress the input through uniform sampling or frame selection, but these strategies optimize different objectives, either broad temporal coverage or local question relevance, and neither preserves both global storyline context and fine-grained evidence. We propose VideoRouter (VR), which rethinks long-video understanding as coordinating complementary evidence views rather than selecting a single subset of frames. VideoRouter first organizes each video into a question-agnostic temporal hierarchy that partitions it into coarse-to-fine temporally coherent segments. Upper-level nodes capture broad storyline context and event progression, while lower-level nodes preserve fine-grained local details and evidence-bearing moments. This gives rise to two complementary views: a global view for coverage-oriented reasoning and a local view for detail-oriented evidence recovery. We further introduce a verification-guided router that judges which view is better supported by its own selected evidence and decides the final answer. Across six backbones, routing improves over both views in all settings, and the choice of view is shown to be dataset-dependent, confirming that no single evidence granularity is universally preferable. On VideoMME, our method outperforms state-of-the-art frame selection methods by 2.5 points, under the LLaVA-Video-7B backbone. We will release the code.