Organizations: City University of Hong Kong · Shenzhen Loop Institute · Xidian University · JX Intelligence Technology Co., Ltd. · University of Hong Kong
Long-video question answering is challenging since the answer often hinges on a few decisive moments scattered throughout the video, while memory constraints force Multimodal Large Language Models (MLLMs) to sample frames at extremely low rates, resulting in sparsity that risks omitting key evidence. Though aggressively enlarging the context size is intuitive, we argue that performance remains fundamentally constrained by the bias, i.e., the misleading cues caused by spurious linguistic correlations and salient yet irrelevant visual observations. In this paper, we first introduce {S-MME}, a benchmark designed to systematically diagnose shortcut-biased behavior in MLLMs. To mitigate this failure, we further propose a training-free framework, Counterfactual Long-video Evidence-Aware Reasoning (CLEAR). By treating the full video-question pair as a complete view, CLEAR estimates bias effects via augmented views by only keeping the question or locally salient visual clips. Then, we intervene hidden states to mitigate the discrepancy at inference such that the correct answer can be predicted confidently. With experimental studies across five benchmarks, we show that our method is generalizable and can be integrated in different MLLMs, yielding consistent gains in prediction and robustness. We hope CLEAR can offer a complementary direction in test-time scaling for long-video understanding.
Figures & tables
Figure 1: MLLMs often fail despite the presence of decisive evidence, either by following spurious linguistic correlations or by over-relying on salient but misleading visual observations. CLEAR diagnoses these shortcut-biased trajectories with controlled probes and steers the full-video representation toward evidence-consistent prediction.
Figure 2: Construction pipeline of S-MME. We first mine candidate questions with explicit long-video evidence requirements, then use restricted views to identify shortcut-consistent distractors, followed by human validation of answer uniqueness, evidence sufficiency, and shortcut plausibility.
Category
Count
Avg. Dur. (min)
Min Dur. (min)
Max Dur. (min)
Linguistic-Correlation Conflict
148
11.9
5.6
74.7
Salient-Visual Conflict
172
17.6
12.8
106.6
Total
320
15.0
5.6
106.6
Table 1: Statistics of the S-MME diagnostic benchmark.
Figure 3: Overview of CLEAR. (a) The full-video path produces the final answer, while question-only and salient-view probes reveal spurious linguistic correlations and salient-yet-misleading visual observations. The salient view is formed by selecting top- K locally relevant clips according to query-conditioned visual attention. (b) CLEAR extracts the hidden state of the shared decision token z at selected late decoder layers, and constructs residual vectors rlq=hlf−hlq and rls=hlf−hls from the restricted-view probes to the full-video state. These residual vectors are injected into the full-video path to steer its hidden state away from shortcut anchors before final decoding.
Figure 4: Effect of intervention location. Late-layer intervention performs best, with the last block achieving the highest accuracy.
Model
Size
Frames
MLVU
Video-MME
LongVB
LVBench
Overall
Long
Closed-source MLLMs
GPT-4o [ 28 ]
–
–
54.9
65.3
59.0
66.7
48.9
Gemini-1.5-Pro [ 35 ]
–
–
53.8
67.4
61.8
64.0
33.1
Open-source video MLLMs
LongVU [ 31 ]
7B
1 fps †
63.7
60.9
54.3
59.5
–
Table 2: General comparison on long-video understanding benchmarks. CLEAR is evaluated on three open-source MLLMs, with closed-source and public models included for reference.
Model
Size
Frames
Lang.-Prior
Part.-Evid.
Overall
Acc. ↑
ShortErr. ↓
Acc. ↑
ShortErr. ↓
Acc. ↑
ShortErr. ↓
Human Reference
Human
–
–
92.4
2.6
90.1
3.8
91.2
3.2
Closed-source MLLMs
GPT-4o
–
–
47.2
18.6
43.8
22.4
45.5
20.5
Gemini-1.5-Pro
–
–
54.1
14.2
50.3
17.9
52.2
16.1
Table 3: Diagnostic comparison on S-MME. ShortErr denotes the percentage of predictions that incorrectly choose the annotated shortcut-consistent distractor.
Variant
Q
S
Branch
Linguistic Bias
Visual Bias
Overall
Acc. ↑
ShortErr. ↓
Acc. ↑
ShortErr. ↓
Acc. ↑
Baseline
–
–
–
22.2
41.7
18.8
46.2
20.5
Q-only
✓
–
Single
26.1
36.8
20.4
43.5
23.3
S-only
–
✓
Single
24.5
38.7
22.9
41.0
23.7
CLEAR
✓
✓
Dual
28.4
34.1
24.5
39.6
26.5
Table 4: Effect of shortcut probes on S-MME. Q-only and S-only apply residual correction from the language-prior and salient-view probes, respectively. CLEAR combines both probes for complementary correction.
Method
Final F.
Aux. Probe
Time
Acc. ↑
Qwen3-VL
32
–
0.62 ×
62.8
Qwen3-VL
64
–
1.00 ×
65.1
Qwen3-VL
128
–
1.86 ×
67.0
Qwen3-VL + CLEAR
32
Q+S
1.34 ×
68.4 ↑ 5.6
Qwen3-VL + CLEAR
64
Q+S
1.78 ×
70.9 ↑ 5.8
Qwen3-VL + CLEAR
128
Q+S
2.72 ×
71.5 ↑ 4.5
Table 5: Efficiency and intervention analysis. (a) On Video-MME, CLEAR outperforms simple frame scaling under comparable inference budgets. (b) On S-MME, moderate residual strength yields the best accuracy–shortcut suppression trade-off.
Figure 5: Context scaling analysis. CLEAR consistently improves accuracy and reduces ShortErr under the same frame budget.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Frames
Method
VideoEvalPro
LLaVA-Video-7B
64
Baseline
22.8
LLaVA-Video-7B
64
+ CLEAR
24.0
InternVL3-8B
64
Baseline
23.8
InternVL3-8B
64
+ CLEAR
25.3
Qwen3-VL-8B
64
Baseline
25.2
Qwen3-VL-8B
64
+ CLEAR
28.6
Appendix
Table 6: Open-ended evaluation on VideoEvalPro [ 25 ] . CLEAR improves free-form video question answering, suggesting that its gains are not limited to multiple-choice task.
Figure 6: Qualitative effect of CLEAR on open-ended video QA.
Recent advances in Video Large Language Models (Video-LLMs) have enabled performance on long-video understanding tasks. However, existing methods still face two key limitations: evidence acquisition often relies on a single search intent, and answer generation lacks an effective visual feedback mechanism. To address these limitations, we propose CoVER, a Comprehensive Visual Evidence and Reflection framework for long-video understanding. CoVER enables Video-LLMs to See More by dynamically gathering query-expanded visual evidence, and Think Deeper by verifying draft answers with effective answer-specific visual feedback. Together, these mechanisms shift long-video understanding from answer-centric generation to evidence-centric and visually verifiable reasoning. Experimental results show that CoVER-7B substantially outperforms models with the same parameter scale and even surpasses state-of-the-art closed-source models on certain metrics.
Shuning Wang, Zhiheng Wu, YiNuo Lu +6
Baidu Inc. · Harbin Institute of Technology · Hong Kong University of Science and Technology
Long video understanding remains challenging for Multi-modal Large Language Models (MLLMs) due to high memory costs and context-length limits. Prior approaches mitigate this by scoring and selecting frames/tokens within short clips, but they lack a principled mechanism to (i) compare relevance across distant video clips and (ii) stop processing once sufficient evidence has been gathered. We propose AdaptToken, a training-free framework that turns an MLLM's self-uncertainty into a global control signal for long-video token selection. AdaptToken splits a video into groups, extracts cross-modal attention to rank tokens within each group, and uses the model's response entropy to estimate each group's prompt relevance. This entropy signal enables a global token budget allocation across groups and further supports early stopping (AdaptToken-Lite), skipping the remaining groups when the model becomes sufficiently certain. Across four long-video benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) and multiple base MLLMs (7B-72B), AdaptToken consistently improves accuracy (e.g., +6.7 on average over Qwen2.5-VL 7B) and continues to benefit from extremely long inputs (up to 10K frames), while AdaptToken-Lite reduces inference time by about half with comparable performance. Project page: https://haozheqi.github.io/adapt-token
Multimodal Large Language Models (MLLMs) have made strong progress in video understanding, yet long videos remain difficult: the visual token budget grows with video length, so temporally sparse evidence is easily lost. Existing methods compress the input through uniform sampling or frame selection, but these strategies optimize different objectives, either broad temporal coverage or local question relevance, and neither preserves both global storyline context and fine-grained evidence. We propose VideoRouter (VR), which rethinks long-video understanding as coordinating complementary evidence views rather than selecting a single subset of frames. VideoRouter first organizes each video into a question-agnostic temporal hierarchy that partitions it into coarse-to-fine temporally coherent segments. Upper-level nodes capture broad storyline context and event progression, while lower-level nodes preserve fine-grained local details and evidence-bearing moments. This gives rise to two complementary views: a global view for coverage-oriented reasoning and a local view for detail-oriented evidence recovery. We further introduce a verification-guided router that judges which view is better supported by its own selected evidence and decides the final answer. Across six backbones, routing improves over both views in all settings, and the choice of view is shown to be dataset-dependent, confirming that no single evidence granularity is universally preferable. On VideoMME, our method outperforms state-of-the-art frame selection methods by 2.5 points, under the LLaVA-Video-7B backbone. We will release the code.