Organizations: City University of Hong Kong · Shenzhen Loop Institute · Xidian University · JX Intelligence Technology Co., Ltd. · University of Hong Kong
Long-video question answering is challenging since the answer often hinges on a few decisive moments scattered throughout the video, while memory constraints force Multimodal Large Language Models (MLLMs) to sample frames at extremely low rates, resulting in sparsity that risks omitting key evidence. Though aggressively enlarging the context size is intuitive, we argue that performance remains fundamentally constrained by the bias, i.e., the misleading cues caused by spurious linguistic correlations and salient yet irrelevant visual observations. In this paper, we first introduce {S-MME}, a benchmark designed to systematically diagnose shortcut-biased behavior in MLLMs. To mitigate this failure, we further propose a training-free framework, Counterfactual Long-video Evidence-Aware Reasoning (CLEAR). By treating the full video-question pair as a complete view, CLEAR estimates bias effects via augmented views by only keeping the question or locally salient visual clips. Then, we intervene hidden states to mitigate the discrepancy at inference such that the correct answer can be predicted confidently. With experimental studies across five benchmarks, we show that our method is generalizable and can be integrated in different MLLMs, yielding consistent gains in prediction and robustness. We hope CLEAR can offer a complementary direction in test-time scaling for long-video understanding.
Figures & tables
Figure 1: MLLMs often fail despite the presence of decisive evidence, either by following spurious linguistic correlations or by over-relying on salient but misleading visual observations. CLEAR diagnoses these shortcut-biased trajectories with controlled probes and steers the full-video representation toward evidence-consistent prediction.
Figure 2: Construction pipeline of S-MME. We first mine candidate questions with explicit long-video evidence requirements, then use restricted views to identify shortcut-consistent distractors, followed by human validation of answer uniqueness, evidence sufficiency, and shortcut plausibility.
Category
Count
Avg. Dur. (min)
Min Dur. (min)
Max Dur. (min)
Linguistic-Correlation Conflict
148
11.9
5.6
74.7
Salient-Visual Conflict
172
17.6
12.8
106.6
Total
320
15.0
5.6
106.6
Table 1: Statistics of the S-MME diagnostic benchmark.
Figure 3: Overview of CLEAR. (a) The full-video path produces the final answer, while question-only and salient-view probes reveal spurious linguistic correlations and salient-yet-misleading visual observations. The salient view is formed by selecting top- K locally relevant clips according to query-conditioned visual attention. (b) CLEAR extracts the hidden state of the shared decision token z at selected late decoder layers, and constructs residual vectors rlq=hlf−hlq and rls=hlf−hls from the restricted-view probes to the full-video state. These residual vectors are injected into the full-video path to steer its hidden state away from shortcut anchors before final decoding.
Figure 4: Effect of intervention location. Late-layer intervention performs best, with the last block achieving the highest accuracy.
Model
Size
Frames
MLVU
Video-MME
LongVB
LVBench
Overall
Long
Closed-source MLLMs
GPT-4o [ 28 ]
–
–
54.9
65.3
59.0
66.7
48.9
Gemini-1.5-Pro [ 35 ]
–
–
53.8
67.4
61.8
64.0
33.1
Open-source video MLLMs
LongVU [ 31 ]
7B
1 fps †
63.7
60.9
54.3
59.5
–
Table 2: General comparison on long-video understanding benchmarks. CLEAR is evaluated on three open-source MLLMs, with closed-source and public models included for reference.
Model
Size
Frames
Lang.-Prior
Part.-Evid.
Overall
Acc. ↑
ShortErr. ↓
Acc. ↑
ShortErr. ↓
Acc. ↑
ShortErr. ↓
Human Reference
Human
–
–
92.4
2.6
90.1
3.8
91.2
3.2
Closed-source MLLMs
GPT-4o
–
–
47.2
18.6
43.8
22.4
45.5
20.5
Gemini-1.5-Pro
–
–
54.1
14.2
50.3
17.9
52.2
16.1
Table 3: Diagnostic comparison on S-MME. ShortErr denotes the percentage of predictions that incorrectly choose the annotated shortcut-consistent distractor.
Variant
Q
S
Branch
Linguistic Bias
Visual Bias
Overall
Acc. ↑
ShortErr. ↓
Acc. ↑
ShortErr. ↓
Acc. ↑
Baseline
–
–
–
22.2
41.7
18.8
46.2
20.5
Q-only
✓
–
Single
26.1
36.8
20.4
43.5
23.3
S-only
–
✓
Single
24.5
38.7
22.9
41.0
23.7
CLEAR
✓
✓
Dual
28.4
34.1
24.5
39.6
26.5
Table 4: Effect of shortcut probes on S-MME. Q-only and S-only apply residual correction from the language-prior and salient-view probes, respectively. CLEAR combines both probes for complementary correction.
Method
Final F.
Aux. Probe
Time
Acc. ↑
Qwen3-VL
32
–
0.62 ×
62.8
Qwen3-VL
64
–
1.00 ×
65.1
Qwen3-VL
128
–
1.86 ×
67.0
Qwen3-VL + CLEAR
32
Q+S
1.34 ×
68.4 ↑ 5.6
Qwen3-VL + CLEAR
64
Q+S
1.78 ×
70.9 ↑ 5.8
Qwen3-VL + CLEAR
128
Q+S
2.72 ×
71.5 ↑ 4.5
Table 5: Efficiency and intervention analysis. (a) On Video-MME, CLEAR outperforms simple frame scaling under comparable inference budgets. (b) On S-MME, moderate residual strength yields the best accuracy–shortcut suppression trade-off.
Figure 5: Context scaling analysis. CLEAR consistently improves accuracy and reduces ShortErr under the same frame budget.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Frames
Method
VideoEvalPro
LLaVA-Video-7B
64
Baseline
22.8
LLaVA-Video-7B
64
+ CLEAR
24.0
InternVL3-8B
64
Baseline
23.8
InternVL3-8B
64
+ CLEAR
25.3
Qwen3-VL-8B
64
Baseline
25.2
Qwen3-VL-8B
64
+ CLEAR
28.6
Appendix
Table 6: Open-ended evaluation on VideoEvalPro [ 25 ] . CLEAR improves free-form video question answering, suggesting that its gains are not limited to multiple-choice task.
Figure 6: Qualitative effect of CLEAR on open-ended video QA.