Organizations: University of Electronic Science and Technology of China · Institute of Software Chinese Academy of Sciences · Harbin Institute of Technology, Shenzhen · Chengdu Institute of Computer Applications, Chinese Academy of Sciences, University of the Chinese Academy of Sciences
Recent Video Large Language Models (Video-LLMs) have demonstrated strong capability in video understanding, yet they still suffer from hallucinations. Existing mitigation methods typically rely on training, input modification, auxiliary guidance, or additional decoding procedures, while largely overlooking a more fundamental challenge. During generation, Video-LLMs tend to over-rely on a limited portion of temporal evidence, leading to temporally imbalanced evidence aggregation across the video. To address this issue, we investigate a decoder-side phenomenon in which the model exhibits a temporally imbalanced concentration pattern. We term the frame with the highest aggregated frame-level attention mass the anchor frame. We find that this bias is largely independent of the input video and instead appears to reflect a persistent, model-specific structural or positional bias, whose over-dominance is closely associated with hallucination-prone generation. Motivated by this insight, we propose Decoder-side Temporal Rebalancing (DTR), a training-free, layer-selective inference method that rebalances temporal evidence allocation in middle-to-late decoder layers without altering visual encoding or requiring auxiliary models. DTR adaptively calibrates decoder-side visual attention to alleviate temporally imbalanced concentration and encourage under-attended frames to contribute more effectively to response generation. In this way, DTR guides the decoder to ground its outputs in temporally broader and more balanced video evidence. Extensive experiments on hallucination and video understanding benchmarks show that DTR consistently improves hallucination robustness across diverse Video-LLM families, while preserving competitive video understanding performance and high inference efficiency.
Figures & tables
Figure 1. Anchor-frame dominance in two Video-LLMs. Video-LLaVA focuses on frame 1, while LLaVA-NeXT-Video focuses on frame 7, leading to incorrect temporal judgments.
Figure 2. Anchor-frame behavior under black-frame intervention. Blue indicates the anchor-frame distribution under normal video inference, and red indicates the distribution after replacing the anchor frame with an all-black frame before visual encoding. In both Video-LLaVA and LLaVA-NeXT-Video, the dominant temporal position remains unchanged. Percentages denote how often each frame is selected as the anchor frame on the Object-Relation and Temporal subsets of VideoHallucer.
Figure 3. Layer-wise visual attention ratio in Video-LLaVA-7B. The curve shows the average visual attention ratio across decoder layers over samples from the Object-Relation and Temporal subsets of VideoHallucer. Visual attention remains low across most decoder layers, particularly in the middle-to-late range.
Setting
All Acc. (%)
Δ
Normal inference
27.12
0.00
Mask anchor frame only
29.29
+2.17
Randomly mask one frame
26.61
-0.51
Mask Frame 4 (non-anchor)
25.30
-1.82
Table 1. Masking study on VideoHallucer ( All accuracy, %) using LLaVA-NeXT-Video-7B. All video frames are first encoded normally by the visual encoder, and masking is applied only to the selected frame tokens in decoder layers 1–31.
Figure 4. Overview of the proposed decoder-side temporal rebalancing (DTR) framework. Given a video and a text prompt, DTR operates in selected middle-to-late decoder layers by computing frame-level scores from raw attention logits, estimating each frame’s gap to the current dominant frame, converting the normalized gaps into positive frame-wise biases, and injecting the biases into visual-token attention logits before softmax. The intervention is global and relative: it preserves all frames while assigning larger compensation to under-attended ones, thereby reducing anchor-frame over-dominance and improving temporal grounding.
Method
EventHallusion ( Zhang et al., 2024a )
VideoHallucer ( Wang et al., 2024b )
MVBench ( Li et al., 2024 )
Entire
Mix
Misleading
Overall
Object-Relation
Temporal
Semantic Detail
Factual
Non-factual
Overall
Overall
LLaVA-NeXT-Video-7B ( Zhang et al., 2024b )
Baseline ( Zhang et al., 2024b )
44.74
68.39
50.00
57.21
52.00
17.61
37.00
3.00
26.00
27.12
43.575
+TCD ( Zhang et al., 2024a )
45.61
69.43
52.94
58.68
51.00
14.20
36.00
1.50
22.50
25.04
43.77
+VCD ( Leng et al., 2024 )
42.11
63.73
51.96
54.77
45.50
29.55
39.50
19.50
38.50
34.51
43.53
+DINO-HEAL ( Li et al., 2025b )
47.37
67.88
50.98
57.95
51.00
18.75
37.50
3.50
27.00
27.55
43.65
Table 2. Results on LLaVA-based video models, including LLaVA-NeXT-Video-7B and Video-LLaVA-7B, across EventHallusion, VideoHallucer, and MVBench. Higher is better, and all values are reported as accuracy (%). Bold indicates the best result.
Method
EventHallusion ( Zhang et al., 2024a )
VideoHallucer (Hallucinated) ( Wang et al., 2024b )
MVBench ( Li et al., 2024 )
Entire
Mix
Misleading
Overall
Object-Relation
Temporal
Semantic Detail
Factual
Non-factual
Overall
Overall
Qwen2.5-VL-7B ( Bai et al., 2025b )
Baseline ( Bai et al., 2025b )
64.04
60.10
88.24
68.22
83.50
78.98
90.00
44.00
81.00
75.50
58.90
+TCD ( Zhang et al., 2024a )
61.40
63.73
84.31
68.22
83.00
74.43
88.50
38.50
76.00
72.09
58.55
+VCD ( Leng et al., 2024 )
59.65
60.10
86.27
66.50
80.50
73.86
86.50
44.00
80.50
73.07
57.20
+DINO-HEAL ( Li et al., 2025b )
69.30
58.55
86.27
68.46
83.00
78.98
91.50
44.50
81.00
75.80
58.93
Table 3. Results on Qwen-VL-based video models, including Qwen2.5-VL-7B and Qwen3-VL-8B, across EventHallusion, VideoHallucer, and MVBench. Higher is better, and all values are reported as accuracy (%). Bold indicates the best result.
Figure 5. Mid-layer attention before and after DTR on the same example. DTR redistributes attention across frames and corrects the answer from “Yes” to “No.”
Method
Samples
Avg. Time (s) ↓
Std. (s) ↓
Min. (s) ↓
Max. (s) ↓
Mean Memory (MB) ↓
Throughput (samples/s) ↑
Baseline ( Lin et al., 2024 )
300
0.607
0.047
0.485
0.728
16697.156
1.647
+TCD ( Zhang et al., 2024a )
300
1.187
0.060
0.976
1.321
18565.375
0.843
+VCD ( Leng et al., 2024 )
300
3.437
0.702
1.603
3.968
19195.997
0.291
+DINO-HEAL ( Li et al., 2025b )
300
0.785
0.209
0.535
1.229
17277.869
1.275
+DTR (Ours)
300
0.645
0.046
0.538
0.751
16697.163
1.550
Table 4. Efficiency comparison on 100 samples from the entire subset of EventHallusion. Each method is run three times, and the reported statistics are computed over all 300 inferences. Lower is better for time and memory, while higher is better for throughput.
Setting
Dom. ↓
Ent. ↑
Non-anchor ↑
Avg. Acc. ↑
Baseline
0.746
0.990
0.254
21.26
Global-only
0.558
1.495
0.442
37.59
Comp.-only
0.490
1.656
0.510
28.59
DTR
0.195
2.027
0.863
43.22
Table 5. Attention analysis on Video-LLaVA-7B over the Object-Relation and Temporal subsets from VideoHallucer. Dom. denotes the average anchor-frame dominance ratio, Entropy measures the dispersion of frame-level attention, and Non-anchor denotes the total attention mass assigned to non-anchor frames.
Figure 6. Ablation study on Video-LLaVA-7B (VideoHallucer). Each plot shows Overall Accuracy (%) and gain Δ over the baseline on the left and right y-axes, respectively. Left: Effect of global adjustment α ( β=0.4 , layers 18–31). Middle: Effect of deficit compensation β ( α=0.5 , layers 18–31). Right: Effect of intervention layer range ( α=0.5 , β=0.4 ). The dashed line marks the baseline accuracy (16.9%).
Figure 7. Anchor-position statistics under normal inputs and black-frame intervention for Video-LLaVA-7B and LLaVA-NeXT-Video-7B under 16-frame sampling. In the black-frame setting, the anchor frame identified from the normal input is replaced with an all-black frame before visual encoding. Percentages denote how often each frame is selected as the anchor frame on the Object-Relation and Temporal subsets of VideoHallucer.
Figure 8. Attention visualizations under 16-frame sampling for Video-LLaVA-7B and LLaVA-NeXT-Video-7B.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Frames
Baseline
+DTR (Ours)
Δ
4
65.53
71.39
+5.86
6
68.46
73.11
+4.65
8
68.22
73.84
+5.62
16
68.70
73.84
+5.14
18
66.99
73.59
+6.60
20
67.73
74.57
+6.84
Appendix
Table 6. Comparison of baseline and +DTR on Qwen2.5-VL-7B under different frame sampling settings on EventHallusion . We report only the All accuracy (%). Higher is better.
Figure 9. Black-frame analysis on Qwen2.5-VL-7B, aggregated over the Object-Relation and Temporal subsets of VideoHallucer . Since Qwen2.5-VL-7B groups every two consecutive frames into one temporal unit, we perform black-frame intervention at the temporal-unit level by simultaneously replacing the two frames within a unit with black frames. Starting from the normal input, we progressively mask temporal units from the beginning, i.e., Black TU0 , Black TU0-1 , …, Black TU0-6 . Percentages denote how often each temporal unit is selected as the anchor.
Figure 10. Anchor-frame distribution of InternVL3.5-8B, aggregated over the Object-Relation and Temporal subsets of VideoHallucer . The count at each frame position indicates how many times that frame is selected as the anchor across the evaluated samples. The model shows a clear overall preference for positions near the temporal boundaries.
Video Large Language Models (Video-LLMs) remain prone to spatiotemporal hallucinations, often generating visually unsupported details or incorrect temporal relations. Existing mitigation methods typically treat hallucination as a uniform decoding failure, applying globally shared correction rules. We instead observe that decoder layers contribute differently to visual grounding and later linguistic composition, indicating that intervention must be layer-aware. Based on this insight, we propose STEAR, a layer-aware spatiotemporal evidence intervention framework. STEAR identifies high-risk decoding steps and selects token-conditioned visual evidence from grounding-sensitive middle layers. It uses this shared evidence for two coupled purposes: restoring missing local grounding in middle layers, and constructing temporally perturbed patch-level counterfactuals to falsify inconsistent reasoning during late-layer decoding. Consequently, STEAR mitigates both spatial and temporal hallucinations within an efficient single-encode inference framework. Experiments across representative Video-LLM backbones and challenging benchmarks demonstrate that STEAR consistently reduces hallucinations while improving faithfulness, temporal consistency, and robustness. Our results confirm that reliable video decoding relies on intervening on precise evidence at the right layer, rather than enforcing a global penalty. The code is provided in the Supplementary Material.
Linfeng Fan, Yuan Tian, Ziwei Li +1
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
Video large language models (Video LLMs) often generate responses unsupported by the input video. Contrastive decoding (CD) can mitigate these hallucinations, but its effectiveness depends on how the negative branch weakens visual evidence. Existing methods rely on predefined perturbations or masking guided by question reconstruction, offering limited control over the model's visual reliance during answer generation. We introduce Attention-guided Representation Perturbation for Contrastive Decoding (ARP-CD), a training-free framework for constructing contrastive negatives directly in representation space. ARP-CD uses the attention mass assigned to visual tokens at the answer-prediction position to guide bounded gradient updates of the language model's input visual hidden states. These updates reduce visual attention and produce a negative input, whose predictions are contrasted with those of the original branch throughout generation. Experiments across four Video LLMs and four benchmarks show that ARP-CD outperforms representative training-free baselines in most model--benchmark settings, reducing hallucinations while improving general video understanding. Further analyses show that attention-guided perturbation directions improve negative quality and that the resulting negatives provide effective contrastive references for correcting the original predictions.
Zhixin Cai, Ruoqi Yuan, Xinchen Yang +4
School of Computer Science and Engineering, Beihang University · Beijing Institute for General Artificial Intelligence, BIGAI · Sino-French Engineer School, Beihang University
Large vision-language models (LVLMs) have demonstrated strong performance in open-ended video understanding, yet they remain prone to fluent responses unsupported by video evidence. Existing training-free methods typically apply a globally fixed visual intervention or construct a contrastive branch through input perturbation. The former cannot accommodate video-dependent fusion paths, while the latter can be compensated by cross-frame redundancy. We therefore propose Video-Adaptive Debiasing via Evidence Reweighting (VADER), a training-free framework with two complementary modules. Visual Focus Reallocation (VFR) automatically instantiates an intervention policy for each video-question input: it diagnoses layer-wise visual-to-text evidence flow, determines where to intervene, and derives how strongly to reallocate pre-softmax attention from system-token to video-token blocks. Selective Evidence Erasure (SEE) independently masks high-importance visual tokens in every frame, constructing a prior-biased branch that is difficult to compensate through neighboring frames. Contrastive decoding then down-weights predictions that remain confident after selective evidence erasure. Across multiple VideoLLMs, VADER yields substantial improvements on event-level grounding and temporal consistency; on LLaVA-Video-7B, it reaches 72.60% accuracy on EventHallusion.
Dong Xing, Jiaxin Chen, Hang Yang +3
Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences Changchun, Jilin 130033, China · University of Chinese Academy of Sciences, Beijing 100049, China · City University of Hong Kong, Hong Kong, China