Organizations: University of Electronic Science and Technology of China · Institute of Software Chinese Academy of Sciences · Harbin Institute of Technology, Shenzhen · Chengdu Institute of Computer Applications, Chinese Academy of Sciences, University of the Chinese Academy of Sciences
Recent Video Large Language Models (Video-LLMs) have demonstrated strong capability in video understanding, yet they still suffer from hallucinations. Existing mitigation methods typically rely on training, input modification, auxiliary guidance, or additional decoding procedures, while largely overlooking a more fundamental challenge. During generation, Video-LLMs tend to over-rely on a limited portion of temporal evidence, leading to temporally imbalanced evidence aggregation across the video. To address this issue, we investigate a decoder-side phenomenon in which the model exhibits a temporally imbalanced concentration pattern. We term the frame with the highest aggregated frame-level attention mass the anchor frame. We find that this bias is largely independent of the input video and instead appears to reflect a persistent, model-specific structural or positional bias, whose over-dominance is closely associated with hallucination-prone generation. Motivated by this insight, we propose Decoder-side Temporal Rebalancing (DTR), a training-free, layer-selective inference method that rebalances temporal evidence allocation in middle-to-late decoder layers without altering visual encoding or requiring auxiliary models. DTR adaptively calibrates decoder-side visual attention to alleviate temporally imbalanced concentration and encourage under-attended frames to contribute more effectively to response generation. In this way, DTR guides the decoder to ground its outputs in temporally broader and more balanced video evidence. Extensive experiments on hallucination and video understanding benchmarks show that DTR consistently improves hallucination robustness across diverse Video-LLM families, while preserving competitive video understanding performance and high inference efficiency.
Figures & tables
Figure 1. Anchor-frame dominance in two Video-LLMs. Video-LLaVA focuses on frame 1, while LLaVA-NeXT-Video focuses on frame 7, leading to incorrect temporal judgments.
Figure 2. Anchor-frame behavior under black-frame intervention. Blue indicates the anchor-frame distribution under normal video inference, and red indicates the distribution after replacing the anchor frame with an all-black frame before visual encoding. In both Video-LLaVA and LLaVA-NeXT-Video, the dominant temporal position remains unchanged. Percentages denote how often each frame is selected as the anchor frame on the Object-Relation and Temporal subsets of VideoHallucer.
Figure 3. Layer-wise visual attention ratio in Video-LLaVA-7B. The curve shows the average visual attention ratio across decoder layers over samples from the Object-Relation and Temporal subsets of VideoHallucer. Visual attention remains low across most decoder layers, particularly in the middle-to-late range.
Setting
All Acc. (%)
Δ
Normal inference
27.12
0.00
Mask anchor frame only
29.29
+2.17
Randomly mask one frame
26.61
-0.51
Mask Frame 4 (non-anchor)
25.30
-1.82
Table 1. Masking study on VideoHallucer ( All accuracy, %) using LLaVA-NeXT-Video-7B. All video frames are first encoded normally by the visual encoder, and masking is applied only to the selected frame tokens in decoder layers 1–31.
Figure 4. Overview of the proposed decoder-side temporal rebalancing (DTR) framework. Given a video and a text prompt, DTR operates in selected middle-to-late decoder layers by computing frame-level scores from raw attention logits, estimating each frame’s gap to the current dominant frame, converting the normalized gaps into positive frame-wise biases, and injecting the biases into visual-token attention logits before softmax. The intervention is global and relative: it preserves all frames while assigning larger compensation to under-attended ones, thereby reducing anchor-frame over-dominance and improving temporal grounding.
Method
EventHallusion ( Zhang et al., 2024a )
VideoHallucer ( Wang et al., 2024b )
MVBench ( Li et al., 2024 )
Entire
Mix
Misleading
Overall
Object-Relation
Temporal
Semantic Detail
Factual
Non-factual
Overall
Overall
LLaVA-NeXT-Video-7B ( Zhang et al., 2024b )
Baseline ( Zhang et al., 2024b )
44.74
68.39
50.00
57.21
52.00
17.61
37.00
3.00
26.00
27.12
43.575
+TCD ( Zhang et al., 2024a )
45.61
69.43
52.94
58.68
51.00
14.20
36.00
1.50
22.50
25.04
43.77
+VCD ( Leng et al., 2024 )
42.11
63.73
51.96
54.77
45.50
29.55
39.50
19.50
38.50
34.51
43.53
+DINO-HEAL ( Li et al., 2025b )
47.37
67.88
50.98
57.95
51.00
18.75
37.50
3.50
27.00
27.55
43.65
Table 2. Results on LLaVA-based video models, including LLaVA-NeXT-Video-7B and Video-LLaVA-7B, across EventHallusion, VideoHallucer, and MVBench. Higher is better, and all values are reported as accuracy (%). Bold indicates the best result.
Method
EventHallusion ( Zhang et al., 2024a )
VideoHallucer (Hallucinated) ( Wang et al., 2024b )
MVBench ( Li et al., 2024 )
Entire
Mix
Misleading
Overall
Object-Relation
Temporal
Semantic Detail
Factual
Non-factual
Overall
Overall
Qwen2.5-VL-7B ( Bai et al., 2025b )
Baseline ( Bai et al., 2025b )
64.04
60.10
88.24
68.22
83.50
78.98
90.00
44.00
81.00
75.50
58.90
+TCD ( Zhang et al., 2024a )
61.40
63.73
84.31
68.22
83.00
74.43
88.50
38.50
76.00
72.09
58.55
+VCD ( Leng et al., 2024 )
59.65
60.10
86.27
66.50
80.50
73.86
86.50
44.00
80.50
73.07
57.20
+DINO-HEAL ( Li et al., 2025b )
69.30
58.55
86.27
68.46
83.00
78.98
91.50
44.50
81.00
75.80
58.93
Table 3. Results on Qwen-VL-based video models, including Qwen2.5-VL-7B and Qwen3-VL-8B, across EventHallusion, VideoHallucer, and MVBench. Higher is better, and all values are reported as accuracy (%). Bold indicates the best result.
Figure 5. Mid-layer attention before and after DTR on the same example. DTR redistributes attention across frames and corrects the answer from “Yes” to “No.”
Method
Samples
Avg. Time (s) ↓
Std. (s) ↓
Min. (s) ↓
Max. (s) ↓
Mean Memory (MB) ↓
Throughput (samples/s) ↑
Baseline ( Lin et al., 2024 )
300
0.607
0.047
0.485
0.728
16697.156
1.647
+TCD ( Zhang et al., 2024a )
300
1.187
0.060
0.976
1.321
18565.375
0.843
+VCD ( Leng et al., 2024 )
300
3.437
0.702
1.603
3.968
19195.997
0.291
+DINO-HEAL ( Li et al., 2025b )
300
0.785
0.209
0.535
1.229
17277.869
1.275
+DTR (Ours)
300
0.645
0.046
0.538
0.751
16697.163
1.550
Table 4. Efficiency comparison on 100 samples from the entire subset of EventHallusion. Each method is run three times, and the reported statistics are computed over all 300 inferences. Lower is better for time and memory, while higher is better for throughput.
Setting
Dom. ↓
Ent. ↑
Non-anchor ↑
Avg. Acc. ↑
Baseline
0.746
0.990
0.254
21.26
Global-only
0.558
1.495
0.442
37.59
Comp.-only
0.490
1.656
0.510
28.59
DTR
0.195
2.027
0.863
43.22
Table 5. Attention analysis on Video-LLaVA-7B over the Object-Relation and Temporal subsets from VideoHallucer. Dom. denotes the average anchor-frame dominance ratio, Entropy measures the dispersion of frame-level attention, and Non-anchor denotes the total attention mass assigned to non-anchor frames.
Figure 6. Ablation study on Video-LLaVA-7B (VideoHallucer). Each plot shows Overall Accuracy (%) and gain Δ over the baseline on the left and right y-axes, respectively. Left: Effect of global adjustment α ( β=0.4 , layers 18–31). Middle: Effect of deficit compensation β ( α=0.5 , layers 18–31). Right: Effect of intervention layer range ( α=0.5 , β=0.4 ). The dashed line marks the baseline accuracy (16.9%).
Figure 7. Anchor-position statistics under normal inputs and black-frame intervention for Video-LLaVA-7B and LLaVA-NeXT-Video-7B under 16-frame sampling. In the black-frame setting, the anchor frame identified from the normal input is replaced with an all-black frame before visual encoding. Percentages denote how often each frame is selected as the anchor frame on the Object-Relation and Temporal subsets of VideoHallucer.
Figure 8. Attention visualizations under 16-frame sampling for Video-LLaVA-7B and LLaVA-NeXT-Video-7B.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Frames
Baseline
+DTR (Ours)
Δ
4
65.53
71.39
+5.86
6
68.46
73.11
+4.65
8
68.22
73.84
+5.62
16
68.70
73.84
+5.14
18
66.99
73.59
+6.60
20
67.73
74.57
+6.84
Appendix
Table 6. Comparison of baseline and +DTR on Qwen2.5-VL-7B under different frame sampling settings on EventHallusion . We report only the All accuracy (%). Higher is better.
Figure 9. Black-frame analysis on Qwen2.5-VL-7B, aggregated over the Object-Relation and Temporal subsets of VideoHallucer . Since Qwen2.5-VL-7B groups every two consecutive frames into one temporal unit, we perform black-frame intervention at the temporal-unit level by simultaneously replacing the two frames within a unit with black frames. Starting from the normal input, we progressively mask temporal units from the beginning, i.e., Black TU0 , Black TU0-1 , …, Black TU0-6 . Percentages denote how often each temporal unit is selected as the anchor.
Figure 10. Anchor-frame distribution of InternVL3.5-8B, aggregated over the Object-Relation and Temporal subsets of VideoHallucer . The count at each frame position indicates how many times that frame is selected as the anchor across the evaluated samples. The model shows a clear overall preference for positions near the temporal boundaries.
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
School of Computer Science and Engineering, Beihang University · Beijing Institute for General Artificial Intelligence, BIGAI · Sino-French Engineer School, Beihang University
Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences Changchun, Jilin 130033, China · University of Chinese Academy of Sciences, Beijing 100049, China · City University of Hong Kong, Hong Kong, China