STEAR: Layer-Aware Spatiotemporal Evidence Intervention for Hallucination Mitigation in Video Large Language Models

Linfeng Fan; Yuan Tian; Ziwei Li; Zhiwu Lu

cs.CVApr 3, 2026

STEAR: Layer-Aware Spatiotemporal Evidence Intervention for Hallucination Mitigation in Video Large Language Models

Authors: Linfeng Fan, Yuan Tian, Ziwei Li, Zhiwu Lu

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · King Abdullah University of Science and Technology, Thuwal, Saudi Arabia

Abstract

Video Large Language Models (Video-LLMs) remain prone to spatiotemporal hallucinations, often generating visually unsupported details or incorrect temporal relations. Existing mitigation methods typically treat hallucination as a uniform decoding failure, applying globally shared correction rules. We instead observe that decoder layers contribute differently to visual grounding and later linguistic composition, indicating that intervention must be layer-aware. Based on this insight, we propose STEAR, a layer-aware spatiotemporal evidence intervention framework. STEAR identifies high-risk decoding steps and selects token-conditioned visual evidence from grounding-sensitive middle layers. It uses this shared evidence for two coupled purposes: restoring missing local grounding in middle layers, and constructing temporally perturbed patch-level counterfactuals to falsify inconsistent reasoning during late-layer decoding. Consequently, STEAR mitigates both spatial and temporal hallucinations within an efficient single-encode inference framework. Experiments across representative Video-LLM backbones and challenging benchmarks demonstrate that STEAR consistently reduces hallucinations while improving faithfulness, temporal consistency, and robustness. Our results confirm that reliable video decoding relies on intervening on precise evidence at the right layer, rather than enforcing a global penalty. The code is provided in the Supplementary Material.

Figures & tables

Figure 1. Given a single video encoding, STEAR first diagnoses risky decoding steps via token uncertainty, then selects token-conditioned key evidence shared by both middle-layer reinjection and deep-layer counterfactual decoding. This unified design addresses spatial hallucination by restoring missing grounding and suppresses temporal hallucination by falsifying temporally corrupted key evidence. dsp1

Figure 2. (a) Grounding score peaks in the middle decoder layers, while language dominance drops in the middle and rises again in the late layers, indicating a transition from evidence consolidation to language-dominant decoding. This suggests that hallucinations arise when token-relevant visual evidence is not sufficiently grounded before the model enters the late reasoning regime. (b) Middle-layer reinjection yields the largest gain, whereas others are ineffective. dsp2

Figure 3. For each risky token, KES aggregates middle-layer cross-attention to identify the top- $k$ key patches that best support the current prediction. The selected evidence is then reinjected into grounding-sensitive middle layers, restoring token-required but potentially neglected visual support without disturbing stable decoding steps. dsp2

Backbone	Method	EventHallusion $\uparrow$	VidHalluc $\uparrow$	NExT-QA $\uparrow$	MVBench $\uparrow$	Avg. $\uparrow$
LLaVA-Video-7B	Baseline	46.70 $\pm$ 0.00	63.07 $\pm$ 0.00	58.43 $\pm$ 0.00	41.28 $\pm$ 0.00	52.37 $\pm$ 0.00
	+VCD	40.17 $\downarrow$ 6.53	65.48 $\uparrow$ 2.41	60.27 $\uparrow$ 1.84	38.92 $\downarrow$ 2.36	51.21 $\downarrow$ 1.16
	+MemVR	47.10 $\uparrow$ 0.40	65.69 $\uparrow$ 2.62	60.47 $\uparrow$ 2.04	41.25 $\downarrow$ 0.03	53.63 $\uparrow$ 1.26
	+TCD	51.62 $\uparrow$ 4.92	64.39 $\uparrow$ 1.32	60.27 $\uparrow$ 1.84	37.47 $\downarrow$ 3.81	53.44 $\uparrow$ 1.07
	+DINO-HEAL	46.70 $\pm$ 0.00	65.58 $\uparrow$ 2.51	60.27 $\uparrow$ 1.84	40.94 $\downarrow$ 0.34	53.37 $\uparrow$ 1.00
	+ STEAR (Ours)	54.77 $\uparrow$ 8.07	65.81 $\uparrow$ 2.74	61.88 $\uparrow$ 3.45	41.74 $\uparrow$ 0.46	56.05 $\uparrow$ 3.68

Table 1. Cross-backbone overall comparison. All numbers are accuracies (%). Avg. denotes the arithmetic mean over the four reported overall metrics under the same backbone. For each non-baseline method, the inline delta shows the absolute change relative to the corresponding baseline of the same backbone. Best and second-best results within each backbone are highlighted in bold and underline, respectively.

Method	EventHallusion $\uparrow$				NExT-QA $\uparrow$				MVBench $\uparrow$
	Entire	Misleading	Mix	Overall	C	T	D	Acc.	Loc.	Pred.	Seq.	Obj.	Overall
Baseline	29.82	43.13	58.55	46.70	57.32	53.44	72.53	58.43	29.33	36.60	55.12	41.60	41.28
+MemVR	30.70	42.15	59.06	47.10	59.65	55.33	73.87	60.47	28.83	36.80	55.12	41.30	41.25
+VCD	28.90	43.10	45.10	40.10	59.38	55.33	73.49	60.27	29.17	33.10	52.00	39.60	38.92
+TCD	47.36	43.13	58.54	51.62	59.61	55.02	73.36	60.27	27.33	33.60	47.00	39.20	37.47
+DINO-HEAL	28.90	43.10	59.10	46.70	59.49	55.09	73.62	60.27	29.33	35.70	54.75	41.30	40.94

Table 2. Fine-grained breakdown on LLaVA-Video-7B. STEAR is particularly strong on misleading event cases and consistently improves all NExT-QA and MVBench sub-groups. Best results are in bold and second-best results are underlined.

Method	Enc.	Rel. Latency $\downarrow$	Rel. Memory $\downarrow$	EH Overall $\uparrow$
VCD	2	2.42 $\times$	1.13 $\times$	40.17
TCD	2	2.25 $\times$	1.13 $\times$	51.62
STEAR (Ours)	1	1.89 $\times$	1.01 $\times$	54.77

Table 3. Efficiency comparison on LLaVA-Video-7B. “Enc.” denotes the number of visual encoding passes, and “EH” refers to EventHallusion. STEAR preserves a single-encode design while achieving the strongest hallucination mitigation performance.

Core components
Variant	EventHallusion Overall $\uparrow$	VidHalluc (ACH-BQA) $\uparrow$
Baseline	47.43 $\pm$ 0.00	41.02 $\pm$ 0.00
STEAR (full)	54.77 $\uparrow$ 7.34	44.14 $\uparrow$ 3.12
w/o uncertainty trigger (apply to all tokens)	48.41 $\uparrow$ 0.98	41.91 $\uparrow$ 0.89
w/o middle-layer reinjection	50.61 $\uparrow$ 3.18	43.50 $\uparrow$ 2.48
w/o temporal counterfactual decoding	47.92 $\uparrow$ 0.49	41.37 $\uparrow$ 0.35

Table 4. Core ablations on LLaVA-Video-7B for validating the mechanism of STEAR. The table is organized to test three claims: selective intervention via uncertainty triggering, grounding repair in middle layers, and evidence-level temporal falsification through local counterfactuals. Inline deltas show absolute changes relative to the baseline.

Figure 4. Compared with Video-LLMs, STEAR better grounds local evidence and suppresses temporally inconsistent generations. dsp5

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Models
Apr 14, 2026Zijian Liu, Sihan Cao, Pengcheng Zheng +5Video-Language Models Attention Mechanisms
University of Electronic Science and Technology of China · Institute of Software Chinese Academy of Sciences · Harbin Institute of Technology, Shenzhen +1
VADER: Adaptive Debiasing for Hallucination Mitigation in Video Large Language Models
Aug 9, 2026Dong Xing, Jiaxin Chen, Hang Yang +3Video-Language Models VLM Hallucination Mitigation
Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences Changchun, Jilin 130033, China · University of Chinese Academy of Sciences, Beijing 100049, China · City University of Hong Kong, Hong Kong, China
Mitigating Video Hallucinations via Attention-Guided Representation Perturbation
Oct 14, 2026Zhixin Cai, Ruoqi Yuan, Xinchen Yang +4VLM Hallucination Mitigation Hallucination Mitigation
School of Computer Science and Engineering, Beihang University · Beijing Institute for General Artificial Intelligence, BIGAI · Sino-French Engineer School, Beihang University

Apr 14, 2026J/K move · Enter open · S save

Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Models

Zijian Liu, Sihan Cao, Pengcheng Zheng, Kuien Liu+4

University of Electronic Science and Technology of China · Institute of Software Chinese Academy of Sciences · Harbin Institute of Technology, Shenzhen+1

Video-Language Models Attention Mechanisms Temporal Video Understanding

Open paper AskPDF arXiv

Recent Video Large Language Models (Video-LLMs) have demonstrated strong capability in video understanding, yet they still suffer from hallucinations. Existing mitigation methods typically rely on training, input modification, auxiliary guidance, or additional decoding procedures, while largely overlooking a more fundamental challenge. During generation, Video-LLMs tend to over-rely on a limited portion of temporal evidence, leading to temporally imbalanced evidence aggregation across the video. To address this issue, we investigate a decoder-side phenomenon in which the model exhibits a temporally imbalanced concentration pattern. We term the frame with the highest aggregated frame-level attention mass the anchor frame. We find that this bias is largely independent of the input video and instead appears to reflect a persistent, model-specific structural or positional bias, whose over-dominance is closely associated with hallucination-prone generation. Motivated by this insight, we propose Decoder-side Temporal Rebalancing (DTR), a training-free, layer-selective inference method that rebalances temporal evidence allocation in middle-to-late decoder layers without altering visual encoding or requiring auxiliary models. DTR adaptively calibrates decoder-side visual attention to alleviate temporally imbalanced concentration and encourage under-attended frames to contribute more effectively to response generation. In this way, DTR guides the decoder to ground its outputs in temporally broader and more balanced video evidence. Extensive experiments on hallucination and video understanding benchmarks show that DTR consistently improves hallucination robustness across diverse Video-LLM families, while preserving competitive video understanding performance and high inference efficiency.

Abstract

VLM Hallucination Mitigation

Explore similar work

Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Models

VADER: Adaptive Debiasing for Hallucination Mitigation in Video Large Language Models

Mitigating Video Hallucinations via Attention-Guided Representation Perturbation