HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models

Xinyun Liu

cs.CVApr 13, 2026

HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models

Authors: Xinyun Liu

Organizations: Sichuan University, Chengdu, China

Abstract

Large vision-language models (LVLMs) achieve strong multimodal performance, but still suffer from hallucinations caused by unstable visual grounding and over-reliance on language priors. Existing training-free decoding methods typically apply calibration at every decoding step, introducing unnecessary computation and potentially disrupting stable predictions. We address this problem by identifying layer-wise hesitation, a simple signal of grounding instability reflected by fluctuations in token preference across intermediate layers. Based on this observation, we propose Hesitation-Triggered Differential Calibration (HTDC), a training-free decoding framework that preserves standard full-branch inference and activates calibration only at hesitation-prone steps. When triggered, HTDC contrasts the full branch with two lightweight probes, a visual-nullification probe and a semantic-nullification probe, to suppress hallucination-prone candidates while avoiding unnecessary intervention on stable steps. Experiments on representative hallucination benchmarks show that HTDC consistently reduces hallucinations while maintaining strong task accuracy, achieving a favorable trade-off between effectiveness and computational overhead.

Figures & tables

Figure 1. Motivation of our method. The base LVLM hallucinates the scene context and incorrectly describes the toaster as being on a kitchen counter. Continuous calibration refines all tokens indiscriminately, leading to over-refinement without correcting the error. In contrast, our HTDC selectively calibrates only hallucination-prone tokens based on token-level hesitation scores, yielding a more accurate description.

Figure 2. Overview of HTDC . At each decoding step, the model first performs standard full-branch inference and estimates a hesitation weight $w_{t}$ from intermediate-layer dynamics. When $w_{t}>w_{\min}$ , two auxiliary probes are activated: a visual perturbation probe ( $V0$ ) and a semantic nullification probe ( $X0$ ). Their score differences with respect to the full branch are then used for differential calibration, yielding the final prediction. A hysteresis margin $\epsilon$ is further applied to avoid unstable switching.

\Block 2-1 Method	LLaVA-v1.5			Qwen3-VL
	CHAIRs ( $\downarrow$ )	CHAIRi ( $\downarrow$ )	Recall ( $\uparrow$ )	CHAIRs ( $\downarrow$ )	CHAIRi ( $\downarrow$ )	Recall ( $\uparrow$ )
Regular	18.2	6.2	61.9	21.6	7.8	46.8
VCD	12.2	14.3	26.5	9.2	3.0	43.4
ICD	22.4	8.5	55.1	6.0	3.0	29.1
DoLa	16.6	9.8	55.8	4.2	5.3	29.0
DAMO	17.4	6.1	63.1	3.4	1.8	39.6

Table 1. Experimental results on the CHAIR benchmark.

Model	Method	Perception(Total) $\uparrow$	Cognition (Total) $\uparrow$
LLaVA-v1.5	Regular	1508.38	357.86
	VCD	1477.74	357.86
	ICD	1441.19	333.93
	DoLa	1412.13	298.57
	DAMO	1512.01	297.86
	Ours	1515.89	367.50

Table 2. Experimental results on MME benchmark.

\Block 2-1Dataset	\Block 2-1Model	\Block 2-1Method	Random		Popular		Adversarial
			Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$
\Block [fill=white]12-1MSCOCO	\Block [fill=white]6-1LLaVA-v1.5	Regular	0.8804	0.8718	0.8856	0.8756	0.8507	0.8413
		VCD	0.8818	0.8734	0.8727	0.8614	0.8513	0.8418
		ICD	0.8433	0.8330	0.8607	0.8487	0.8433	0.8330
		DoLa	0.8821	0.8737	0.8790	0.8736	0.8403	0.8396
		DAMO	0.8687	0.8564	0.8603	0.8446	0.8433	0.8290

Table 3. Experimental results on POPE benchmark.

Method	Intervention	Trigger Rate	$N_{fwd}$	Latency (ms)	Overhead
Regular	None	0.0%	1.000	60.77	-
VCD	Logit (Continuous)	100.0%	2.000	168.80	+177.8%
ICD	Logit (Continuous)	100.0%	2.000	169.75	+179.3%
Ours	Logit (Dynamic)	5.3%	1.107	71.08	+17.0%

Table 4. Computational overhead comparison on the MME benchmark

Model	Method	rec ( $\uparrow$ )	ocr ( $\uparrow$ )	know ( $\uparrow$ )	gen ( $\uparrow$ )	spat ( $\uparrow$ )	math ( $\uparrow$ )	Total ( $\uparrow$ )
\Block [fill=white]6-1LLaVA-v1.5	Regular	34.4	23.0	16.1	20.5	25.2	11.5	29.9
	VCD	7.5	8.9	10.1	14.5	6.3	5.8	7.1
	ICD	27.0	20.1	15.9	20.6	24.6	7.8	29.8
	DoLa	33.2	22.6	17.3	13.0	28.1	18.5	30.7
	DAMO	33.5	20.4	18.0	19.7	23.7	7.7	29.3
	Ours	35.1	23.2	17.4	21.0	25.5	11.5	30.7

Table 5. Experimental results on MM-Vet benchmark. We report the performance across different capability dimensions: recognition (rec), OCR (ocr), knowledge (know), generation (gen), spatial (spat), and math (math).

Figure 3. Qualitative comparison on complex reasoning (MM-Vet). VCD’s continuous intervention destroys the initial OCR-based reasoning chain, whereas HTDC preserves stable generation and accurate reasoning capabilities.

Figure 4. Impact of key hyperparameters on the CHAIR benchmark, including (a) dynamic vs. static gating strategy, (b) visual noise scale ( $\sigma$ ), and (c) layer selection depth ( $k$ ). Line and bar charts demonstrating that dynamic gating recovers recall, noise scale at 0.8 optimizes CHAIRi, and layer depth adjustments trade off between recall and hallucination suppression.

\Block 2-1Method	CHAIR			POPE
	C S $\downarrow$	C I $\downarrow$	Recall $\uparrow$	Acc $\uparrow$	F1 $\uparrow$
Ours (Full HTDC)	11.6	4.7	58.2	86.94	86.91
w/o visual branch	11.6	4.7	58.3	87.26	87.11
w/o semantic branch	18.2	6.6	62.2	86.12	85.45
w/o both (Baseline)	18.2	6.2	61.9	86.86	85.89

Table 6. Ablation studies on CHAIR and POPE.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models
May 18, 2025Kai Tang, Jinhao You, Yichen Guo +8Vision-Language Models Hallucination in Language Models
Peking University · University of Pennsylvania · Nanyang Technological University +4
CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMs
May 22, 2026Xiaoyi Huang, Kejia Zhang, Zhiming LuoHallucination Detection in VLMs Efficient VLM Inference
Institute of Artificial Intelligence, Xiamen University · Department of Artificial Intelligence, Xiamen University
Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models
Apr 28, 2026Chengsheng Zhang, Chenghao Sun, Xinyan Jiang +2Vision-Language Models Hallucination in Language Models
University of Science and Technology of China · Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China · University of Chinese Academy of Sciences, Beijing, China

May 18, 2025J/K move · Enter open · S save

Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models

Kai Tang, Jinhao You, Yichen Guo, Yiding Sun+7

Peking University · University of Pennsylvania · Nanyang Technological University+4

Vision-Language Models Hallucination in Language Models Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation VLM Inference

Open paper AskPDF arXiv

Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucinations, where generated content is inconsistent with the input image. Existing training-free hallucination mitigation methods often suffer from unstable performance and high sensitivity to hyperparameter settings, which limits their practicality and broader adoption. In this paper, we propose Decoding with Inter-layer Consistency via Layer Aggregation (DCLA), a training-free decoding mechanism that requires no retraining, fine-tuning, or access to external knowledge bases. Specifically, DCLA constructs a dynamic semantic reference by aggregating representations from previous layers and uses it to correct semantically deviated layers, thereby enforcing inter-layer consistency. Experiments across seven LVLMs and multiple benchmarks demonstrate the generality of DCLA: it surpasses standard decoding by 28.58 MME points on LLaVA1.5-7B and 42.6 MME points on Qwen2.5-VL, while improving POPE accuracy by 2.74 percentage points in the strongest setting.

Abstract

VLM Hallucination Mitigation

Explore similar work

Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models

CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMs

Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models