Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models

Kairong Yu; Zixin Zhu; Le Yu; Hongwei Wang

cs.CVAug 28, 2026

Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models

Authors: Kairong Yu, Zixin Zhu, Le Yu, Hongwei Wang

Organizations: Zhejiang University · Southeast University

Abstract

Large Vision-Language Models (LVLMs) remain prone to hallucinations, producing responses that are irrelevant or inconsistent with the multimodal input. Existing mitigation methods mainly rely on external supervision, output calibration, or attention regulation, leaving the internal representation dynamics of autoregressive generation underexplored. We identify an inference-time failure mode in which cross-modal representations degrade across decoder layers and drift across generation steps, destabilizing token prediction and increasing hallucination risk. We propose Dynamic Alignment Compensation (DAC), a training-free inference-time method that detects representation divergence and selectively applies lightweight residual compensation. DAC combines Layer-wise Semantic Compensation to mitigate inter-layer degradation with Sequential Semantic Correction to constrain temporal drift. Experiments on nine hallucination-focused and general-purpose multimodal benchmarks across multiple LVLM backbones show that DAC consistently reduces hallucinations while maintaining strong overall performance.

Figures & tables

Figure 1: Performance comparison of different methods across multiple benchmarks. Hal-Bench denotes HallusionBench.

Figure 2: Layer-wise and temporal JS-divergence patterns in hallucinated and non-hallucinated LVLM generation. (a) Inter-layer JS divergence is projected onto image and text tokens at representative decoder layers. Hallucinated cases (bottom) show more frequent and stronger high-divergence regions than non-hallucinated cases (top), mainly in middle-to-late layers. (b) Across decoding steps, hallucinated object tokens exhibit larger temporal drift than non-hallucinated object tokens, with a similar middle-to-late-layer concentration.

Figure 3: Overview of Dynamic Alignment Compensation (DAC). LSC compensates hidden states with large inter-layer representation divergence, while SSC constrains temporal drift of the current token using a layer-wise cache. Green checks denote triggered updates; red crosses denote unchanged states.

Backbone	Method	MME			MM-Vet	MMBench	TextVQA	MMBench-CN
Backbone	Method	Perception $\uparrow$	Cognition $\uparrow$	Overall $\uparrow$	Score $\uparrow$	Accuracy $\uparrow$	Accuracy $\uparrow$	Accuracy $\uparrow$
LLaVA-1.5	Regular	1406.53	335.36	1741.89	31.10	62.80	54.18	57.56
	OPERA	1473.62	310.71	1784.34	32.00	62.80	–	–
	ICD	1341.63	295.36	1636.99	25.90	39.78	47.59	45.36
	VCD	1302.66	290.36	1593.02	30.20	54.21	45.71	51.46
	MemVR	1465.78	360.71	1826.49	32.40	63.75	54.22	56.62

Table 1: Performance evaluation on MME, MM-Vet, MMBench, TextVQA, and MMBench-CN benchmarks.

Method	Random		Popular		Adversarial		Average
	Accuracy $\uparrow$	F1-score $\uparrow$	Accuracy $\uparrow$	F1-score $\uparrow$	Accuracy $\uparrow$	F1-score $\uparrow$	Accuracy $\uparrow$	F1-score $\uparrow$
LLaVA-1.5	83.49	82.28	79.98	79.34	76.03	76.26	79.83	79.29
+OPERA	87.53	86.45	84.21	83.50	80.88	80.69	84.21	83.55
+ICD	84.87	83.27	82.93	81.45	81.07	79.96	82.96	81.56
+VCD	86.84	86.83	82.65	83.37	77.31	79.28	82.27	83.16
+MemVR	88.50	87.34	87.10	86.01	85.20	84.28	86.93	85.88

Table 2: Performance evaluation on the POPE benchmark.

Backbone	Method	MM-Vet (Score) $\uparrow$	MMBench (Acc.) $\uparrow$	TextVQA (Acc.) $\uparrow$	MMBench-CN (Acc.) $\uparrow$	LLaVA-Bench (Score) $\uparrow$
Qwen2.5-VL-3B	Regular	62.1	78.35	77.41	77.06	82.37
	+MemVR	60.2	77.83	77.45	75.51	83.42
	+ICD	20.9	75.94	77.09	73.71	78.29
	+VCD	57.4	75.17	72.76	74.57	80.38
	+DAC	63.6	78.92	77.82	77.41	86.60
Qwen2.5-VL-7B	Regular	67.5	83.50	77.86	81.10	89.85

Table 3: Generalization to recent LVLM backbones on general-purpose multimodal benchmarks.

Method	$C_{s}\downarrow$	$C_{i}\downarrow$	Recall $\uparrow$	Len.
LLaVA-1.5	47.6	13.3	80.6	99.7
+OPERA	47.6	13.5	79.0	93.2
+ICD	56.2	16.3	16.3	103.4
+VCD	55.0	15.8	77.4	101.2
+MemVR	46.6	13.0	80.8	99.6
+DAC	28.2	7.3	72.0	88.9

Table 4: Performance evaluation on CHAIR.

C_{s}

and

C_{i}

denote sentence-level and instance-level hallucination rates, respectively; Len. denotes average response length.

Backbone	Method	C. $\uparrow$	D. $\uparrow$	Cp. $\uparrow$	All $\uparrow$	Avg. $\uparrow$
LLaVA-1.5	Regular	51.2	50.3	70.0	56.6	57.03
	ICD	49.3	46.9	63.1	54.9	53.55
	VCD	50.0	47.2	62.0	54.7	53.48
	MemVR	53.4	50.7	71.1	57.4	60.53
	DAC	57.5	55.8	73.1	59.8	61.55
Qwen-VL-Chat	Regular	68.9	61.6	82.3	73.3	71.53

Table 5: Performance evaluation on LLaVA-Bench. C., D., Cp., and Avg. denote Conversation, Detail, Complex, and Average, respectively.

Backbone	Method	MME(Perception) $\uparrow$	MME(Cognition) $\uparrow$	MME(Overall) $\uparrow$	POPE(Acc.) $\uparrow$	POPE(F1) $\uparrow$
Qwen2.5-VL-3B	Regular	1518.15	621.78	2139.93	87.38	86.20
	+MemVR	1499.00	625.35	2124.35	87.51	86.36
	+ICD	1473.30	571.42	2044.72	87.31	86.08
	+VCD	1512.03	595.35	2107.38	86.91	85.80
	+DAC	1528.00	621.78	2149.78	87.67	86.43
Qwen2.5-VL-7B	Regular	1692.78	624.28	2317.06	87.67	86.54

Table 6: Comparison on MME and POPE across recent LVLM backbones.MME reports perception, cognition, and overall scores; POPE reports average accuracy and F1 over three splits. Best or tied-best results within each backbone block are bolded.

Method	fACC $\uparrow$	qACC $\uparrow$	easyA $\uparrow$	hardA $\uparrow$	aACC $\uparrow$
LLaVA-1.5	17.9	8.13	36.0	36.7	41.5
+OPERA	16.2	5.49	37.6	35.4	41.2
+ICD	13.9	8.35	36.9	33.5	38.2
+VCD	13.9	11.4	33.0	34.7	41.1
+MemVR	17.9	9.01	36.9	37.7	42.5
+DAC	14.2	17.05	44.4	37.3	45.5

Table 7: Performance evaluation on HallusionBench.

LSC	SSC	Rec.	OCR	Kno.	Gen.	Spa.	Math	Total
$\times$	$\times$	36.4	20.1	22.3	22.2	24.8	7.7	31.1
$\times$	$\checkmark$	37.2	23.5	23.3	22.6	28.9	7.7	33.0
$\checkmark$	$\times$	37.9	23.6	24.6	24.5	28.3	7.7	33.3
$\checkmark$	$\checkmark$	37.9	25.1	24.9	24.1	30.1	7.7	33.9

Table 8: Ablation results of LSC and SSC.

Figure 4: JS-divergence diagnostics on CHAIR with LLaVA-1.5. (a),(b) show the inter-layer high-JS ratio and mean JS across decoder layers, while (c),(d) show the corresponding temporal statistics across decoding steps. Mean-JS curves compare hallucinated and non-hallucinated object tokens. Gray regions highlight the middle-to-late layers where high-JS outliers concentrate, and shaded bands denote standard error.

Method	Latency (ms/token) $\downarrow$	Throughput (tokens/ms) $\uparrow$	T/80 Tokens (ms) $\downarrow$	Memory (MB) $\downarrow$
Greedy	36.8	0.027	2946.91	13797
Sampling	37.1	0.027	2972.99	13797
MemVR	36.8	0.027	2949.12	13800
ICD	79.0	0.013	6320.52	13778
OPERA	137.1	0.008	11501.57	21110
VCD	76.1	0.013	6085.18	13781

Table 9: Inference efficiency on MM-Vet with LLaVA-1.5. Best or tied-best values are bolded.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Method	HObj@100 μ $\downarrow$	HObj@100 M $\downarrow$
Regular	1.1206	1.0314
DAC	0.5797	0.6569

Appendix

Table 10: Length-normalized CHAIR analysis on LLaVA-1.5. HObj@100 μ measures hallucinated object mentions per 100 generated tokens at the corpus level, while HObj@100 M averages this rate over samples.

Figure 5: A case study comparing the levels of hallucination among various baselines

Figure 6: A case study comparing the levels of hallucination among various baselines

Figure 7: A case study comparing the levels of hallucination among various baselines

Figure 8: A case study comparing the levels of hallucination among various baselines

Figure 9: A case study comparing the levels of hallucination among various baselines

Figure 10: A case study comparing the levels of hallucination among various baselines

Figure 11: A case study comparing the levels of hallucination among various baselines

Figure 12: A case study comparing the levels of hallucination among various baselines

Figure 13: A case study comparing the levels of hallucination among various baselines

Figure 14: A case study comparing the levels of hallucination among various baselines

Figure 15: A case study comparing the levels of hallucination among various baselines

Figure 16: A case study comparing the levels of hallucination among various baselines

Figure 17: A case study comparing the levels of hallucination among various baselines

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models
May 18, 2025Kai Tang, Jinhao You, Yichen Guo +8Vision-Language Models Hallucination in Language Models
Peking University · University of Pennsylvania · Nanyang Technological University +4
Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models
Feb 28, 2026April FuCross-Layer Attention Self-Correction in VLMs
Independent Researcher
HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models
Apr 13, 2026Xinyun LiuVLM Hallucination Mitigation VLM Calibration
Sichuan University, Chengdu, China

May 18, 2025J/K move · Enter open · S save

Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models

Kai Tang, Jinhao You, Yichen Guo, Yiding Sun+7

Peking University · University of Pennsylvania · Nanyang Technological University+4

Vision-Language Models Hallucination in Language Models Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation VLM Inference

Open paper AskPDF arXiv

Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucinations, where generated content is inconsistent with the input image. Existing training-free hallucination mitigation methods often suffer from unstable performance and high sensitivity to hyperparameter settings, which limits their practicality and broader adoption. In this paper, we propose Decoding with Inter-layer Consistency via Layer Aggregation (DCLA), a training-free decoding mechanism that requires no retraining, fine-tuning, or access to external knowledge bases. Specifically, DCLA constructs a dynamic semantic reference by aggregating representations from previous layers and uses it to correct semantically deviated layers, thereby enforcing inter-layer consistency. Experiments across seven LVLMs and multiple benchmarks demonstrate the generality of DCLA: it surpasses standard decoding by 28.58 MME points on LLaVA1.5-7B and 42.6 MME points on Qwen2.5-VL, while improving POPE accuracy by 2.74 percentage points in the strongest setting.

Abstract

VLM Hallucination Mitigation

Explore similar work

Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models

Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models

HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models