ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models

Jihae Jeong; Junha Choi; Hwanjo Yu

cs.CVAug 19, 2026

ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models

Authors: Jihae Jeong, Junha Choi, Hwanjo Yu

Organizations: Pohang University of Science and Technology (POSTECH)

Abstract

Large vision-language models (LVLMs) often hallucinate, generating content that the input image does not support. Preventing such content during decoding calls for a candidate-specific measure of how strongly the image supports the token under consideration. The model's visual-token states offer a natural source of this evidence because projecting each state through the output head reveals which vocabulary items that position favors. These position-wise readouts cannot be pooled directly because their probability magnitudes are not comparable across visual positions. Vocabulary ranks provide a scale-invariant basis for pooling, but tokens still differ systematically in their typical rank-based evidence. We propose ReWEIGH, a training-free decoding intervention that aggregates these ranks across visual positions and compares each candidate with a token-specific reference estimated from unlabeled images. At inference, ReWEIGH caches the image evidence during prefill and applies a bounded penalty only to candidates that fall below their reference. On four 7B backbones, ReWEIGH reduces hallucinated object mentions by up to 21.3% while largely preserving or improving descriptive and general performance. With evidence cached, the average added latency is 1.33% per token, and the reductions extend across six architecture families to 32B parameters.

Figures & tables

Figure 1: Across-position spread of the rank-10 probability before and after scale equalization. Error bars are retrospective image-bootstrap 95% intervals, and equalized bars are point estimates.

Figure 2: Decomposition of DMRR variability on LLaVA-1.5-7B. Each bar sums to 100%.

Figure 3: Out-of-sample calibration error of token-specific, global, and shuffled references. Errors are in

b_{0}

units with every token counted once. Shading spans the full shuffle range, and the tick marks its median.

Figure 4: Overview of ReWEIGH. Measure reads visual-token representations as vocabulary rankings. Register constructs a reliable token-specific reference table from unlabeled calibration images. Intervene applies a bounded negative shift to candidates with an evidence deficit. Measure is shared by offline calibration and online inference, whereas registration runs only once.

	CHAIR			AMBER							Cost
Method	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	F1 $\uparrow$	CHAIR $\downarrow$	Cover $\uparrow$	Hal. $\downarrow$	Cog. $\downarrow$	Acc. $\uparrow$	F1 $\uparrow$	Score $\uparrow$	Lat. $\downarrow$	Mem. $\downarrow$
LLaVA-1.5-7B	52.60	15.61	80.66	3.69	50.47	19.02	3.82	71.96	74.81	85.56	1.00 $\times$	1.00 $\times$
VCD	59.00	17.97	78.50	6.25	51.81	32.07	4.49	66.81	70.37	82.06	2.01 $\times$	1.06 $\times$
OPERA	51.20	14.19	80.96	3.70	48.79	17.03	2.70	74.94	78.09	87.20	7.26 $\times$	1.51 $\times$
DoLa	53.00	15.87	79.28	4.52	51.03	21.12	3.84	72.17	75.09	85.28	1.10 $\times$	1.00 $\times$
PAI	42.40	13.51	75.55	4.68	43.11	19.32	1.97	38.29	18.93	57.12	2.14 $\times$	1.02 $\times$

Table 1: Main results on CHAIR and AMBER with decoding cost. The AMBER columns include generative and discriminative metrics, followed by the benchmark’s combined score. Lat. and Mem. are per-token latency and peak allocated memory relative to each backbone’s greedy decoding under the cached-evidence protocol. Green and red shade improvements and degradations of more than 5% against each backbone’s greedy baseline, darker shades mark more than 20%, and unshaded cells differ by less than 5%. Bold and underlined values indicate the best and second-best result, respectively, within each backbone block. Arrows indicate the preferred direction.

Backbone	Metric	Base	VCD	OPERA	DoLa	PAI	ReVisiT	ReWEIGH
LLaVA-1.5-7B	MMHal Score $\uparrow$	2.76	2.16	2.57	2.65	2.28	2.33	2.78
	MMHal Hall. Rate (%) $\downarrow$	51.04	65.62	53.12	53.12	43.75	58.33	48.96
	MM-Vet Acc. $\uparrow$	35.41	32.84	36.74	36.06	18.26	33.58	37.20
Qwen2.5-VL-7B	MMHal Score $\uparrow$	4.28	4.11	4.50	3.69	3.27	3.78	4.42
	MMHal Hall. Rate (%) $\downarrow$	15.62	17.71	17.71	20.83	26.04	19.79	15.62
	MM-Vet Acc. $\uparrow$	56.65	60.87	70.96	46.97	44.86	56.38	59.54

Table 2: MMHal-Bench and MM-Vet results. Hall. Rate is the percentage of examples with score below 3, and MM-Vet accuracy is on a 0–100 scale.

Configuration	CHAIR			AMBER
Configuration	$C_{S}\,\downarrow$	$C_{I}\,\downarrow$	F1 $\uparrow$	CHAIR $\downarrow$	Cover $\uparrow$	Score $\uparrow$
[3pt][3pt] Full method	44.8	12.67	80.85	2.98	50.21	85.91
Global reference	58.4	17.45	80.31	4.10	52.78	85.36
Shuffled references	56.0	14.74	81.54	4.11	51.63	85.36
Mismatched evidence	50.2	14.45	79.73	3.70	49.35	85.56
No abstention	44.4	12.64	80.90	2.97	50.37	86.62

Table 3: Component ablations on LLaVA-1.5-7B.

C_{S}

and

C_{I}

denote CHAIR S and CHAIR I , and arrows indicate the preferred direction. Arm definitions are in Table 19 .

Figure 5: Calibration-size sweep on LLaVA-1.5-7B. Dashed lines mark base decoding, and the open circle marks the deployed 500-image table.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Checkpoint ID	Language backbone
InstructBLIP-7B	Salesforce/LAVIS blip2_vicuna_instruct/vicuna7b	Vicuna-7B
LLaVA-1.5-7B	liuhaotian/llava-v1.5-7b	Vicuna-7B-v1.5
LLaVA-NeXT-7B	liuhaotian/llava-v1.6-mistral-7b	Mistral-7B-Instruct-v0.2
Qwen2.5-VL-7B	Qwen/Qwen2.5-VL-7B-Instruct	Qwen2.5-7B
InternVL2.5-8B	OpenGVLab/InternVL2_5-8B	internlm/internlm2_5-7b-chat
Gemma-3-12B	google/gemma-3-12b-it	Gemma-3-12B

Appendix

Table 4: Model checkpoints and their language backbones.

Backbone	$\ell$	$\beta$	$b_{0}$	Registered
LLaVA-1.5	29	1.1	0.00324809	2,580
LLaVA-NeXT	30	1.2	0.00274012	3,734
InstructBLIP	29	0.5	0.00184383	2,708
Qwen2.5-VL	27	1.1	0.000653155	7,028

Appendix

Table 5: Frozen backbone-specific operating points and reference-stability table statistics.

Benchmark	Split	Records	Max.	Reported metrics
CHAIR	MS COCO val500	500	512	$C_{S}$ , $C_{I}$ , object recall, F1, length
AMBER–Generative	official generative	1,004	128	CHAIR, Cover, Hal., Cog.
AMBER–Discriminative	official discriminative	14,216	8	accuracy, precision, recall, F1,
AMBER–Discriminative	official discriminative	14,216	8	unparsed, AMBER score
MMHal-Bench	full set	96	256	quality score, hallucination rate
MM-Vet	original test	218	1,024	total score

Appendix

Table 6: Evaluation datasets, record counts, maximum numbers of newly generated tokens (Max.), and reported metrics.

Threshold $\tau$	LLaVA-1.5-7B	InstructBLIP-7B	LLaVA-NeXT-7B	Qwen2.5-VL-7B
$10^{-3}$	8.6 (8.5)	4.1 (3.9)	9.2 (9.2)	5.8 (5.7)
$10^{-4}$	62.5 (59.4)	7.1 (6.7)	87.7 (84.0)	7.4 (7.4)
$10^{-5}$	93.8 (89.4)	4.6 (4.6)	477.3 (459.4)	11.5 (10.9)

Appendix

Table 7: Across-position dispersion of the rank depth of a fixed probability threshold. Each cell is the image-median p90/p10 ratio of

N_{j}(\tau)

, the number of vocabulary items with probability at least

\tau

at visual position

j

. Values in parentheses use the 200-image measurement.

	LLaVA-1.5-7B			Qwen2.5-VL-7B
Rank depth $k$	Original	Equalized	Surv.	Original	Equalized	Surv.
1	4.64	3.86	83%	4.11	3.53	86%
2	3.59	2.91	81%	3.29	2.70	82%
5	2.55	2.15	84%	2.47	1.87	76%
10	2.63	2.40	91%	2.05	1.45	70%
20	—	—	—	1.79	1.20	67%

Appendix

Table 8: Across-position dispersion of the rank-

k

log probability before and after scale equalization. Each cell is the image-median p90–p10 spread in log units. Surv. is the share of the original spread that survives equalization. Depths absent from a backbone’s stored grid are marked —.

Signal	Quantity	Value	95% CI	Value	95% CI
		LLaVA-1.5-7B		Qwen2.5-VL-7B
Rank	Suppression-strength change	$0^{\dagger}$	—	$0^{\dagger}$	—
Probability	Suppression-strength mean $\|\Delta\|$	0.0445	$[0.0430,\,0.0460]$	0.0462	$[0.0447,\,0.0478]$
	Candidate-score Spearman	0.964	—	0.961	—
	Edited-set Jaccard	0.805	—	0.857	—
	Strongest edited candidate changed	18.6%	—	17.8%	—

Appendix

Table 9: Order-preserving rescaling stress test at the selected readout layers. The probability rows report how the probability-derived edit field reacts to the rescaling, and the winner is the candidate-local one-step argmax after the edit. The equalization row reports the closed share of the probability-to-rank gap. Intervals are image-bootstrap 95%, retrospective for the rescaling rows and paired for the equalization row.

\dagger

marks an exact zero. The shaded row is the rank signal used by ReWEIGH.

		LLaVA-1.5-7B				Qwen2.5-VL-7B
Support	Readout policy	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	Recall $\uparrow$	F1 $\uparrow$	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	Recall $\uparrow$	F1 $\uparrow$
Matched	Ordinal readout (DMRR)	45.8	12.78	75.2	80.77	26.0	8.64	58.0	70.99
	Standardized ( $z$ -score)	52.8	14.87	76.9	80.83	24.4	6.45	55.6	69.77
Native	Base decoding	52.6	16.07	77.3	80.48	28.8	8.26	56.6	69.97
	Ordinal readout (DMRR)	45.8	12.77	75.2	80.77	25.8	8.59	57.8	70.82
	Standardized ( $z$ -score)	52.4	14.90	76.9	80.81	24.0	6.38	55.0	69.29

Appendix

Table 10: End-to-end control between the ordinal and standardized readouts. Matched rows evaluate both policies on identical candidate support. Native rows let each policy form its own candidate sets, which is why the two blocks differ slightly. The native probability rows are not the matched-budget rank–probability comparison of Table 13 . Shaded rows are the ordinal readout used by ReWEIGH. Arrows indicate the preferred direction.

(a) Weighting sensitivity of the decomposition (LLaVA-1.5-7B)
Analysis unit	Between tokens	Between images, within token
Token–image balanced, min. 1 image/token	66.40%	33.60%
Token–image balanced, min. 2 images/token	66.33%	33.67%
Token–image balanced, min. 5 images/token (primary)	66.36%	33.64%
Token–image balanced, min. 10 images/token	66.59%	33.41%
Occurrence weighted	74.62%	25.38%

Appendix

Table 11: Token-identity component of DMRR variability. Panel (a) varies the rare-token filter and the weighting unit of the decomposition in Figure 2 . Panel (b) reports the intraclass correlation of token identity on image-unique observations at the selected readout layers.

Backbone	Aggregation	Global	Shuffled	Token-specific	Reduction
		Out-of-sample error ( $b_{0}$ units)
LLaVA-1.5-7B	token-unweighted median	0.849	1.010	0.067	92.1%
	candidate-mass-weighted mean	4.211	5.195	0.341	91.9%
Qwen2.5-VL-7B	token-unweighted median	0.831	0.510	0.077	90.7%
	candidate-mass-weighted mean	7.506	8.501	0.612	91.9%

Appendix

Table 12: Out-of-sample calibration error of global, shuffled, and token-specific references. The shuffled column is the median of 1,000 shuffles, and Reduction is the error decrease of the correct reference relative to the global reference. The shaded column is the reference used by ReWEIGH.

Signal	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	Removed $\uparrow$	Introduced $\downarrow$	Net $\uparrow$
Rank	44.8	12.67	329	198	131
Probability	50.0	14.03	283	227	56

Appendix

Table 13: Matched-budget end-to-end comparison of the rank and probability signals on LLaVA-1.5-7B. Removed and Introduced count hallucinated-object occurrences relative to the same base captions, and Net is their difference. The shaded row is the rank signal used by ReWEIGH. Arrows indicate the preferred direction.

Figure 6: Matched-budget rank and probability interventions on one CHAIR image. The visible utensil is a pizza server rather than a fork. Red marks scorer-flagged object mentions, and blue marks scorer-matched objects.

		LLaVA-1.5-7B		Qwen2.5-VL-7B
Group	Quantity	Value	95% CI	Value	95% CI
Effect	Rank-magnitude donor	$+2.32$ pp	$[+1.09,\,+3.53]$	$+0.55$ pp	$[-1.00,\,+2.12]$
	Probability-magnitude donor	$+2.06$ pp	$[+0.95,\,+3.14]$	$+1.37$ pp	$[-0.15,\,+2.91]$
Role	Structural tokens	$+2.36$ pp	$[+1.28,\,+3.48]$	$+1.03$ pp	$[-0.36,\,+2.46]$
	Lexical content tokens	$+0.33$ pp	$[-0.39,\,+1.04]$	$+0.51$ pp	$[-0.65,\,+1.72]$
	Punctuation tokens	$-0.74$ pp	$[-1.56,\,+0.09]$	$+0.72$ pp	$[-1.19,\,+2.62]$

Appendix

Table 14: Candidate-allocation analysis of the calibrated rank and probability edit fields. Effect and Role values are rank-minus-probability selectivity differences in percentage points with image-bootstrap 95% intervals. Share values are fractions of allocation-exclusive events that fall on structural tokens. Role and Share use the rank-magnitude donor. The definitions are in the accompanying text.

Quantity	LLaVA-1.5-7B	Qwen2.5-VL-7B
Hallucinated onsets in the top-confidence quartile	13.9% $[10.3,\,17.7]$	7.3% $[4.4,\,10.7]$
(count)	52 / 373	20 / 274
Median evidence deficit at those onsets	1.18	0.10
Median evidence deficit, grounded onsets in the same quartile	0.00	0.00
Median difference $[95\%$ CI $]$	1.18 $[0.47,\,5.56]$	0.10 $[0.00,\,7.04]$

Appendix

Table 15: Object-mention onsets in the top quartile of output confidence. Brackets are image-bootstrap 95% intervals, and the evidence deficit is

\max(b(v)-\operatorname{DMRR}_{I}(v),0)/b_{0}

before clipping.

Group	Quantity	Greedy	ReWEIGH	Overhead
Latency	Cached evidence (mean ratio)	$1.000000\times$	$1.013323\times$	$+1.33\%$
	Online DMRR computation (mean ratio)	$1.000000\times$	$1.023974\times$	$+2.40\%$
Memory	Peak allocated memory (MiB)	14,353	14,397	$+0.31\%$
Footprint	Disk, 500 images (MiB)	—	183.11	—
	Per image (KiB)	—	375.0	—

Appendix

Table 16: Inference-cost accounting. Latency ratios average measurements from three backbones, with 100 prompts per backbone. Memory and cache-footprint measurements use LLaVA-1.5-7B. Overhead is relative to greedy decoding, and the shaded column is ReWEIGH.

Backbone	Metric	Base	VCD	OPERA	DoLa	PAI	ReVisiT	ReWEIGH
LLaVA-1.5-7B	CHAIRs $\downarrow$	52.60	59.00	51.20	53.00	42.40	51.40	44.80
	CHAIRi $\downarrow$	15.61	17.97	14.19	15.87	13.51	15.44	12.67
	Recall $\uparrow$	77.25	75.26	76.63	74.95	67.06	76.94	75.26
	F1 $\uparrow$	80.66	78.50	80.96	79.28	75.55	80.57	80.85
Qwen2.5-VL-7B	CHAIRs $\downarrow$	31.60	33.80	20.20	23.60	26.20	24.00	25.40
	CHAIRi $\downarrow$	9.58	9.65	7.01	16.76	9.61	6.75	7.54

Appendix

Table 17: Full CHAIR results. Bold and underlined values are the best and second-best result within each backbone block, and the shaded column is ReWEIGH. Arrows indicate the preferred direction.

	Gen				Disc					Score
Method	CHAIR $\downarrow$	Cover $\uparrow$	Hal. $\downarrow$	Cog. $\downarrow$	Acc. $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Unp.% $\downarrow$	AMBER $\uparrow$
LLaVA-1.5-7B	3.69	50.47	19.02	3.82	71.96	92.65	62.73	74.81	0.08	85.56
VCD	6.25	51.81	32.07	4.49	66.81	86.22	59.45	70.37	0.67	82.06
OPERA	3.70	48.79	17.03	2.70	74.94	92.89	67.36	78.09	0.01	87.20
DoLa	4.52	51.03	21.12	3.84	72.17	92.35	63.27	75.09	0.08	85.28
PAI	4.68	43.11	19.32	1.97	38.29	73.46	10.86	18.93	16.66	57.12

Appendix

Table 18: Full AMBER results. Gen and Disc denote the generative and discriminative evaluations, respectively. Bold and underlined values are the best and second-best result within each backbone block, and shaded rows are ReWEIGH. Arrows indicate the preferred direction.

Figure 7: Layer-specific sensitivity to the intervention strength

\beta

on the CHAIR validation set. Circles mark the selected operating point (

\ell=29

\beta=1.1

Arm	Modification
Global reference	Replace the token-specific reference $b(v)$ with the single pooled scale $b_{0}$ .
Shuffled references	Randomly permute $b(v)$ across token identities while preserving the multiset of reference values.
Mismatched evidence	Rotate the image-conditioned evidence $\operatorname{DMRR}_{I}(v)$ across examples with a fixed derangement, so no example retains its own evidence.
No abstention	Apply the fallback $b_{0}$ to candidates absent from the registered set $\mathcal{R}$ instead of leaving them unchanged.
Bidirectional update	Remove the suppression-only rectifier (the lower clipping bound), permitting evidence to either decrease or increase a candidate logit.
Unbounded update	Remove the upper clipping bound on the intervention magnitude.

Appendix

Table 19: Definitions of the component ablations.

Configuration	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	Recall $\uparrow$	F1 $\uparrow$	Length	Fire (%)
Full method	44.8	12.67	75.3	80.85	90.3	80.4
Global reference	58.4	17.45	78.2	80.31	90.3	67.0
Shuffled references	56.0	14.74	78.1	81.54	92.9	72.6
Mismatched evidence	50.2	14.45	74.6	79.73	90.2	87.8
No abstention	44.4	12.64	75.3	80.90	90.2	82.7
Bidirectional update	50.4	16.07	75.9	79.74	92.0	98.7

Appendix

Table 20: Detailed CHAIR ablation results. Fire is the fraction of candidate positions at which the intervention is active, and Escapes counts updates that would exceed the full method’s clipping bound. The shaded row is the full method.

Evidence supplied	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	Recall $\uparrow$	F1 $\uparrow$
None (base decoding)	52.6	15.61	77.3	80.66
Own image (ReWEIGH)	44.8	12.67	75.3	80.85
Wrong image, fixed derangement	50.2	14.45	74.6	79.73
Wrong image, cyclic shift	50.4	15.01	75.0	79.66
Image-blind ( $\operatorname{DMRR}_{I}\equiv 0$ )	49.8	13.98	74.1	79.61

Appendix

Table 21: Image-specificity study on LLaVA-1.5-7B. Each row replaces the cached evidence vector with the stated substitute. The fixed-derangement row is the mismatched-evidence arm of Table 3 . Arrows indicate the preferred direction, and the shaded row is the unmodified method.

	Gen				Disc					Score
Configuration	CHAIR $\downarrow$	Cover $\uparrow$	Hal. $\downarrow$	Cog. $\downarrow$	Acc. $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Unp.% $\downarrow$	AMBER $\uparrow$
Full method	2.98	50.21	16.43	3.13	71.96	92.65	62.73	74.81	0.08	85.91
Global reference	4.10	52.78	21.91	3.51	71.98	92.65	62.75	74.82	0.05	85.36
Shuffled references	4.11	51.63	21.51	4.14	71.98	92.65	62.75	74.82	0.05	85.36
Mismatched evidence	3.70	49.35	18.03	3.15	71.96	92.65	62.73	74.81	0.08	85.56
No abstention	2.97	50.37	16.14	3.01	72.82	91.17	65.46	76.20	0.19	86.62

Appendix

Table 22: Detailed AMBER ablation results. Gen and Disc denote the generative and discriminative evaluations, respectively. The shaded row is the full method. Arrows indicate the preferred direction.

$\beta$	$C_{S}\downarrow$	$C_{I}\downarrow$	F1 $\uparrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	F1 $\uparrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	F1 $\uparrow$
	$\ell=28$			$\ell=29$			$\ell=30$
0.0	52.6	15.61	80.66	52.6	15.61	80.66	52.6	15.61	80.66
0.1	53.2	15.37	80.81	53.0	15.11	80.99	52.8	15.18	81.20
0.2	52.0	15.26	80.72	52.2	14.76	80.95	51.6	15.14	81.02
0.3	49.6	15.10	80.55	48.6	14.23	81.08	49.8	14.27	81.27
0.4	48.8	14.51	80.12	47.6	13.95	80.72	49.2	14.32	80.87

Appendix

Table 23: Full layer

\times

\beta

grid on the CHAIR validation split (LLaVA-1.5-7B).

C_{S}

and

C_{I}

denote CHAIR S and CHAIR I , the

\beta{=}0

rows equal base decoding, and the shaded cell is the frozen operating point.

Backbone	Registration rule	$\|\mathcal{R}\|$	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	Recall $\uparrow$	F1 $\uparrow$	AMBER-disc F1 $\uparrow$
LLaVA-1.5-7B	none (base)	–	52.6	15.61	77.3	80.66	74.81
	count $n(v)\geq 5$	3,252	44.6	12.69	75.4	80.91	74.81
	count $n(v)\geq 10$	2,476	44.8	12.70	75.4	80.91	74.81
	reference stability ( $\Delta e<0.5$ )	2,580	44.8	12.67	75.3	80.85	74.81
InstructBLIP-7B	none (base)	–	49.6	14.03	73.1	79.01	81.40
	count $n(v)\geq 5$	3,524	46.2	12.50	73.3	79.80	78.51

Appendix

Table 24: Design-space contrast of registration rules.

|\mathcal{R}|

is the number of registered tokens. Arrows indicate the preferred direction, and the shaded rows are the reference-stability rule used by ReWEIGH.

Calibration images	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	Recall $\uparrow$	F1 $\uparrow$
None (base decoding)	52.6	15.61	77.3	80.66
10	48.6	14.29	74.64	79.80
25	48.0	14.04	75.14	80.19
50	48.2	13.99	76.01	80.70
100	44.8	13.29	75.76	80.87
200	45.2	12.82	75.14	80.71

Appendix

Table 25: Calibration-size sweep on LLaVA-1.5-7B. Arrows indicate the preferred direction, and the shaded row is the deployed 500-image configuration.

Calibration corpus	Images	$\|\mathcal{R}\|$	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	Recall $\uparrow$	F1 $\uparrow$
None (base decoding)	–	–	52.6	15.61	77.3	80.66
MS COCO (default)	500	2,580	44.8	12.67	75.3	80.85
GQA	500	2,429	44.6	13.15	75.4	80.71
AMBER	1,004	2,982	48.0	12.97	75.6	80.93
Pooled (COCO+GQA+AMBER)	2,004	3,900	46.6	13.22	75.9	81.00

Appendix

Table 26: Calibration-corpus transfer on LLaVA-1.5-7B.

|\mathcal{R}|

is the number of registered tokens. Arrows indicate the preferred direction, and the shaded row is the default configuration.

Model	Method	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	Recall $\uparrow$	F1 $\uparrow$
InstructBLIP-7B	Greedy	49.60	14.03	73.09	79.01
	ReWEIGH	46.00	12.50	73.30	79.80
LLaVA-1.5-7B	Greedy	52.60	15.61	77.25	80.66
	ReWEIGH	44.80	12.67	75.30	80.85
LLaVA-NeXT-7B	Greedy	34.00	8.67	64.82	75.83
	ReWEIGH	32.60	7.78	63.70	75.36

Appendix

Table 27: CHAIR results across model sizes and architecture families. Shaded cells mark ReWEIGH.

Candidate	$z_{t}(v)$	$\operatorname{DMRR}_{I}(v)$	$b(v)$	Deficit	$s_{I}(v)$	$\beta\,s_{I}(v)$	$z^{\prime}_{t}(v)$
		Evidence ( $\times 10^{-3}$ )
cars	17.000	0.758	8.321	7.563	1.000	1.100	15.900
people	16.359	2.225	6.128	3.903	1.000	1.100	15.259
traffic	16.344	40.513	6.984	0.000	0.000	0.000	16.344
p	15.891	7.390	23.583	16.192	1.000	1.100	14.791
pal	15.703	72.867	124.762	51.895	1.000	1.100	14.603

Appendix

Table 28: Per-candidate arithmetic of ReWEIGH at a single recorded decoding step. The step uses the deployed LLaVA-1.5-7B configuration of Table 5 . The table lists the eight candidates with the highest pre-edit logits among the

K{=}22

candidates, including the pre-edit argmax (bold) and the emitted token (highlighted). The columns reproduce the arithmetic of Eqs. ( 5 ), ( 7 ), and ( 8 ) from the recorded trace.

Panel of Figure 8 — Figure 8: Mechanism case studies read from recorded traces. Each panel plots decoder logits before (filled) and after (open) the edit at one recorded decoding step, and each arrow spans the bounded suppression $\beta s_{I}(v)$ . The annotation under each candidate compares its evidence $\operatorname{DMRR}_{I}(v)$ with its reference $b(v)$ in units of $10^{-3}$ . Panel (a) shows LLaVA-1.5-7B at step 71 with four candidates from Table 28 that span full, proportional, and zero suppression. Panel (b) shows LLaVA-NeXT-7B ( $\beta{=}1.2$ , layer-30 readout) at step 60. Red marks the hallucinated pre-edit argmax, and blue marks the grounded token that becomes the post-edit argmax.

Panel of Figure 9 — Figure 9: Qualitative CHAIR examples of three common hallucination patterns. Panels (a), (b), and (c) show contextual scene completion, repeated object generation, and peripheral errors, respectively. Red marks CHAIR-hallucinated objects, and blue marks ground-truth objects.

Figure 10: Generative example from AMBER. Red marks terms from the annotated hallucination vocabulary, and blue marks annotated ground-truth objects.

Panel of Figure 11 — Figure 11: Failure modes of ablated components on LLaVA-1.5-7B. Arm definitions are in Table 19 . In (a), the arm caption is abridged, and the omitted middle, marked […], repeats only the two tokens “bus” and “car”. Red in (a) marks the repeated tokens in the shown tail. In (b), red marks CHAIR-hallucinated objects, and blue marks ground-truth objects.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Jul 5, 2026

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual understanding tasks such as image captioning and visual question answering. However, they remain susceptible to hallucinations, generating content that is inconsistent with the actual visual input. Existing methods primarily intervene at the decoding stage, while overlooking a critical source of hallucinations: irrelevant or noisy visual tokens that mislead the decoding process. To address this issue, we propose SeeMe, a training-free framework that introduces the concept of feature engineering from traditional machine learning into LVLMs. SeeMe restructures visual tokens through a three-stage token engineering process to suppress hallucination sources while preserving informative visual evidence. Experiments on MME, POPE, and AMBER benchmarks across four LVLMs demonstrate that SeeMe consistently reduces hallucinations and improves output consistency, providing a novel perspective for mitigating hallucinations in LVLMs.

Kai Tang, Jinhao You, Bohua Zhang +6

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Pennsylvania +4

Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation

Ask PDF

May 20, 2026

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD). Specifically, we obtain the attention weights from early generated tokens to visual tokens across layers and identify the tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain attention scores of generated text to visual evidence, in order to select and emphasize text tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple benchmark evaluations conducted on five recently released models show that our method can consistently mitigate hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.

Yutong Xie, Zhenglin Hua, Ran Wang +3

School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3

Vision-Language Models Hallucination in Language Models VLM Hallucination Large Vision-Language Models VLM Hallucination Mitigation

Ask PDF

Jun 2, 2026

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

Large vision language models (LVLMs) have made rapid advancements and are deployed across various applications, yet hallucinations remain a major challenge. Activation steering is appealing due to its minimal training overhead and controllability at inference time. However, we found that during autoregressive decoding, visual conditioning affects token prediction sparsely and locally across decoding steps, and many existing methods that average image-versus-no-image differences over the entire sequence dilute these critical signals, yielding low signal-to-noise ratio steering directions. Additionally, many existing methods apply a fixed steering strength, which misallocates the intervention budget, over-perturbs non-critical tokens, and can cause instability. To address these limitations, we propose Token-Level Visual-Sensitivity Steering (TLVS) for hallucination mitigation. Our approach first extracts token-level steering vectors and refines them, and then applies fine-grained, visual-sensitivity-adaptive steering only where it matters. This lightweight, plug-and-play mechanism requires only minimal training for calibration and can be applied across diverse vision-language models. It modulates the steering strength at each decoding step, selectively suppressing hallucination-prone spans while preserving evidence-grounded content. We evaluate TLVS on several benchmarks, including POPE, AMBER, CHAIR (COCO), MMHal, and HallusionBench, demonstrating consistent improvements over previous steering methods.

Ruipeng Zhang, Zhihao Li, C. L. Philip Chen +1

Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China · Pazhou Lab, Guangzhou, China.

VLM Hallucination Large Vision-Language Models Activation Steering VLM Hallucination Mitigation

Ask PDF

Abstract

VLM Hallucination Mitigation

Explore similar work

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation