HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

Reihaneh Zohrabi; Hosein Hasani; Akshita Gupta; Mahdieh Soleymani Baghshah; Anna Rohrbach; Marcus Rohrbach

cs.CVApr 7, 2026

HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

Authors: Reihaneh Zohrabi, Hosein Hasani, Akshita Gupta, Mahdieh Soleymani Baghshah, Anna Rohrbach, Marcus Rohrbach

Organizations: Multimodal AI Lab, Technical University of Darmstadt, Darmstadt, Germany · Department of Computer Engineering, Sharif University of Technology, Tehran, Iran

Abstract

Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly relies on the model's attention weights on visual tokens as a detection signal. We reveal that coarse-grained attention-based analysis is unreliable due to hidden confounders, specifically token position and object repetition in a description. This leads to Simpson's paradox: the attention trends reverse or disappear when statistics are aggregated. Based on this observation, we introduce HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities. HaloProbe uses balanced training to isolate internal evidence and combines it with a learned prior over external features to recover the true posterior. While intervention-based mitigation methods often degrade utility or fluency by modifying models' internals, we use HaloProbe as an external scoring signal for non-invasive mitigation. Our experiments show that HaloProbe-guided decoding reduces hallucinations more effectively than state-of-the-art intervention-based methods while preserving utility.

Figures & tables

Figure 1: Overview of HaloProbe. Given an image and a prompt, an LVLM generates a caption. HaloProbe adopts a Bayesian formulation that combines internal features (e.g., attention and logit statistics) with external caption features (e.g., object repetition and its token position) through a balanced estimator and a prior estimator to produce token-level hallucination scores. HaloProbe enables reliable hallucination detection and downstream hallucination mitigation without modifying model internals.

Panel of Figure 2 — Figure 2: Illustration of Simpson’s paradox induced by token position. (a) Token-position–conditioned image attention, averaged over heads, layers, and samples, for correct and hallucinated object tokens. Image attention is computed by averaging attention values from layers 5 to 18 of LLaVA-1.5-7B and over 5K samples from the MS COCO dataset. Across most positions, hallucinated tokens receive higher conditional attention than correct tokens. (b) Class-conditional token position distributions, showing that hallucinated tokens tend to appear at later positions than correct tokens. When conditioning is removed by marginalizing over token position using the distributions in (b), the expected attention values (dashed lines in (a)) reverse, with correct object tokens exhibiting higher overall attention.

Figure 3: Distribution of object repetition counts (

r\in\{1,2,3,4\}

) conditioned on class. Hallucinated objects are typically mentioned only once, while correct objects are more frequently repeated within a caption.

Panel of Figure 4 — Figure 4: Illustration of Simpson’s paradox induced by object repetition. (a) Token-position-conditioned image attention for correct and hallucinated object tokens, shown separately for first and non-first occurrences. First mentions consistently exhibit higher image attention, even when the object is hallucinated, while non-first mentions attend less to the image. Conditioning on object occurrence largely removes the apparent attention gap between correct and hallucinated tokens. (b) Class-conditional probability of first occurrence as a function of token position, showing that hallucinated objects are more likely to appear as first mentions.

Figure 5: Proportion of correct versus hallucinated objects across token positions in the 5K random samples of MS COCO dataset. The dataset is highly imbalanced, particularly at early token positions.

Method	Acc. $\uparrow$	AUROC $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
IC	62.56	–	61.93	81.60	70.42
UT	50.57	–	53.60	70.62	60.95
EAZY	78.77	–	78.41	83.38	80.82
DIML	84.46	90.19	–	72.34	–
HaloProbe	90.00	93.50	92.50	95.80	94.10

Table 1: Performance comparison of different object hallucination detection methods on LLaVA-1.5-7B backbone. Missing values are indicated with “–”.

Decoding	Method	LLaVA-1.5			Shikra			MiniGPT-4
Decoding	Method	$\mathbf{C_{\text{s}}}\!\downarrow$	$\mathbf{C_{\text{i}}}\!\downarrow$	$\mathbf{F1}\!\uparrow$	$\mathbf{C_{\text{s}}}\!\downarrow$	$\mathbf{C_{\text{i}}}\!\downarrow$	$\mathbf{F1}\!\uparrow$	$\mathbf{C_{\text{s}}}\!\downarrow$	$\mathbf{C_{\text{i}}}\!\downarrow$	$\mathbf{F1}\!\uparrow$
Nucleus	Baseline	53.0	15.2	74.2	54.4	16.0	72.3	30.4	10.4	68.7
	PAI ( Liu et al., 2024c ) *	42.0	13.1	72.0	50.0	14.6	73.6	28.6	9.7	67.2
	HaloProbe + Post-process	15.6	4.2	75.4	13.2	4.3	72.5	10.8	3.7	68.8
Greedy	Baseline	51.6	15.2	75.1	53.0	15.9	72.4	30.6	9.8	69.4
	EAZY ( Che et al., 2025 )	38.8	11.4	-	26.6	8.9	-	-	-	-

Table 2: Comparison of mitigation methods across different decoding strategies on three vision-language models. Lower

C_{\text{s}}

and

C_{\text{i}}

values and higher F1 scores indicate better performance. The best results are shown in bold , and the second-best results are underlined .“–” indicates unavailable or non-comparable results and ”*” indicates reproduced results.

Decoding	Method	$\mathbf{C_{\text{s}}}\!\downarrow$	$\mathbf{C_{\text{i}}}\!\downarrow$	$\mathbf{F1}\!\uparrow$
Nucleus	Baseline	25.2	8.4	74.4
Nucleus	HaloProbe + Post-process	14.4	4.7	74.5
Greedy	Baseline	24.8	8.0	74.6
Greedy	HaloProbe + Post-process	12.2	4.3	74.6
Beam	Baseline	25.8	7.9	74.9
Beam	HaloProbe + Beam	15.6	4.9	75.2

Table 3: Mitigation results on Qwen3-VL. Lower

C_{\text{s}}

and

C_{\text{i}}

values and higher F1 scores indicate better performance.

Decoding	Method	$\mathbf{C_{\text{s}}}\!\downarrow$	$\mathbf{C_{\text{i}}}\!\downarrow$	$\mathbf{F1}\!\uparrow$
Nucleus	Baseline	33.8	9.6	73.8
Nucleus	HaloProbe + Post-process	22.2	6.7	75.1
Greedy	Baseline	31.6	8.7	74.5
Greedy	HaloProbe + Post-process	16.6	5.1	75.7
Beam	Baseline	34.6	9.5	73.7
Beam	HaloProbe + Beam	15.0	5.2	74.0

Table 4: Mitigation results on InternVL3.5. Lower

C_{\text{s}}

and

C_{\text{i}}

values and higher F1 scores indicate better performance.

Attn.	Logits	Tok. Pos.	Tok. Rep.	Tok. Occ.	Acc. $\uparrow$	AUROC $\uparrow$
$\times$	✓	✓	✓	✓	84.4	82.8
✓	$\times$	✓	✓	✓	89.7	92.9
✓	✓	$\times$	✓	✓	88.2	92.0
✓	✓	✓	$\times$	✓	88.8	92.7
✓	✓	✓	✓	$\times$	89.4	92.6
$\times$	$\times$	✓	✓	✓	83.9	81.7

Table 5: Feature ablation study for hallucination detection with HaloProbe. Internal features include attention weights and decoder logit-based signals, while external features include token position, token repetition, and token occurrence. Ablated features are replaced with Gaussian noise.

Train Balanced	Test Balanced	Accuracy $\uparrow$	AUROC $\uparrow$
$\times$	$\times$	89.8	92.2
✓	$\times$	87.2	92.3
$\times$	✓	72.5	87.6
✓	✓	77.6	87.7

Table 6: Effect of class balancing during training and evaluation for the internal estimator

f_{\theta}

. Training and testing are performed either on the natural (imbalanced) distribution or on a position-based class-balanced distribution.

Fine-grained	External	Bayesian	Beam		PP – Greedy		PP – Nucleus		Average
Attention	Features	Decomposition	$C_{\text{s}}\downarrow$	$C_{\text{i}}\downarrow$	$C_{\text{s}}\downarrow$	$C_{\text{i}}\downarrow$	$C_{\text{s}}\downarrow$	$C_{\text{i}}\downarrow$	$C_{\text{s}}\downarrow$	$C_{\text{i}}\downarrow$
$\times$	$\checkmark$	$\checkmark$	26.4	7.3	20.0	6.8	19.8	6.9	22.0	7.0
$\checkmark$	$\times$	$\checkmark$	29.2	8.3	37.4	10.2	41.8	12	36.1	10.1
$\checkmark$	$\checkmark$	$\times$	25.5	7.4	28.8	7.7	23.2	6.2	25.8	7.1
$\checkmark$	$\checkmark$	$\checkmark$	25.2	7.2	17.6	5.2	15.6	4.2	19.4	5.5

Table 7: Ablation of HaloProbe components on the hallucination mitigation task using the LLaVA-1.5 model. We remove fine-grained attention features, external caption features, and the Bayesian decomposition, and evaluate their impact under three mitigation settings: HaloProbe-guided beam search and post-processing (PP) applied to captions generated with greedy and nucleus decoding. Performance is reported using CHAIR metrics, where lower values indicate fewer hallucinated objects.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Balanced Estimator Network	Dimensionality
First occurrence (binary)	1
Repetition count (clipped $[1,4]$ )	1
Mean of top-20 image attentions (current decoding step, all layers $\times$ heads)	$32\times 32$
Mean of top-20 image attentions (next decoding step, all layers $\times$ heads)	$32\times 32$
Top-20 image attentions entropy (current decoding step, all layers $\times$ heads)	$32\times 32$
Top-20 image attentions entropy (next decoding step, all layers $\times$ heads)	$32\times 32$

Appendix

Table 8: Input features for the hallucination detection balanced estimator and the prior network. The balanced estimator uses normalized features including attention statistics, logit-based features, and token metadata. The prior network uses only repetition and token position normalized to

[0,1]

. The reported dimensionality corresponds to the LLaVA-1.5 model, which consists of 32 layers and 32 attention heads.

Condition	Len	Vocab	RE-2	Rep-2	Dist-2	Span
No Intervention	91.48	53.76	0.094	0.165	0.906	3.23
Attention Intervention	96.18	49.48	0.154	0.260	0.846	6.98

Appendix

Table 9: Decoding stability and redundancy metrics for greedy decoding with and without attention intervention. Lower is better for redundancy metrics and repeated span length.

Figure 6: Qualitative comparison of image description results. Given the same user prompt, the baseline model LLaVA-1.5 produces a coherent description with a low repetition score, while the intervention induces severe repetitive generation, reflected by a high RE-4 score.

Figure 7: Qualitative comparison of image description results. Given the same user prompt, the baseline model LLaVA-1.5 produces a coherent description with a low repetition score, while the intervention induces severe repetitive generation, reflected by a high RE-4 score.

Figure 8: Averaged image attention for first-occurrence object tokens, averaged over early and late transformer layers. Early (first 10) layers exhibit a rapid decay in image attention as generation progresses, while late (last 10) layers maintain relatively stable attention across token positions. Attention in early layers is largely non-discriminative between correct and hallucinated tokens, whereas in late layers, hallucinated tokens counterintuitively receive higher image attention than correct tokens. Compared with Fig. 4(a) , this result indicates that LVLM layers behave differently and that averaging attention across all layers can obscure meaningful patterns.

Model	Accuracy $\uparrow$	AUROC $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
LLaVA-1.5	90.0	93.5	92.5	95.8	94.1
Shikra	90.2	91.8	92.9	96.0	94.4
MiniGPT-4	91.0	93.6	93.7	96.1	95.0

Appendix

Table 10: Performance comparison of HaloProbe across different vision-language models.

Figure 9: Consistent performance of HaloProbe across token positions.

Panel of Figure 10 — Figure 10: ROC and Precision-Recall curves of HaloProbe for token-level hallucination detection, illustrating performance under class imbalance.

Task	Method	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
Relation	Baseline	67.4	83.2	55.2	66.3
Relation	HaloProbe	81.2	88.6	80.2	84.2
Attribute	Baseline	75.1	76.0	81.1	78.5
Attribute	HaloProbe	88.1	87.6	88.8	88.2

Appendix

Table 11: Results on the AMBER benchmark using LLaVA-1.5. Higher values indicate better performance. The best results are shown in bold .

Dataset	Method	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
COCO	Baseline	86.8	94.0	78.7	85.7
COCO	HaloProbe	90.7	93.0	88.1	90.5
GQA	Baseline	84.8	84.6	85.7	85.1
GQA	HaloProbe	89.0	88.1	90.3	89.2
AOKVQA	Baseline	86.2	85.7	87.8	86.6
AOKVQA	HaloProbe	90.2	88.9	92.2	90.5

Appendix

Table 12: Results on the POPE benchmark using LLaVA-1.5. Higher values indicate better performance. The best results are shown in bold .

Figure 11: Qualitative comparison of image description results using LLaVA-1.5-7B. The Baseline model hallucinates objects that are not present in the image, while HaloProbe + Post-Process effectively removes such hallucinations while preserving most of the correctly mentioned objects and maintaining caption fluency. Green = correct object; red = hallucinated object.

Figure 12: Qualitative comparison of image description results using LLaVA-1.5-7B. The Baseline model hallucinates objects that are not present in the image, while HaloProbe + Post-Process effectively removes such hallucinations while preserving most of the correctly mentioned objects and maintaining caption fluency. Green = correct object; red = hallucinated object.

Figure 13: Qualitative comparison of image description results using LLaVA-1.5-7B. The Baseline model hallucinates objects that are not present in the image, while HaloProbe + Post-Process effectively removes such hallucinations while preserving most of the correctly mentioned objects and maintaining caption fluency. Green = correct object; red = hallucinated object.

Figure 14: Qualitative comparison of image descriptions using LLaVA-1.5-7B. The Beam Search baseline hallucinates objects not present in the image. In contrast, HaloProbe + Beam Search selects the beam with the fewest hallucinated words and the highest number of correctly mentioned objects. Green = correct object; red = hallucinated object.

Figure 15: Qualitative comparison of image descriptions using LLaVA-1.5-7B. The Beam Search baseline hallucinates objects not present in the image. In contrast, HaloProbe + Beam Search selects the beam with the fewest hallucinated words and the highest number of correctly mentioned objects Green = correct object; red = hallucinated object.

Figure 16: Qualitative comparison of image descriptions using LLaVA-1.5-7B. The Beam Search baseline hallucinates objects not present in the image. In contrast, HaloProbe + Beam Search selects the beam with the fewest hallucinated words and the highest number of correctly mentioned objects. Green = correct object; red = hallucinated object.

Hallucination Detection in VLMs

Hallucination detection in vision-language models identifies claims that are unsupported by images, video, or other visual evidence.

See all 72 papers

HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token
Mar 5, 2026Sai Akhil Kogilathota, Sripadha Vallabha E G, Luzhe Sun +1Hallucination Detection in VLMs
Stony Brook University · Toyota Technological Institute at Chicago
Robustness of Internal Signals for Hallucination Detection in Vision-Language Models
Oct 19, 2026Allison Chen, William Yang, Salma Abdel Magid +3Hallucination Detection in VLMs VLM Robustness
Department of Computer Science, Princeton University, Princeton, NJ 08544
Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models
Apr 4, 2026Sohyeon Kim, Sang Yeon Yoon, Kyeongbo KongObject Hallucination in VLMs VLM Hallucination Mitigation
Pusan National University · Pukyong National University

Mar 5, 2026J/K move · Enter open · S save

HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token

Sai Akhil Kogilathota, Sripadha Vallabha E G, Luzhe Sun, Jiawei Zhou

Stony Brook University · Toyota Technological Institute at Chicago

Hallucination Detection in VLMs

Open paper AskPDF arXiv

Hallucinations remain a persistent challenge for vision-language models (VLMs), which often describe nonexistent objects or fabricate facts. Existing detection methods typically operate after text generation, making intervention both costly and untimely. We investigate whether hallucination risk can instead be predicted before any token is generated by probing a model's internal representations in a single forward pass. Across a diverse set of vision-language tasks and eight modern VLMs, including Llama-3.2-Vision, Gemma-3, Phi-4-VL, and Qwen2.5-VL, we examine three families of internal representations: (i) visual-only features without multimodal fusion, (ii) vision-token representations within the text decoder, and (iii) query-token representations that integrate visual and textual information before generation. Probes trained on these representations achieve strong hallucination-detection performance without decoding, reaching up to 0.93 AUROC on Gemma-3-12B, Phi-4-VL 5.6B, and Molmo 7B. Late query-token states are the most predictive for most models, while visual or mid-layer features dominate in a few architectures (e.g., ~0.79 AUROC for Qwen2.5-VL-7B using visual-only features). These results demonstrate that (1) hallucination risk is detectable pre-generation, (2) the most informative layer and modality vary across architectures, and (3) lightweight probes have the potential to enable early abstention, selective routing, and adaptive decoding to improve both safety and efficiency.

Abstract

Hallucination Detection in VLMs

Explore similar work

HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token

Robustness of Internal Signals for Hallucination Detection in Vision-Language Models

Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models