EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectors

Ryuhei Miyazato; Shunsuke Kitada; Kei Harada

cs.CVApr 3, 2026

EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectors

Authors: Ryuhei Miyazato, Shunsuke Kitada, Kei Harada

Organizations: The University of Electro-Communications

Abstract

Vision-Language Models (VLMs) excel at multimodal tasks, but they remain vulnerable to hallucinations that are factually incorrect or ungrounded in the input image. Recent work suggests that hallucination detection using internal representations is more efficient and accurate than approaches that rely solely on model outputs. However, existing internal-representation-based methods typically rely on a single representation or detector, limiting their ability to capture diverse hallucination signals. In this paper, we propose EnsemHalDet, an ensemble-based hallucination detection framework that leverages multiple internal representations of VLMs, including attention outputs and hidden states. EnsemHalDet trains independent detectors for each representation and combines them through ensemble learning. Experimental results across multiple VQA datasets and VLMs show that EnsemHalDet consistently outperforms prior methods and single-detector models in terms of AUC. These results demonstrate that ensembling diverse internal signals significantly improves robustness in multimodal hallucination detection.

Figures & tables

Figure 1: VLMs can produce hallucinated responses that are inconsistent with factual knowledge or image content. However, such hallucinations leave detectable signals in the model’s internal representations. We leverage multiple internal states of VLMs to achieve robust and accurate hallucination detection

Figure 2: Overview of EnsemHalDet: This method extracts attention heads and hidden states across multiple layers. For each representation, binary classifiers are trained via logistic regression to detect hallucinations. By integrating detectors from the most effective layers and heads through ensemble learning, we detect hallucinated answers.

Figure 3: Overview of the detector-level ensemble process. For attention-head-based features (AH), we train

L\times H

detectors, one for each layer-head pair. For hidden-state-based features (HS), we train

L

detectors, one for each layer. The detectors are first ranked on validation 1 , filtered to top-K candidates, further refined by greedy forward selection on validation 2 , and finally combined by a stacking-based meta-classifier (logistic regression).

Methods	CRAG-MM			MMMU-Pro
	Llama-3.2-11B	Qwen2.5-VL-7B	Pixtral-12B	Llama-3.2-11B	Qwen2.5-VL-7B	Pixtral-12B
SAPLMA	0.817 $\pm$ 0.023	0.797 $\pm$ 0.023	0.730 $\pm$ 0.045	0.850 $\pm$ 0.026	0.615 $\pm$ 0.076	0.791 $\pm$ 0.015
MIND	0.822 $\pm$ 0.011	0.830 $\pm$ 0.018	0.750 $\pm$ 0.021	0.866 $\pm$ 0.039	0.686 $\pm$ 0.047	0.801 $\pm$ 0.034
MHAD ${}_{\text{attn}}$	0.850 $\pm$ 0.019	0.867 $\pm$ 0.012	0.784 $\pm$ 0.012	0.870 $\pm$ 0.033	0.751 $\pm$ 0.055	0.822 $\pm$ 0.034
MHAD ${}_{\text{hs}}$	0.838 $\pm$ 0.028	0.849 $\pm$ 0.011	0.763 $\pm$ 0.033	0.889 $\pm$ 0.023	0.707 $\pm$ 0.056	0.792 $\pm$ 0.046
MHAD ${}_{\text{ffn}}$	0.830 $\pm$ 0.029	0.855 $\pm$ 0.020	0.777 $\pm$ 0.038	0.874 $\pm$ 0.040	0.742 $\pm$ 0.050	0.810 $\pm$ 0.042

Table 1: Main comparison of hallucination detection methods with AUC. All results are reported as the mean and standard deviation over five runs of feature extraction with different random seeds. Best results are in bold.

Variants	CRAG-MM			MMMU-Pro
	Llama-3.2-11B	Qwen2.5-VL-7B	Pixtral-12B	Llama-3.2-11B	Qwen2.5-VL-7B	Pixtral-12B
Attention-based (AH)
AH ${}_{\text{top1}}$	0.787 $\pm$ 0.022	0.842 $\pm$ 0.029	0.744 $\pm$ 0.046	0.812 $\pm$ 0.032	0.749 $\pm$ 0.025	0.737 $\pm$ 0.032
AH ${}_{\text{concat}}$	0.795 $\pm$ 0.031	0.833 $\pm$ 0.017	0.734 $\pm$ 0.029	0.817 $\pm$ 0.047	0.689 $\pm$ 0.044	0.738 $\pm$ 0.042
AH ${}_{\text{average}}$	0.843 $\pm$ 0.021	0.870 $\pm$ 0.021	0.801 $\pm$ 0.029	0.855 $\pm$ 0.034	0.736 $\pm$ 0.035	0.803 $\pm$ 0.028
AH ${}_{\text{weighted}}$	0.842 $\pm$ 0.021	0.871 $\pm$ 0.023	0.805 $\pm$ 0.026	0.855 $\pm$ 0.034	0.747 $\pm$ 0.040	0.797 $\pm$ 0.029

Table 2: Ablation study of EnsemHalDet. We compare feature types (AH, HS, MIX) and combination strategies (top1, concatenation, average ensemble, weighted ensemble, and stacking).

	CRAG-MM
	Llama	Qwen	Pixtral
HS ${}_{\text{top1}}$ (w/ PCA)	0.857	0.863	0.818
HS ${}_{\text{top1}}$ (w/o PCA)	0.755	0.775	0.721
HS ${}_{\text{stack}}$ (w/ PCA)	0.860	0.864	0.817
HS ${}_{\text{stack}}$ (w/o PCA)	0.774	0.782	0.750

Table 3: Effect of PCA on CRAG-MM for hidden-state-based detectors. PCA consistently improves both single-detector (top1) and ensemble (stack) performance. Full results are provided in Table 6 .

Method	Feature Extraction (s)	Detection (s)	Total (s)	Percentage (%)
Baseline	2.511	–	2.511	100.0
SAPLMA	3.767	0.000060	3.767	150.0
MIND	5.591	0.001465	5.592	222.7
MHAD ${}_{\text{attn}}$	3.095	0.001835	3.097	123.3
MHAD ${}_{\text{hs}}$	2.952	0.002083	2.954	117.7
MHAD ${}_{\text{ffn}}$	3.182	0.001325	3.183	126.8

Table 4: Inference time comparison for hallucination detection. The percentage indicates the total runtime relative to the baseline generation time (100%). We report GPU-based inference results for SAPLMA, MIND, and MHAD.

Model	#Layers	#Heads	Hidden Dim	Multimodal Fusion	Hallucination Rate
Llama-3.2-11B-Vision-Instruct	40	32	4096	Cross-attention	0.795
Qwen2.5-VL-7B-Instruct	28	28	3584	Projector + Self-attention	0.837
Pixtral-12B-2409	40	32	4096	Projector + Self-attention	0.823

Table 5: Architectural details of VLMs, including hidden state dimensionality.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Prompt used for hallucination evaluation. We follow the CRAG-MM evaluation protocol.

Variants	CRAG-MM			MMMU-Pro
	Llama-3.2-11B	Qwen2.5-VL-7B	Pixtral-12B	Llama-3.2-11B	Qwen2.5-VL-7B	Pixtral-12B
HS ${}_{\text{top1}}$ (w/ PCA)	0.857 $\pm$ 0.015	0.863 $\pm$ 0.014	0.818 $\pm$ 0.024	0.845 $\pm$ 0.027	0.737 $\pm$ 0.062	0.801 $\pm$ 0.030
HS ${}_{\text{top1}}$ (w/o PCA)	0.755 $\pm$ 0.023	0.775 $\pm$ 0.019	0.721 $\pm$ 0.023	0.807 $\pm$ 0.076	0.657 $\pm$ 0.063	0.761 $\pm$ 0.065
HS ${}_{\text{average}}$ (w/ PCA)	0.858 $\pm$ 0.014	0.865 $\pm$ 0.012	0.816 $\pm$ 0.014	0.860 $\pm$ 0.036	0.758 $\pm$ 0.027	0.825 $\pm$ 0.025
HS ${}_{\text{average}}$ (w/o PCA)	0.774 $\pm$ 0.022	0.782 $\pm$ 0.020	0.750 $\pm$ 0.020	0.816 $\pm$ 0.073	0.682 $\pm$ 0.067	0.784 $\pm$ 0.051
HS ${}_{\text{stack}}$ (w/ PCA)	0.860 $\pm$ 0.013	0.864 $\pm$ 0.012	0.817 $\pm$ 0.013	0.860 $\pm$ 0.036	0.762 $\pm$ 0.021	0.826 $\pm$ 0.027

Appendix

Table 6: Ablation study on the effect of PCA for hidden-state-based detectors. Applying PCA consistently improves performance for top1, average ensemble, weighted ensemble, and stacking, while the effect on concatenation is mixed.

Method	Feature Extraction (s)	Detection (s)	Total (s)	Percentage (%)
Baseline	2.511	–	2.511	100.0
SAPLMA (CPU)	3.767	0.000219	3.767	150.0
SAPLMA (GPU)	3.767	0.000060	3.767	150.0
MIND (CPU)	5.591	0.001465	5.592	222.7
MIND (GPU)	5.591	0.001430	5.592	222.7
MHAD ${}_{\text{attn}}$ (CPU)	3.095	0.001835	3.097	123.3

Appendix

Table 7: Detailed runtime breakdown. Feature extraction corresponds to internal representation extraction, while hallucination detection denotes classifier inference time.

Hallucination Detection in VLMs

Hallucination detection in vision-language models identifies claims that are unsupported by images, video, or other visual evidence.

See all 72 papers

Robustness of Internal Signals for Hallucination Detection in Vision-Language Models
Oct 19, 2026Allison Chen, William Yang, Salma Abdel Magid +3Hallucination Detection in VLMs VLM Robustness
Department of Computer Science, Princeton University, Princeton, NJ 08544
HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token
Mar 5, 2026Sai Akhil Kogilathota, Sripadha Vallabha E G, Luzhe Sun +1Hallucination Detection in VLMs
Stony Brook University · Toyota Technological Institute at Chicago
UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space
Aug 4, 2026Amir Mohammad Ezzati, Kiyan Rezaee, Bardiya Kariminia +4Hallucination Detection in VLMs Vision-Language Models
Sharif University of Technology · Michigan State University · Shahid Beheshti University +2

Oct 19, 2026J/K move · Enter open · S save

Robustness of Internal Signals for Hallucination Detection in Vision-Language Models

Allison Chen, William Yang, Salma Abdel Magid, Jonathan Williams+2

Department of Computer Science, Princeton University, Princeton, NJ 08544

Hallucination Detection in VLMs VLM Robustness Object Hallucination in VLMs Visual Evidence Sensitivity in VLMs

Open paper AskPDF arXiv

When given an image and a text prompt, vision-language models (VLMs) can produce seemingly sophisticated text responses. However, they are also prone to hallucinate objects in their output, which can have detrimental consequences in safety-critical applications. To detect these hallucinations, certain methods interestingly leverage internal signals from the VLM as it generates the text response; potentially, these methods use the same signals that led to a hallucination to detect it. We conduct an in-depth analysis to understand the robustness of these detection methods. First, we show that a naive baseline of a generated word's token position in the VLM text response achieves competitive performance. Further, we reveal how some detection methods rely on token position and use this heuristic as a shortcut. Second, we explore detection methods' sensitivity to language and vision modalities. Our experiments suggest that strong detection performance is often driven by detecting language-based hallucinations, and when methods rely on token position, they tend to be less visually grounded and exhibit reduced robustness when hallucinations occur earlier in a text response. Lastly, we demonstrate that different detection methods can carry complementary signals; when ensembling methods together, adding weaker methods to an ensemble of higher performing ones still offers performance improvements. Altogether, while there may be useful internal VLM signals to detect hallucinations, our investigation reveals important limitations of the current methods that try to leverage them.

Abstract

Hallucination Detection in VLMs

Explore similar work

Robustness of Internal Signals for Hallucination Detection in Vision-Language Models

HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token

UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space