PLANE: Plug-and-Play Fine-Grained Visual Experts for Hallucination Mitigation in Large Vision-Language Models

Jianghao Yin; Lang Yu; Qianxi Yu; Qin Chen; Jie Zhou; Zhikai Lei; Liang He

cs.CVOct 18, 2026

PLANE: Plug-and-Play Fine-Grained Visual Experts for Hallucination Mitigation in Large Vision-Language Models

Authors: Jianghao Yin, Lang Yu, Qianxi Yu, Qin Chen, Jie Zhou, Zhikai Lei, Liang He

Organizations: School of Computer Science and Technology, East China Normal University · Shanghai Qiji Zhifeng Co., Ltd.

Abstract

Large Vision-Language Models (LVLMs) commonly suffer from hallucinations, which appear as fabricating non-existent objects or outputting incorrect attributes and spatial relationships in images. Previous works empirically indicate that CLIP-style vision encoders tend to focus on primary objects while ignoring image details, which is one of the causes of hallucinations. To address this issue, we propose plug-and-play fine-grained visual experts (PLANE) comprising two components: LoRA visual experts operating on CLIP for multi-aspect image encoding, and an instruction-aware gate network for adaptive image feature fusion. During training, we construct a fine-grained image-caption pairs dataset, where images are described from three aspects: objects, attributes, and relations between objects. Image-caption pairs from different aspects are used to train the three fine-grained LoRA experts respectively. Aiming at comprehensively evaluate our method's capabilities, we manually annotate a fine-grained hallucination evaluation benchmark for LVLMs which is GPT4-independent during evaluation phase (GIFT). We conduct experiments both on popular hallucination evaluation benchmarks and GIFT. The results show that our method effectively reduces hallucinations across models of different types and sizes while introducing minimal inference latency. Compared to advanced baselines, our model also demonstrates superior performance.

Figures & tables

Figure 1: A hallucination case in LVLMs, with different colors representing object hallucinations, attribute hallucinations, and relation hallucinations.

Figure 2: The construction process for the fine-grained image-caption pairs dataset, where different colors indicate descriptions related to object, attribute, and relation.

Figure 3: Overview of the PLANE. The input image is encoded by three LoRA visual experts and the original LVLM encoder, generating multi-level image features. These features are fused by Ins-Gate based on the instruction for LVLM inference.

Figure 4: Pipeline of GIFT. The LVLM output is fed into a scene graph parser to extract triplets, which are then input into Llama-Judge along with the detailed caption and manually annotated triplets for hallucination detection.

Model	Method	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Total $\uparrow$
LLaVA-1.5 7B	Original	$179.00_{(\pm 2.24)}$	$115.00_{(\pm 15.76)}$	$104.67_{(\pm 20.26)}$	$156.71_{(\pm 4.88)}$	555.38
	OHD-Caps	$180.00_{(\pm 3.54)}$	$120.67_{(\pm 9.47)}$	$101.67_{(\pm 17.36)}$	$157.00_{(\pm 9.08)}$	559.34
	HA-DPO	$183.33_{(\pm 2.89)}$	$118.89_{(\pm 7.88)}$	$112.78_{(\pm 3.47)}$	$149.44_{(\pm 5.85)}$	564.44
	HALVA	$181.67_{(\pm 2.89)}$	$116.33_{(\pm 14.43)}$	$\mathbf{122.33_{(\pm 1.92)}}$	$160.00_{(\pm 10.00)}$	580.33
	VCD	$182.00_{(\pm 2.74)}$	$128.00_{(\pm 7.58)}$	$115.00_{(\pm 15.76)}$	$147.66_{(\pm 20.26)}$	572.66
	DROPOUT DEC.	$183.00_{(\pm 5.70)}$	$121.33_{(\pm 12.38)}$	$119.67_{(\pm 3.41)}$	$148.67_{(\pm 6.61)}$	572.67

Table 1: Results on MME. The best results are shown in bold, and

\Delta

indicates the change in performance between our method compared to the original model. Standard deviation is shown in parentheses.

Model	Method	GIFT		AMBER
Model	Method	Hallu $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	CHAIR $\downarrow$	Cog $\downarrow$
LLaVA-1.5	Original	23.39	52.96	36.5	8.1	3.4
	OHD-Caps	22.77	53.06	36.6	7.8	3.5
	HA-DPO	23.43	52.78	35.0	7.7	3.4
	VCD	25.25	53.55	38.1	8.4	4.2
	VAF	23.05	53.74	37.9	8.0	3.9

Table 2: Results on GIFT and AMBER.

Figure 5: Inference time comparison.

Method	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Total $\uparrow$
Original	176.33	86.00	84.33	87.00	433.66
w Object	186.00	81.67	84.33	94.67	446.67
w Relation	179.00	97.67	88.67	102.67	468.01
w Attribute	175.33	92.00	76.67	108.33	452.33
w/o Ins-Gate	180.66	89.00	87.33	106.00	462.99
PLANE	183.67	99.00	85.67	103.00	471.34

Table 3: Ablation study.

	Human		LLaMA-Judge			GPT4
Method	Hallu $\downarrow$	Cover $\uparrow$	Hallu $\downarrow$	Cover $\uparrow$	CWH $\uparrow$	Hallu $\downarrow$	Cover $\uparrow$	CWH $\uparrow$
Original	33.99	28.04	32.78	28.56	83.70	54.82	19.20	71.81
PLANE	30.66	28.09	28.97	28.84	83.70	50.41	20.14	71.81

Table 4: Human Evaluation on GIFT. CWH (Consistency with Human) measures the percentage of triplets where model judgments match human annotations.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Aspect	Count
Image-Object Caption Pairs	7800
Image-Attribute Caption Pairs	18121
Image-Relation Caption Pairs	12454
Total	38375

Appendix

Table 5: Fine-grained image-caption dataset statistical information.

Dataset	Setting	Method	Accuracy $\uparrow$	Precision	Recall	F1 Score $\uparrow$
A-OKVQA	Random	Regular	83.45	87.24	78.36	82.56
		VCD Leng et al. (2024)	86.15	85.18	87.53	86.34
		RITUAL Woo et al. (2024)	85.17	79.79	-	86.40
		DeGF Zhang et al. (2025)	86.93	84.28	-	87.42
		Ours	88.93	95.13	82.07	88.11
	Popular	Regular	79.90	80.85	78.36	79.59

Appendix

Table 6: Results on POPE. Best results are in bold, and second-best values are underlined.

Model	Hallu Rate $\downarrow$	Cover Rate $\uparrow$
Vary Toy 1.8B	29.7	29.8
LLaVA 7B	19.0	53.2
LLaVA 13B	19.4	53.4
LLaVA-1.5 7B	23.4	53.0
QwenVL 7B	14.3	21.4
Cambrian 8B	19.3	66.0

Appendix

Table 7: Results on GIFT. The best results are shown in bold.

Benchmark	Object	Attribute	Relation	GPT Free
CHAIR	✓	✗	✗	✓
AMBER	✓	✗	✗	✓
GAVIE	✓	✗	✗	✗
Faithscore	✓	✓	✓	✗
GIFT	✓	✓	✓	✓

Appendix

Table 8: Comparison between GIFT and existing generative hallucination benchmarks. Object, Attribute, and Relation denote coverage of corresponding hallucination types, while GPT-free indicates whether the evaluation runs without relying on ChatGPT/GPT-4.

Figure 6: Results on MME-Full set.

Method	Trainable Parameters
HA-DPO	340.8 M
HALVA	340.8 M
PLANE	332.4 M

Appendix

Table 9: Trainable Parameters Comparison.

Method	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Total $\uparrow$
Original	176.33	86.00	84.33	87.00	433.66
All in one	165.00	90.00	86.67	103.89	445.56
PLANE	183.67	99.00	85.67	103.00	471.34

Appendix

Table 10: Ablation on the necessary of Separate Experts.

Model	Method	Length (Avg. Triplets)	Hallu Rate (%)	Cover Rate (%)
LLaVA-1.5	Original	7.18	23.39	52.96
LLaVA-1.5	PLANE	7.03	20.30	54.07
Vary-Toy	Original	5.02	29.71	29.79
Vary-Toy	PLANE	5.08	28.31	30.12

Appendix

Table 11: Response Length Analysis.

Figure 7: Case study on GIFT. The sentences that experience hallucinations are marked in red.

Figure 8: Prompt for LLaMA-Judge.

Model	Method	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Total $\uparrow$
LLaVA-1.5 7B	Original	$179.00_{(\pm 2.24)}$	$115.00_{(\pm 15.76)}$	$104.67_{(\pm 20.26)}$	$156.71_{(\pm 4.88)}$	555.38
	Image-Level	$\mathbf{184.00_{(\pm 2.24)}}$	$113.34_{(\pm 1.67)}$	$\mathbf{112.33_{(\pm 16.01)}}$	$151.33_{(\pm 15.20)}$	561.00
	Token-Level	$\mathbf{184.00_{(\pm 4.18)}}$	$\mathbf{119.67_{(\pm 5.05)}}$	$112.00_{(\pm 13.10)}$	$\mathbf{166.67_{(\pm 12.19)}}$	582.34
Vary Toy 1.8B	Original	$176.33_{(\pm 2.74)}$	$86.00_{(\pm 2.74)}$	$84.33_{(\pm 2.74)}$	$87.00_{(\pm 9.76)}$	433.66
	Image-Level	$174.67_{(\pm 3.41)}$	$90.00_{(\pm 5.27)}$	$\mathbf{90.67_{(\pm 5.22)}}$	$89.00_{(\pm 2.24)}$	444.34
	Token-Level	$\mathbf{183.67_{(\pm 3.20)}}$	$\mathbf{99.00_{(\pm 2.74)}}$	$85.67_{(\pm 2.74)}$	$\mathbf{103.00_{(\pm 1.90)}}$	471.34

Appendix

Table 12: Comparison of different fusion strategies. The best results are shown in bold. Standard deviation is shown in parentheses.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 227 papers

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
May 20, 2026Yutong Xie, Zhenglin Hua, Ran Wang +3Vision-Language Models Hallucination in Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs
Apr 23, 2026Pegah Khayatan, Jayneel Parekh, Arnaud Dapogny +3VLM Evaluation LLM Hallucination Mitigation
1ISIR, Sorbonne Universit´e, Paris, France · 2Valeo.ai, Paris, France

May 20, 2026J/K move · Enter open · S save

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Yutong Xie, Zhenglin Hua, Ran Wang, Wing W. Y. Ng+2

School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China+3

Vision-Language Models Hallucination in Language Models VLM Hallucination Large Vision-Language Models VLM Hallucination Mitigation

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD). Specifically, we obtain the attention weights from early generated tokens to visual tokens across layers and identify the tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain attention scores of generated text to visual evidence, in order to select and emphasize text tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple benchmark evaluations conducted on five recently released models show that our method can consistently mitigate hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.

Abstract

VLM Hallucination Mitigation

Explore similar work

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs