PVFH-Bench: Benchmarking Premise-Induced Visual Fixation Hallucinations in Vision-Language Models

Xinhao Wang; Junbo Cui; Zhenghao Liu; Yuan Yao; Xiaohua Li; Yu Gu; Ge Yu

cs.CLOct 10, 2026

PVFH-Bench: Benchmarking Premise-Induced Visual Fixation Hallucinations in Vision-Language Models

Authors: Xinhao Wang, Junbo Cui, Zhenghao Liu, Yuan Yao, Xiaohua Li, Yu Gu, Ge Yu

Organizations: School of Computer Science and Engineering, Northeastern University, Shenyang, China · ModelBest Inc. · State Key Lab. of Novel Software Technology, Nanjing University, P.R. China · Department of Computer Science and Technology, Institute for AI, Tsinghua University, China

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable progress across a wide range of multimodal tasks. However, object hallucination remains a persistent challenge, undermining model reliability in applications such as visual question answering, embodied agents, and human-AI decision support systems. In this paper, inspired by cognitive science studies on perceptual bias, we investigate a new failure mode termed Premise-Induced Visual Fixation Hallucination (PVFH). Specifically, PVFH refers to a premise-induced reasoning error in which models become anchored to visually confusable evidence, accept a false textual premise, and subsequently generate responses about an absent yet visually plausible object. To systematically study this phenomenon, we introduce PVFH-Bench, a benchmark that automatically constructs visually confusable but absent targets and pairs the core hallucination setting with diagnostic subsets to trace how models move from object recognition to Premise-Induced Visual Fixation Hallucination. Experimental results on 10 state-of-the-art LVLMs demonstrate that all evaluated models are highly vulnerable to Premise-Induced Visual Fixation Hallucination. Moreover, we find that explicit chain-of-thought reasoning can further exacerbate this failure mode by encouraging models to overinterpret weak visual cues as supporting evidence for hallucinated objects.

Figures & tables

Figure 1: A motivating example of Premise-induced Visual Fixation Hallucination: visual fixation is guided by a false premise, leading the model to fabricate a spatial relation for an absent object.

Figure 2: Premise-Induced Visual Fixation Hallucination example and PVFH-Bench construction pipeline.

Model	Vanilla	Visual Fixation	False Premise	Premise-Induced Visual Fixation	Overall
	Actual Object	Confusable Object	Absent Object	Core Setting	Avg. Acc.
Closed-Source Models
Qwen3.5-Plus	98.84	79.84	93.80	$\mathbf{74.81}_{{\color[rgb]{1,0,0}\text{{-5.04}}}}$	86.82
+Reasoning	100.00	78.29	63.18	$36.82_{{\color[rgb]{1,0,0}\text{{-41.47}}}}$	69.57
Seed-2.0-Lite	97.29	82.56	82.95	$68.60_{{\color[rgb]{1,0,0}\text{{-13.95}}}}$	82.85
+Reasoning	96.12	82.95	77.13	$52.33_{{\color[rgb]{1,0,0}\text{{-30.62}}}}$	77.13

Table 1: Model accuracy on PVFH-Bench across the core Premise-Induced Visual Fixation setting and three diagnostic subsets. Red italic subscripts in the Premise-Induced Visual Fixation column indicate the accuracy difference from the Visual Fixation column for the same row. +Reasoning rows report reasoning-mode results for the corresponding base model.

Panel of Figure 3 — Figure 3: Average reasoning length and accuracy along two diagnostic paths for reasoning-mode models. The left column isolates the added effect of false-premise pressure. The right column isolates the added effect of visually confusable object.

Panel of Figure 4 — Figure 4: Failure analysis of Premise-Induced Visual Fixation errors for reasoning-based LVLM.

Figure 5: Analyses of false-premise induced reasoning.

Figure 6: Seed model accuracy under different reasoning settings.

Figure 7: Two representative hallucination cases in Qwen3.5 Plus.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Property	Valid / Total	Valid Rate
True object	41 / 43	95.3%
False object	113 / 114	99.1%

Appendix

Table 2: Human validation of benchmark construction and automatic grading. Left: validity of deduplicated image-level presence/absence annotations. Right: agreement between blind human decisions and Qwen3.5-Flash labels on a stratified sample of 200 responses. Core PVFH denotes the Premise-Induced Visual Fixation setting.

Model	Vanilla	Visual Fixation	False Premise	Core PVFH	Overall	Overall $\Delta$
Qwen3.5-Plus	99.22	79.46	93.41	72.09	86.05	$-0.78$
Seed-2.0-Lite	97.29	82.17	79.46	54.26	78.29	$-4.55$
Qwen3-VL-8B-Instruct	98.84	81.01	81.01	58.91	79.94	$+0.29$
Qwen3-VL-32B-Instruct	97.67	74.42	86.82	56.98	78.97	$-0.58$
LLaMA-4-Scout	94.19	81.78	84.88	50.39	77.81	$-0.48$
LLaMA-3.2-11B-Vision-Instruct	98.06	76.74	87.21	46.90	77.23	$+0.29$

Appendix

Table 3: Full non-reasoning leaderboard re-graded by the independent GPT-5.4-mini judge. Scores are accuracies in percentage points. Overall

\Delta

is measured relative to Qwen3.5-Flash grading.

Setting	Stage	Mean Attn./Token	Median Attn./Token	SD Attn./Token	Mean Total Attn.
Core PVFH	Reasoning	0.0097	0.0042	0.0144	0.1491
Core PVFH	Answer	0.0069	0.0025	0.0114	0.0351
Visual Fixation	Reasoning	0.0104	0.0048	0.0140	0.0600
Visual Fixation	Answer	0.0059	0.0024	0.0090	0.0249

Appendix

Table 4: Attention assigned to the grounded true-object region in 50 paired examples. Per-token statistics summarize regional attention over the corresponding generation stage; mean total attention aggregates it across all tokens in that stage.

Statistic	Value
Images / object pairs	258
Evaluation instances	1032
Unique actual objects	160
Unique confusable objects	160
Unique generated absent objects	36
Unique objects across all roles	293

Appendix

Table 5: Dataset statistics for PVFH-Bench.

Evaluation	Images	Instances
POPE on MSCOCO ( Li et al., 2023b )	500	3,000
MME: Existence, Count, Position, Color ( Fu et al., 2025 )	120	240
PVFH-Bench	258	1,032

Appendix

Table 6: Scale comparison with controlled object-level hallucination diagnostics.

Figure 8: Chinese prompt used in the construction code for identifying a true-false object pair.

Figure 9: English rendering of the object-pair identification prompt in Figure 8 .

Relation type	Template
left_right_of_anchor	Is the {false object} to the left or to the right of the {true object}?
above_below_anchor	Is the {false object} above or below the {true object}?
closer_farther_anchor	Is the {false object} closer to the camera or farther from the camera than the {true object}?
larger_smaller_anchor	Does the {false object} appear larger or smaller than the {true object}?
overlap_left_right_anchor	Is the {false object} positioned more to the left or more to the right relative to the {true object}?
image_left_right	Is the {false object} located more toward the left side or the right side?

Appendix

Table 7: Spatial-relation templates used to instantiate false-premise questions.

Figure 10: Evaluation prompts for Raw Query, Visual Fixation Query, Premise-Induced Visual Fixation Query, and False-Premise Diagnostic Query.

Setting	Qwen3-VL-8B	Qwen3-VL-32B
POPE	88.84	89.11
Vanilla	98.45	97.67
Visual Fixation	80.62	74.03
False-Premise	81.40	88.37
Premise-Induced Visual Fixation	58.14	58.14

Appendix

Table 8: Reported POPE accuracy ( Huang et al., 2026 ) compared with PVFH-Bench accuracy for Qwen3-VL models.

Figure 11: Prompt used to classify failure mechanisms for Visual-Fixation-correct but Premise-Induced-Visual-Fixation-failed cases.

Figure 12: Prompt used to judge reasoning-chain alignment between Visual Fixation and Premise-Induced Visual Fixation settings.

Object Hallucination in VLMs

Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.

See all 128 papers

When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs
Apr 23, 2026Pegah Khayatan, Jayneel Parekh, Arnaud Dapogny +3VLM Evaluation LLM Hallucination Mitigation
1ISIR, Sorbonne Universit´e, Paris, France · 2Valeo.ai, Paris, France
Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
Aug 7, 2026Zichuan Wang, Songlin Yang, Bo Peng +4Hallucination Detection in VLMs Vision-Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Hong Kong University of Science and Technology · 2New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science
DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models
Apr 18, 2026JiYang Wang, Jiawei Chen, Mengqi Xiao +3VLM Evaluation Hallucination Detection in VLMs
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy

Apr 23, 2026J/K move · Enter open · S save

When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs

Pegah Khayatan, Jayneel Parekh, Arnaud Dapogny, Mustafa Shukor+2

1ISIR, Sorbonne Universit´e, Paris, France · 2Valeo.ai, Paris, France

VLM Evaluation LLM Hallucination Mitigation Hallucination in Language Models VLM Hallucination VLM Hallucination Mitigation

Open paper AskPDF arXiv

Despite impressive progress in capabilities of large vision-language models (LVLMs), these systems remain vulnerable to hallucinations, i.e., outputs that are not grounded in the visual input. Prior work has attributed hallucinations in LVLMs to factors such as limitations of the vision backbone or the dominance of the language component, yet the relative importance of these factors remains unclear. To resolve this ambiguity, We propose HalluScope, a benchmark to better understand the extent to which different factors induce hallucinations. Our analysis indicates that hallucinations largely stem from excessive reliance on textual priors and background knowledge, especially information introduced through textual instructions. To mitigate hallucinations induced by textual instruction priors, we propose HalluVL-DPO, a framework for fine-tuning off-the-shelf LVLMs towards more visually grounded responses. HalluVL-DPO leverages preference optimization using a curated training dataset that we construct, guiding the model to prefer grounded responses over hallucinated ones. We demonstrate that our optimized model effectively mitigates the targeted hallucination failure mode, while preserving or improving performance on other hallucination benchmarks and visual capability evaluations. To support reproducibility and further research, we will publicly release our evaluation benchmark, preference training dataset, and code at https://pegah-kh.github.io/projects/prompts-override-vision/ .

Abstract

Object Hallucination in VLMs

Explore similar work

When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs

Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models