FINER: MLLMs Hallucinate under Fine-grained Negative Queries

Rui Xiao; Sanghwan Kim; Yongqin Xian; Zeynep Akata; Stephan Alaniz

cs.CVMar 18, 2026

FINER: MLLMs Hallucinate under Fine-grained Negative Queries

Authors: Rui Xiao, Sanghwan Kim, Yongqin Xian, Zeynep Akata, Stephan Alaniz

Organizations: Technical University of Munich · Munich Center for Machine Learning · Helmholtz Munich · Google · LTCI, Télécom Paris, Institut Polytechnique de Paris, France

Abstract

Multimodal large language models (MLLMs) struggle with hallucinations, particularly with fine-grained queries, a challenge underrepresented by existing benchmarks that focus on coarse image-related questions. We introduce FIne-grained NEgative queRies (FINER), alongside two benchmarks: FINER-CompreCap and FINER-DOCCI. Using FINER, we analyze hallucinations across four settings: multi-object, multi-attribute, multi-relation, and ``what'' questions. Our benchmarks reveal that MLLMs hallucinate when fine-grained mismatches co-occur with genuinely present elements in the image. To address this, we propose FINER-Tuning, leveraging Direct Preference Optimization (DPO) on FINER-inspired data. Finetuning four frontier MLLMs with FINER-Tuning yields up to 24.2% gains (InternVL3.5-14B) on hallucinations from our benchmarks, while simultaneously improving performance on eight existing hallucination suites, and enhancing general multimodal capabilities across six benchmarks. Code, benchmark, and models are available at https://explainableml.github.io/finer-project/.

Figures & tables

Figure 1: We compare the performance InternVL3.5-14B [ 45 ] (Baseline) with the one fine-tuned by FINER-Tuning under negative queries of seven different granularity levels.

Figure 2: Data construction pipeline for FINER benchmarks. For FINER-DOCCI , we extract the positive scene graph (SG) from DOCCI [ 34 ] captions, while for FINER-CompreCap , the SG is provided by CompreCap [ 31 ] . From the positive SG, we generate the negative SG using Qwen3-14B [ 51 ] as negatives generator for FINER-CompreCap and Gemini-2.0-Flash [ 41 ] for FINER-DOCCI . Finally, a rule-based query construction pipeline builds multiple choice questions. In practice, choices are shuffled in both benchmarks.

Figure 3: Training data generation pipeline for FINER-Tuning. (1) We adopt long captions from Pixmo [ 11 ] and extract diverse phrases with PHI-4-14B [ 1 ] . (2) We then prompt the same LLM to modify and generate negative phrases. (3) We construct both positive and negative query-answer tuples via template-based composition or LLM generation.

		FINER-CompreCap				FINER-DOCCI
Models	Size	Multi-obj	Multi-attr	Multi-rel	Wh	Multi-obj	Multi-attr	Multi-rel	Wh
Random Guess	-	4.0	4.0	4.0	4.0	4.0	4.0	4.0	4.0
LRV-V2 [ 24 ]	13B	6.1	6.8	5.6	4.0	6.3	5.4	6.1	5.2
LLaVA-RLHF [ 40 ]	13B	11.4	2.0	1.1	6.9	7.3	3.0	5.1	5.3
RLHF-V [ 54 ]	13B	13.4	6.1	1.6	10.8	13.2	7.2	8.1	7.0
OPA-DPO [ 52 ]	13B	10.9	3.0	2.2	6.9	8.1	5.5	8.3	8.0

Table 1: Paired accuracy (

\text{Acc}_{\text{paired}}

) results on FINER-CompreCap and FINER-DOCCI. ∗ For Gemini-2.5-Flash, we evaluate on the whole FINER-CompreCap and on 3K MCQs per setting in FINER-DOCCI due to the scale of the benchmark.

Figure 4:

\text{Acc}_{\text{paired}}

versus the number of objects, attributes, and relations. Top: FINER-CompreCap ; Bottom: FINER-DOCCI . Dashed arrows show the gain from FINER-Tuning.

Models	M.S.	Text	Chart	M.P.	N.B.	V ∗	Avg.
OmniLMM - 12B	39.7	64.5	24.2	69.7	26.9	52.9	46.3
+RLAIF-V	40.9	64.5	25.1	70.0	19.4	54.4	45.7
LLaVA-1.6-7B	37.6	63.7	54.4	65.0	15.7	53.9	48.4
+FINER-Tuning	39.2	63.9	54.9	68.7	19.8	55.0	50.3
Qwen2.5 - VL-7B	63.7	84.9	87.0	76.7	34.1	72.7	69.8
+FINER-Tuning	64.7	85.1	86.4	77.3	34.1	72.8	70.1

Table 3: Results on six general purpose MLLM benchmarks. M.S.: MMStar [ 7 ] ; Text: TextVQA [ 39 ] ; Chart: ChartQA [ 32 ] ; M.P.: MMVP [ 42 ] ; N.B.: NaturalBench [ 21 ] ; V ∗ : V ∗ Bench [ 48 ]

Figure 5: Qualiative examples of FINER-CompreCap MCQs for each category together with MLLM answers.

Method	Q.Type		FINER-CompreCap				Other
	Neg	Both	Obj	Attr	Rel	Wh	RePOPE	M.S.
Base	-	-	74.2	71.9	49.8	25.5	91.5	68.0
+SFT	✓		47.4	59.7	53.8	38.7	69.1	61.7
+SFT		✓	37.5	49.5	55.2	18.9	92.2	63.3
+DPO	✓		75.8	75.2	52.4	29.8	93.1	68.3
+DPO		✓	76.5	78.3	64.1	36.1	93.1	68.3

Table 4: Ablation study on different training strategies. SFT methods only use

a^{+}

. The base model is InternVL-3.5-8B [ 45 ] . Q.Type: Query Type; M.S.: MMStar [ 7 ]

Train Subset	FINER-CompreCap				Other
	Obj	Attr	Rel	Wh	RePOPE	M.S.
Base	74.2	71.9	49.8	25.5	91.5	68.0
Obj	78.8	76.4	54.2	28.7	93.5	67.9
Attr	71.3	76.7	56.8	26.5	91.5	68.2
Rel	69.2	73.0	66.7	24.1	91.4	67.7
Wh	75.9	75.3	55.0	46.5	92.9	68.3

Table 5: Training-on-subset ablation for FINER-Tuning with InternVL-3.5-8B [ 45 ] . Obj/Attr/Rel denote Multi-obj/Multi-attr/Multi-rel for both training and evaluation.

Figure 6: Positional bias analysis on FINER-CompreCap . We select all

q^{\pm}_{\text{multi-obj}}

q^{\pm}_{\text{multi-attr}}

, and

q^{\pm}_{\text{multi-rel}}

that contain three entities. Since each

q^{-}

always has exactly one negated entity, we cyclically move that negated entity to each of the three positions (and move the corresponding positive entity accordingly), and compute the averaged paired accuracy

\text{Acc}_{\text{paired}}

for each position.

Figure 7: Example of positive scene graph (SG) in FINER-CompreCap . CompreCap [ 31 ] already pairs each image with SG-like annotation. We further adopts Qwen3-14B [ 51 ] to simply rewrite attribute sentences into phrases.

	FINER-CompreCap			FINER-DOCCI
	Obj	Attr	Rel	Obj	Attr	Rel
Acc. (%)	96.4	91.5	82.8	96.1	88.3	85.1

Table 6: Quality assessment of the extracted positive objects, attributes, and relations for FINER-DOCCI using InternVL3.5-8B [ 45 ] as a binary classifier. As a baseline, we also run InternVL3.5-8B as a binary classifier to classify the human annotations from FINER-CompreCap .

Figure 8: Example positive scene graph (SG) extracted by Gemini-2.0-Flash [ 41 ] . Given a long human-annotated caption from DOCCI [ 34 ] , we apply a two-stage extraction pipeline to obtain the positive SG.

Benchmark		$\theta$	Counts	# Re-gen.
C-SG	Obj	0.8	3505	320
	Attr	0.8	4509	414
	Rel	0.4	3494	173
D-SG	Obj	0.8	24,528	3,242
	Attr	0.4	52,911	2,827
	Rel	0.8	15,342	2,143

Table 7: Statistics for the generating negative scene graph for FINER-CompreCap (denoted as C-SG) and FINER-DOCCI (denoted as D-SG).Counts: number of objects, attributes and relations inside the SG annotation.

\theta

: entropy-based filtering threshold; #Re-gen.: number of re-generated negatives.

	FINER-CompreCap			FINER-DOCCI
	obj	attr	rel	obj	attr	rel
Acc. (%)	89.8	91.1	94.1	89.5	88.3	82.8

Table 8: Quality assessment of generated negatives. We show the classification accuracy of Qwen2.5-VL-72B [ 4 ] after classifying the objects (obj), attributes (attr) and relations (rel) in FINER-CompreCap and FINER-DOCCI

Figure 9: Examples of entropy-based filtering for objects, attributes, and relations. The corresponding objects are shown with red bounding boxes. The ground-truth object/attribute/relation is highlighted in green. We prompt Qwen2.5-VL-72B [ 4 ] to select the positive among four negatives. Green text indicates that the model makes an incorrect prediction and chooses a negative with low entropy scores. Blue text shows new negative candidates generated by the LLM. The examples are from both FINER-CompreCap and FINER-DOCCI .

Benchmark	Setting	$k$	# pairs $n_{k}$
FINER-C	$q^{\pm}_{\text{multi-obj}}$	$(1,6)$	$560,560,560,558,535,377$
	$q^{\pm}_{\text{multi-attr}}$	$(1,3)$	$966,472,231$
	$q^{\pm}_{\text{multi-rel}}$	$(1,3)$	$1217,616,307$
	$q^{\pm}_{\text{wh}}$	-	$1583$
FINER-D	$q^{\pm}_{\text{multi-obj}}$	$(1,6)$	$65,496,909,980,874,1676$
	$q^{\pm}_{\text{multi-attr}}$	$(1,5)$	$2451,5363,3092,1575,1843$

Table 9: Distribution of MCQ pairs over entity counts in FINER-CompreCap ( FINER-C ) and FINER-DOCCI ( FINER-D ). For each setting, we refer the entity counts for Obj/Attr/Rel as

k

and the corresponding number of pairs

n_{k}

in matching order.

(1,6)

represents that

k

ranges from 1 to 6.

Figure 10: Examples from our constructed training set to train FINER-Tuning. Positive queries are in green color, while negative queries are in red color. We show both positive ( $(x,q_{+},a^{+}_{+},a^{-}_{+})$ ) and negative ( $(x,q_{-},a^{+}_{-},a^{-}_{-})$ ) preference tuples across four subsets: Multi-obj, Multi-attr, Multi-rel, Wh.

Config	Llava-1.6-7B	Qwen2.5VL-7B	InternVL3.5-8B	InternVL3.5-14B
Training Data	40K	120K	160K	160K
Global BS	64
Optimizer	AdamW [ 29 ]
Learning rate	$5\times 10^{-6}$
Total epochs	1
Warm up ratio	0.1

Table 10: Fine-tuning hyper-parameters for FINER-Tuning on all baselines. Global BS: global batch size. LR scheduler: learning rate scheduler.

\beta

: inverse temperature parameter in the DPO loss, as shown in Eq. 5 . Val. ratio: ratio of validation data size.

Category	Count	Percentage
natural_image	176,881	78.13%
screenshot_ui	36,701	16.21%
chart_graph	8,061	3.56%
document_text	4,739	2.09%

Table 11: Category statistics for Pixmo-caption [ 11 ] .

Fitered?	FINER-CompreCap				Other
	Obj	Attr	Rel	Wh	RePOPE	M.S.
-	74.2	71.9	49.8	25.5	91.5	68.0
Yes	76.8	78.6	62.8	36.1	93.1	68.1
No	76.5	78.3	64.1	36.1	93.1	68.3

Table 12: Filtering to only keep natural images ablation for FINER-Tuning with InternVL-3.5-8B [ 45 ] . Obj/Attr/Rel denote Multi-obj/Multi-attr/Multi-rel for both training and evaluation. The best results are bold.

Figure 11: Qualitative Results from FINER-DOCCI.

		POPE			RePOPE			AMBER
Models	Size	Ran. $\uparrow$	Pop. $\uparrow$	Adv. $\uparrow$	Ran. $\uparrow$	Pop. $\uparrow$	Adv. $\uparrow$	Exis. $\uparrow$	Attr. $\uparrow$	Rel. $\uparrow$
OmniLMM	12B	89.3	87.8	87.1	95.1	93.2	93.1	85.6	94.2	80.7
+RLAIF-V	12B	89.0 0.3	87.5 0.3	86.8 0.3	95.0 0.1	92.8 0.4	92.6 0.5	86.1 0.5	90.2 4.0	85.7 5.0
LLaVA-1.6 [ 26 ]	7B	89.7	88.4	86.6	93.9	92.1	91.0	82.0	93.6	58.7
+FINER-Tuning	7B	90.4 0.7	88.8 0.4	87.2 0.6	94.9 1.0	92.9 0.8	91.8 0.8	83.5 1.5	92.6 1.0	78.8 20.1
Qwen2.5-VL [ 4 ]	7B	87.0	86.5	85.8	93.6	91.9	91.7	84.1	95.7	75.6

Table 13: Per-subset results on POPE [ 22 ] , RePOPE [ 33 ] , and AMBER [ 44 ] . Rand.: Random; Pop.: Popular; Adv.: Adversarial; Exis.: Existence; Attr.: Attribute; Rel.: Relation

	HallBench			CRPE_R				HaloQuest
Models	aAcc. $\uparrow$	fAcc. $\uparrow$	qAcc. $\uparrow$	Sub. $\uparrow$	Pred. $\uparrow$	Obj. $\uparrow$	Tot. $\uparrow$	VC. $\uparrow$	IC. $\uparrow$	FP. $\uparrow$
LLaVA-1.6-7B	33.0	10.6	8.3	61.7	52.6	61.6	56.5	50.5	38.0	42.9
+FINER-Tuning	36.3 3.3	17.4 6.8	13.0 4.7	62.6 0.9	51.7 0.9	59.8 1.8	56.0 0.5	50.5	57.0 19.0	73.9 31.0
Qwen2.5-VL-7B	65.4	35.8	40.0	77.2	66.1	71.7	69.9	66.5	76.0	79.2
+FINER-Tuning	68.5 3.1	40.0 4.2	43.6 3.6	77.9 0.7	67.0 0.9	72.4 0.7	70.7 0.8	65.9 0.6	86.7 10.7	87.5 8.3
InternVL-3.5-8B	71.0	45.1	47.0	75.6	63.3	70.8	67.7	66.5	51.2	64.4

Table 14: Per-subset results on HallBench [ 16 ] , CRPE relation subset (CRPE_R) [ 46 ] , and HaloQuest [ 47 ] . Sub.: Subject; Pred.: Predicate; Obj.:Object; Tot.: Total; VC.::Visually Challenge subset; IC.: Insufficient Context subset; FP.: False Premise subset;

	AMBER_G
Models	CHAIR $\downarrow$	COVER $\uparrow$	Hal $\downarrow$	Cog $\downarrow$
Qwen2.5-VL-7B	5.3	64.0	27.1	1.9
+FINER-Tuning	5.0 0.3	64.7 0.7	25.9 1.2	1.6 0.3
InternVL-3.5-8B	6.9	61.3	49.9	3.1
+FINER-Tuning	6.3 0.6	61.4 0.1	47.0 2.9	2.5 0.6
InternVL-3.5-14B	7.9	68.6	57.6	5.4

Table 15: Extended results on AMBER generative subset (AMBER_G).

Method	POPE	AMBER	MMHal	HaloQuest
	Acc. $\uparrow$	Acc. $\uparrow$	HR. $\downarrow$	Score $\uparrow$
LLaVA-1.5-7B	85.9	74.7	54.0	22.6
+HALVA [ 38 ]	84.8	83.4	54.0	23.9
+HA-DPO [ 57 ]	86.9	78.1	60.0	-
+DoLA [ 9 ]	85.7	74.5	56.0	22.9
+RLAIF-V [ 55 ]	85.2	76.8	32.3	-

Table 16: Extended comparison with other hallucination reduction methods on LLaVA-1.5-7B [ 25 ] . HR.: Hallucination rate. The best results are bold while the second best results are underlined.

Figure 12: Left: MCQ version of the motivational study. Right: False-positive (FP) and false-negative (FN) rates at each granularity level.

FINER-CompreCap				FINER-DOCCI
Multi-obj	Multi-attr	Multi-rel	Wh	Multi-obj	Multi-attr	Multi-rel	Wh
92.5	92.5	97.5	95.0	92.5	95.0	90.0	90.0

Table 17: Human performance in paired accuracy (

\text{Acc}_{\text{paired}}

) on FINER-CompreCap and FINER-DOCCI.

Figure 13: Success & failure analysis matrix for InternVL3.5-14B [ 45 ] (denoted as “model” in the figure) and Human. All MCQs are included in the human study.

Figure 14: Examples of our human study survey for FINER-CompreCap . Example questions from Multi-rel and Wh are shown in the figure. Ticked boxes represent ground-truth choices. We use blue color to represent the questions for version A, while orange representing the questions for version B.

Figure 15: Examples of our human study survey for FINER-docci . Example questions from Multi-attr and Multi-obj. Ticked boxes represent the ground-truth choice. We use blue color to represent the questions for version A, while orange representing the questions for version B.

Figure 16: Prompt Template for extracting

\Psi_{\textsc{Obj}}^{+}

Figure 17: Prompt Template for extracting

\Psi_{\textsc{Attr}}^{+}

Figure 18: Prompt Template for extracting

\Psi_{\textsc{Rel}}^{+}

Figure 19: Prompt Template for generating

(q^{+},a^{+}_{+})

for WH setting

Figure 20: Prompt Template for generating

\Psi_{\textsc{Obj}}^{-}

Figure 21: Prompt Template for generating

\Psi_{\textsc{Attr}}^{-}

Figure 22: Prompt Template for generating

\Psi_{\textsc{Rel}}^{-}

Figure 23: Prompt Template for generating

(q^{-},a^{+}_{-})

for WH setting

Figure 24: Prompt Template for extracting objects and attributes using Gemini-2.0-Flash [ 41 ] when constructing FINER-DOCCI .

Figure 25: Prompt Template for extracting relations using Gemini-2.0-Flash [ 41 ] when constructing FINER-DOCCI .

Explore similar work

Mar 20, 2026

FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMs

Multimodal Large Language Models (MLLMs) suffer from hallucinations. Existing hallucination evaluation benchmarks are often limited by over-simplified tasks leading to saturated metrics, or insufficient diversity that fails to adequately assess the hallucination extent in state-of-the-art multimodal models. To address this gap, we propose FREAK, a comprehensive multimodal benchmark designed for fine-grained hallucination assessment in MLLMs. Through high-quality photorealistic images featuring fine-grained counter-commonsense edits, FREAK innovatively evaluates hallucination phenomena in detailed visual perception of MLLMs. Extensive experiments on FREAK show severe hallucination issues in SOTA models regarding detailed visual perception. To enable deeper investigation, we curate a controlled subset to indirectly evaluate the model's ability to perceive target detailed information. Through systematic evaluation of prevailing Chain-of-Thought (CoT) prompting techniques within this task, we reveal critical insights regarding hallucination patterns and model reasoning processes.

Zhihan Yin, Jianxin Liang, Yueqian Wang +3

Wangxuan Institute of Computer Technology, Peking University · State Key Laboratory of General Artificial Intelligence

VLM Hallucination VLM Evaluation Counterfactual Evaluation of VLMs

Ask PDF

Jul 23, 2026

HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models

Although Multimodal Large Language Models have achieved strong performance across a wide range of vision-language tasks, they still suffer from hallucinations, where model outputs become inconsistent with the visual content, textual context, or commonsense knowledge. Existing studies primarily address this problem through coarse-grained detection. However, these approaches often provide insufficient diagnostic information for understanding hallucination types and supporting downstream hallucination mitigation. To bridge this gap, we propose fine-grained hallucination diagnosis for MLLMs, a new unified task that jointly performs hallucination detection, classification, and interpretable explanation generation. We develop an automated data generation pipeline and construct HalluScope-30K, a large-scale diagnostic dataset covering eight sources and five task categories. Based on this dataset, we design a multi-granular joint reward function and train two diagnosis models, HalluScope-4B and HalluScope-8B, which achieve state-of-the-art performance on both the MHALO benchmark and our fine-grained hallucination classification benchmark. Notably, detection and classification are mutually beneficial under joint optimization. Furthermore, diagnosis-driven feedback experiments show that the fine-grained diagnostic explanations produced by our model effectively guide target models to correct their hallucinations, with full diagnosis substantially outperforming all baselines on both Qwen3-VL-8B-Instruct and LLaVA-1.5-7B. Our code, data, and models are available at https://github.com/wkinglin/HalluScope.

Weilin Jin, Mingyu Wang, Wenbo Li +5

Peking University Beijing, China · Joy Future Academy Beijing, China

Hallucination Detection in VLMs Multimodal Hallucination LLM Hallucination Detection Multimodal Model Interpretability VLM Hallucination Mitigation

Ask PDF

Aug 31, 2026

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM

Existing research on object hallucination in multimodal large language models (MLLMs) predominantly attributes the problem to language priors such as over-reliance on textual co-occurrence statistics. We challenge this view by presenting quantitative evidence for a complementary, under-explored cause: visual-origin hallucination, where hallucinations arise from incorrect visual feature extraction and misalignment between image and text embeddings. Through cosine similarity analysis and Smooth Grad-CAM entropy measurements, we show that hallucinated samples exhibit systematically lower image-text similarity (average 0.158 vs. -0.122) and inverted attention patterns, where attention is dispersed when the target object is present but wrongly concentrated when it is absent. Guided by this diagnosis, we propose Adversarial Contrastive Fine-Tuning (ACFT). ACFT uses an Adversarial Hallucination Attribute Flipping (AHAF) procedure, involving minimal, targeted adversarial perturbations that flip an image's hallucination attribute, to construct perfectly aligned positive-negative pairs, which are then used for contrastive fine-tuning. AHAF simultaneously serves as a diagnostic probe, revealing that MLLM visual representations lie dangerously close to hallucination decision boundaries. Requiring only 0.9% of the COCO dataset and adding zero inference overhead, ACFT achieves state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL. Code is available at https://github.com/zxp555/ACFT_MM

Peiyang Xu, Xiaopei Zhu, Jun Zhu +1

Tsinghua University Beijing, China

Object Hallucination in VLMs Multimodal Contrastive Learning VLM Hallucination Mitigation

Ask PDF