RelCheck: Dual-Evidence Spatial Grounding for VLM Hallucination Correction

Siddhi Patil; Navrati Saxena; William B. Andreopoulos

cs.CVAug 21, 2026

RelCheck: Dual-Evidence Spatial Grounding for VLM Hallucination Correction

Authors: Siddhi Patil, Navrati Saxena, William B. Andreopoulos

Organizations: Dept. of Computer Science San Jos´e State University

Abstract

Multimodal large language models (MLLMs) fre- quently generate text that is inconsistent with the input image. While object- and attribute-level hallucinations have received considerable attention, relational hallucinations (incorrect de- scriptions of spatial or interactive relationships between objects) remain largely unaddressed by existing post-hoc correction methods. We present RelCheck, a training-free post-hoc correction pipeline that augments object-level visual grounding with dual relational evidence: learned scene-graph triples from RelTR and deterministic spatial predicates from bounding-box geometry. These combine with a Woodpecker-style object claim layer to form a three-layer visual knowledge base, which a language model corrector uses to rewrite hallucinated text. Evaluated on LLaVA v1 13B, RelCheck achieves a total MME hallucination score of 630.0 versus 585.0 for a Woodpecker-style baseline, with the largest gain on the position subtask (+31.7 points, accuracy+ improving from 0.367 to 0.600). A four-configuration ablation confirms that both relational layers contribute independently (McNemar p = 0.025). These results show that structured relational evidence meaningfully improves post-hoc hallucination correction on the spatial reasoning subtasks where current MLLMs are most deficient.

Figures & tables

Fig. 1: RelCheck 3-layer visual knowledge base architecture. Stages 1–4 (shared with Woodpecker) extract entities, generate questions, validate with GroundingDINO and VQA, and produce the Claim layer. RelCheck extends this with two novel layers: the Scene layer from RelTR scene graph generation and the Geom layer from deterministic bounding-box geometry. All three layers feed the KB-aware LLM corrector to produce the grounded output $\hat{y}$ .

Fig. 2: Representative geometric predicates from bounding-box coordinates. (a) left-of: centroid of

B_{1}

left of

B_{2}

beyond deadzone

T

. (b) above: centroid of

B_{1}

above

B_{2}

. (c) on/inside:

B_{1}\subseteq B_{2}

within 5% tolerance.

Fig. 3: Qualitative example: MME position subtask (image 204871). The MLLM responds “Yes, the car is on the left side of the fire hydrant” (GT: no). The GEOM layer computes car right-of fire hydrant from bounding-box centroids. RelCheck corrects to “No.” Woodpecker, lacking geometric evidence, passes the hallucination through unchanged.

Component	Raw	WP	RC
GroundingDINO detection	–	✓	✓
Attribute VQA	–	✓	✓
QA-to-claim (T5)	–	✓	✓
Scene graph (RelTR)	–	–	✓
Geometric spatial layer	–	–	✓
KB layers	–	Claim	Claim+Scene+Geom

TABLE I: System configurations. All share LLaVA v1 13B and Stages 1–4.

Component	Parameter	Value
GroundingDINO	Box confidence $\tau_{\text{obj}}$	0.35
GroundingDINO	Text-matching threshold	0.25
RelTR	Triple confidence $\tau_{\text{rel}}$	0.30
Geometry	Spatial deadzone $T$	0.08
Geometry	Containment tolerance	5%
Geometry	Max detections/image	20

TABLE II: RelCheck hyperparameters (fixed across all experiments).

Split	System	Acc.	Prec.	Recall	F1
Random	RawMLLM	0.723	0.647	0.982	0.780
	Woodpecker	0.872	0.943	0.792	0.861
	RelCheck	0.878	0.952	0.796	0.867
Popular	RawMLLM	0.678	0.611	0.982	0.753
	Woodpecker	0.838	0.872	0.792	0.830
	RelCheck	0.852	0.896	0.796	0.843

TABLE III: POPE results (LLaVA v1 13B, 500 questions/split). Best per column bolded.

Fig. 4: Radar chart of MME hallucination scores. RelCheck (green) traces the outer envelope on all four axes; the gap vs. Woodpecker is most pronounced on position.

Subtask	System	Acc.	Acc. +	Score
Existence	RawMLLM	0.767	0.533	130.0
	Woodpecker	0.950	0.900	185.0
	RelCheck	0.983	0.967	195.0
Count	RawMLLM	0.517	0.067	58.3
	Woodpecker	0.767	0.700	146.7
	RelCheck	0.767	0.700	146.7

TABLE IV: MME hallucination subset results. Score

=(\text{Acc.}+\text{Acc.}^{+})\times 100

, max 200. Best bolded.

Fig. 5: Action-relation correction. The MLLM describes “multiple people walking down the street,” but the Scene layer (RelTR) detects sitting relations and the Claim layer confirms the setting as a café sidewalk via VQA. RelCheck corrects to “multiple people sitting at a café’s sidewalk,” replacing the hallucinated action predicate with the grounded one.

Configuration	Exist.	Count	Pos.	Total
Woodpecker	185.0	146.7	103.3	585.0
ClaimOnly	195.0	146.7	105.0	580.0
+GEOM	195.0	146.7	121.7	618.3
+GEOM+SCENE (Full)	195.0	146.7	135.0	630.0

TABLE V: Ablation on MME hallucination subset (scores

=(\text{Acc}+\text{Acc}^{+})\times 100

Subtask	CLAIM	GEOM	SCENE	Gain
Existence	58	0 1	58	$+10.0$
Count	60	0 0	60	$\phantom{+}0.0$
Position	60	52	58	$+31.7$
Color	59	0 4	60	$+3.3$

TABLE VI: Layer activation: images (out of 60) for which each KB layer produces non-empty evidence per MME subtask.

Emotional Hallucination in VLMs

See all 8 papers

Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs
Aug 5, 2026Yang Yang, Jiawei Chen, Tairan Chen +1Multimodal Grounding Visual Spatial Reasoning
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy · Stevens Institute of Technology
Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM
Aug 31, 2026Peiyang Xu, Xiaopei Zhu, Jun Zhu +1Object Hallucination in VLMs Multimodal Contrastive Learning
Tsinghua University Beijing, China
Geo3R: Mitigating Spatial Reasoning Hallucination in Multimodal Large Language Models
Jul 23, 2026Mingyu Wang, Weilin Jin, Wenbo Li +33D Spatial Reasoning Hallucination in Language Models
Peking University Beijing, China · Joy Future Academy Beijing, China

Aug 5, 2026J/K move · Enter open · S save

Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs

Yang Yang, Jiawei Chen, Tairan Chen, Zhaoxia Yin

Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy · Stevens Institute of Technology

Multimodal Grounding Visual Spatial Reasoning Multimodal Large Language Models Multimodal CoT Reasoning Spatial Reasoning in Language Models Multimodal Spatial Reasoning

Open paper AskPDF arXiv

Although Multimodal Large Language Models (MLLMs) have made substantial progress, their spatial reasoning may still produce intermediate judgments inconsistent with the input image, allowing errors to propagate through the reasoning chain and affect the final answer. Existing methods mainly improve spatial reasoning through training or additional spatial information, without considering whether the reasoning process itself is faithful to the model input. Our study shows that unfaithful reasoning chains significantly reduce final-answer accuracy. To address this issue, we propose a modular and training-free framework for spatial reasoning verification and correction. The framework constructs a Spatial Evidence Graph (SEG), which associates atomic spatial evidence extracted from Chain-of-Thought reasoning with visual entities, spatial relations, source steps, and visual evidence. Spatial Evidence Reliability Assessment (SERA) evaluates the reliability of visual evidence based on object existence, localization, and geometric measurements. The framework then identifies the earliest spatial evidence unit contradicted by reliable visual evidence and guides the original MLLM to revise the subsequent reasoning and final answer. Across 15 model-dataset settings, our method achieves an average accuracy of 68.94%, outperforming the compared baselines by 8.55 percentage points on average. Our code will be open-sourced.

Abstract

Emotional Hallucination in VLMs

Explore similar work

Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM

Geo3R: Mitigating Spatial Reasoning Hallucination in Multimodal Large Language Models