Seeing Clearly without Training: Mitigating Hallucinations in Multimodal LLMs for Remote Sensing

Yi Liu; Jing Zhang; Di Wang; Xiaoyu Tian; Haonan Guo; Bo Du

cs.CVMar 3, 2026

Seeing Clearly without Training: Mitigating Hallucinations in Multimodal LLMs for Remote Sensing

Authors: Yi Liu, Jing Zhang, Di Wang, Xiaoyu Tian, Haonan Guo, Bo Du

Organizations: School of Computer Science, Wuhan University, China · Zhongguancun Academy, China · School of Computer Science, Chongqing University, China

Abstract

Multimodal large language models (MLLMs) suffer from pronounced hallucinations in remote sensing visual question-answering (RS-VQA), primarily caused by visual grounding failures in large-scale scenes or misinterpretation of fine-grained small targets. To systematically analyze these issues, we introduce RSHBench, a protocol-based benchmark for fine-grained diagnosis of factual and logical hallucinations. To mitigate grounding-induced factual hallucinations, we further propose Relative Attention-Driven Actively Reasoning (RADAR), a training-free inference method that leverages intrinsic attention in MLLMs to guide progressive localization and fine-grained local reasoning at test time. Extensive experiments across diverse MLLMs demonstrate that RADAR consistently improves RS-VQA performance and reduces both factual and logical hallucinations. Code and data will be publicly available at: https://github.com/MiliLab/RADAR

Figures & tables

Figure 1: Two grounding failures underlying RS-VQA hallucinations. Type 1 (Cannot find): model attention becomes diffuse and is distracted by irrelevant regions, resulting in missed target localization. Type 2 (Cannot see clearly): attention covers the correct region, but the visual evidence is too small or ambiguous for fine-grained recognition, leading to incorrect predictions.

Figure 2: Query-Conditioned Relative Attention (QCRA). Given an input image and two prompts: a task-focused query $Q$ (top) and a global-comprehension query $Q^{G}$ (bottom), we derive layer-wise attention relevance maps. At each layer, token-level attention is reshaped to the image grid and a relative attention matrix is computed by contrasting the task map against the global map to suppress query-irrelevant saliency. The most informative layers (top- $k$ ) are then aggregated to produce a final query-conditioned attention heatmap, which serves as the grounding signal for region selection and multi-scale evidence construction.

Judge	Accuracy	Cohen’s $\kappa$	MCC
Gemini-3-pro	0.7882	0.5726	0.5770
GPT-5.2	0.9288	0.8553	0.8591
Qwen3-max	0.9045	0.8058	0.8070

Table 1: Leave-one-out (LOO) agreement of expert judges for the binary hallucination decision. We pool all evaluated base models and compare each judge against an LOO pseudo-label defined only when the other two judges agree. We report Accuracy, Cohen’s

\kappa

, and MCC .

Closed-source Models
Models	Factual Hallucination				Logical Hallucination					HR
Models	OBJ	ATT	SPA	$HR_{F}$	IR	CI	INC	SO	$HR_{L}$
Claude-3-7	40.70	28.30	11.32	55.53	20.49	0.27	1.08	14.29	24.53	56.33
Gemini-2.5-pro	33.42	31.54	15.36	48.79	27.49	0.54	0.81	15.09	29.65	49.06
GPT-4o	30.19	26.68	11.32	46.90	19.41	0.27	0.27	11.05	21.02	47.44
Open-source Models

Table 2: Hallucination evaluation on RSHBench. Each model prediction is annotated by three expert judges with a binary hallucination rate (

HR

) and a multi-label taxonomy comprising factual subtypes ( OBJ , ATT , SPA ) and logical subtypes ( IR , CI , INC , SO ), along with coarse categories

HR_{F}

(Factual) and

HR_{L}

(Logical). The overall

HR

is determined by majority vote, while category and subtype labels are aggregated by taking the union of tags from judges who marked the instance as hallucinated. All values are reported as percentages over the full evaluation set .

Methods	LRS-VQA				MME-RealWorld-RS				LHRS-Bench	Avg.
Methods	FAIR	Bridge	STAR	AA	Position	Color	Count	AA	–	Avg.
Llama-4-scout	19.72	29.19	23.73	24.21	26.70	23.72	15.64	22.02	37.33	27.85
Claude-haiku-4-5	5.68	9.79	2.40	5.96	31.73	28.04	18.35	26.04	55.01	29.00
GLM-4.6v	27.55	27.31	29.31	28.06	44.92	28.19	7.53	26.88	67.19	40.71
GPT-4o	22.89	24.39	29.78	25.69	36.37	32.43	15.89	28.23	66.19	40.04
Qwen2.5-VL-7B	22.18	27.40	26.71	25.43	46.22	29.16	11.26	28.88	53.98	36.10

Table 3: Overall performance on remote sensing VQA and real-world reasoning. We report accuracy (%) on LRS-VQA (FAIR, Bridge, STAR, and their average accuracy (AA)), MME-RealWorld-RS (Position, Color, Count, and AA), and LHRS-Bench. Avg. denotes the mean score across the three benchmarks (adopting AA for LRS-VQA and MME-RealWorld-RS). RADAR indicates the application of our training-free inference procedure to the GeoZero backbone (see Table 4 for per-backbone gains), achieving the highest average performance among all evaluated methods.

Method	LRS-VQA			MME-RealWorld-RS			LHRS-Bench
Method	FAIR $\uparrow$	Bridge $\uparrow$	STAR $\uparrow$	Position $\uparrow$	Color $\uparrow$	Count $\uparrow$	–
LLaVA-OV	–	–	–	18.70	14.26	17.37	19.66
+ RADAR (Ours)	–	–	–	22.83 (+4.14)	32.83 (+18.57)	29.53 (+12.15)	23.03 (+3.37)
Qwen3-VL	26.23	29.47	32.86	55.53	40.24	7.59	65.35
+ ViCrop	27.46	28.91	33.31	54.57	42.87	10.60	63.51
+ RADAR (Ours)	30.77 (+4.54)	29.94 (+0.47)	34.98 (+2.13)	56.64 (+1.11)	53.71 (+13.47)	15.01 (+7.42)	67.73 (+2.38)

Table 4: Effect of evidence cropping and progressive localization. We compare each backbone model with a generic cropping baseline ( +ViCrop ) and our ( +RADAR ). Reported values are accuracy (%) on each subset, and red numbers indicate absolute improvements over the corresponding backbone. ViCrop yields limited and often unstable gains, whereas RADAR consistently improves performance, particularly on attribute recognition and counting tasks that require fine-grained visual evidence.

Figure 3: Qualitative examples of RADAR’s QCRA and progressive evidence refinement. For each example, we visualize the QCRA heatmap generated by a where -oriented query on the full image (top) and a what -oriented query on the localized crop (middle), where brighter regions indicate stronger query-conditioned relevance. Dashed boxes mark the regions selected for zoom-in evidence extraction.

Configuration	MME-RealWorld	LHRS-Bench
Baseline	34.45	65.36
RADAR w/o Stage 2	39.05	66.17
RADAR w/o Stage 1	38.88	66.87
RADAR	41.79	67.73

Table 5: Ablation study on the operation stages of RADAR, conducted with the Qwen3-VL-4B model. We report accuracy on MME-RealWorld-RS and LHRS-Bench.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Code	Hallucination Types	Definition
OBJ	Object/Category	Claims the presence/absence of an object or category not supported by the image.
ATT	Attribute	Misstates attributes (e.g., color, shape, size, count, material) not supported by the image.
SPA	Spatial/Relational	Misstates spatial relations (e.g., relative position, containment, adjacency) between objects/regions.
IR	Invalid Reasoning	Uses incorrect premises, unsupported assumptions, or invalid inference steps.
CI	Unjustified Causality	Infers causality, intent, or functionality from correlation without evidence.
INC	Internal Inconsistency	The reasoning process $C^{\prime}$ conflicts with the final answer $A^{\prime}$ .

Appendix

Table 6: Hallucination taxonomy. Multiple labels may apply to the same prediction.

Rater	Accuracy	$\kappa$	MCC
Gemini-3-pro	0.8113	0.6225	0.6259
GPT-5.2	0.9476	0.8951	0.8972
Qwen3-max	0.9314	0.8628	0.8632

Appendix

Table 7: Agreement between each judge and the three-judge majority vote. We report Accuracy, Cohen’s

\kappa

, and MCC.

Rater	Accuracy	$\kappa$	MCC
Gemini-3-pro	0.6700	0.3159	0.3165
GPT-5.2	0.9500	0.8934	0.8953
Qwen3-max	0.7500	0.5000	0.5126

Appendix

Table 8: Agreement between expert judges and human annotations. We report Accuracy, Cohen’s

\kappa

, and MCC.

Figure 4: Example from human calibration for hallucination labeling.

Figure 5: Example from human calibration for hallucination labeling.

Judge: Gemini-3-pro
Models	$HR$	Factual Hallucination				Logical Hallucination
Models	$HR$	OBJ	ATT	SPA	$HR_{F}$	IR	CI	INC	SO	$HR_{L}$
Closed-source Models
Claude-3-7	56.33	55.53	9.43	34.50	23.45	7.82	7.28	0.27	0.81	0.27
Gemini-2.5-pro	45.28	44.20	9.70	18.06	23.18	11.86	7.55	0.00	0.54	0.27
GPT-4o	45.55	43.40	7.82	16.98	22.37	8.89	5.66	0.27	0.54	1.08

Appendix

Table 9: Per-judge hallucination taxonomy statistics across three judges. All values are percentages. Subtype rates are multi-label.

Closed-source Models
Models	Factual Hallucination				Logical Hallucination					HR
Models	OBJ	ATT	SPA	$HR_{F}$	IR	CI	INC	SO	$HR_{L}$
Claude-3-7	49.33	32.61	16.98	60.11	19.95	0.00	0.81	15.09	24.80	60.11
Gemini-2.5-pro	39.08	36.39	19.14	52.83	29.92	0.54	0.54	15.63	33.96	53.64
GPT-4o	34.23	26.68	13.21	47.98	19.95	0.00	0.54	12.67	22.37	48.79
Open-source Models

Appendix

Table 10: Hallucination evaluation on RSHBench (ROI-prioritized setting). Each model prediction is annotated by three expert judges with a binary hallucination rate (

HR

) and a multi-label taxonomy comprising factual subtypes ( OBJ , ATT , SPA ) and logical subtypes ( IR , CI , INC , SO ), along with coarse categories

HR_{F}

(Factual) and

HR_{L}

(Logical). The overall

HR

Closed-source Models
Models	$HR$	Factual Hallucination				Logical Hallucination
Models	$HR$	OBJ	ATT	SPA	$HR_{F}$	IR	CI	INC	SO	$HR_{L}$
Claude-3-7	64.69	48.52	23.99	14.56	64.69	5.93	0.00	1.08	1.08	9.16
Gemini-2.5-pro	61.46	35.85	26.42	20.75	61.19	9.97	0.27	0.00	0.54	12.40
GPT-4o	54.72	31.00	20.75	12.40	53.64	6.47	0.00	0.54	0.54	8.63
Open-source Models

Appendix

Table 11: Per-judge hallucination taxonomy statistics on RSHBench (ROI-prioritized setting). Judge: Gemini-3-pro-preview. All values are percentages. Subtype rates are multi-label.

Closed-source Models
Models	$HR$	Factual Hallucination				Logical Hallucination
Models	$HR$	OBJ	ATT	SPA	$HR_{F}$	IR	CI	INC	SO	$HR_{L}$
Claude-3-7	61.73	34.50	22.91	7.01	58.49	14.82	0.00	0.27	7.82	16.98
Gemini-2.5-pro	57.14	26.95	25.07	12.13	53.91	23.45	0.54	0.54	8.36	24.80
GPT-4o	51.21	24.53	20.49	7.55	48.25	15.63	0.00	0.00	6.47	16.44
Open-source Models

Appendix

Table 12: Per-judge hallucination taxonomy statistics on RSHBench (ROI-prioritized setting). Judge: GPT-5.2. All values are percentages. Subtype rates are multi-label.

Closed-source Models
Models	$HR$	Factual Hallucination				Logical Hallucination
Models	$HR$	OBJ	ATT	SPA	$HR_{F}$	IR	CI	INC	SO	$HR_{L}$
Claude-3-7	52.29	32.08	21.29	6.20	50.67	10.24	0.00	0.00	15.09	17.52
Gemini-2.5-pro	50.13	28.03	22.64	6.20	47.71	11.86	0.00	0.00	12.94	19.14
GPT-4o	50.40	28.84	18.60	5.66	48.25	7.55	0.00	0.00	12.40	15.90
Open-source Models

Appendix

Table 13: Per-judge hallucination taxonomy statistics on RSHBench (ROI-prioritized setting). Judge: Qwen3-max. All values are percentages. Subtype rates are multi-label.

Figure 6: Qualitative examples of RADAR’s query-conditioned relative attention and progressive evidence refinement. For each case, we visualize the QCRA heatmap produced by a where -oriented query on the full image (Stage1) and a what -oriented query on the localized crop (Stage2), where brighter regions indicate stronger question-conditioned relevance. Dashed boxes denote the regions selected for zoom-in evidence extraction.

Remote Sensing VQA

Remote sensing visual question answering (VQA) studies how AI answers natural-language questions about satellite, aerial, and other Earth-observation imagery.

See all 35 papers

Hallucination-aware Faithful Reasoning in Remote Sensing
Oct 13, 2026Yi Liu, Jing Zhang, Xiang Feng +5VLM Hallucination Mitigation Faithfulness in VLMs
School of Computer Science, Wuhan University · Zhongguancun Academy · School of Computer Science, Chongqing University
ReDiS: Reasoning-Diagnosis Self-Evolution for Multimodal Reasoning in Remote Sensing
Oct 13, 2026Yi Liu, Wentao Jiang, Shuo Ni +5Remote Sensing VLMs Multimodal Reasoning
School of Computer Science, Wuhan University · Zhongguancun Academy · National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing, Beijing Institute of Technology
TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering
Oct 7, 2026Jingwen Huang, Yiming Fang, Hanxiao Wu +2Remote Sensing VQA Visual Question Answering
Yunnan University of Finance and Economics, Kunming, China

Oct 13, 2026J/K move · Enter open · S save

Hallucination-aware Faithful Reasoning in Remote Sensing

Yi Liu, Jing Zhang, Xiang Feng, Xiaoyu Tian+4

School of Computer Science, Wuhan University · Zhongguancun Academy · School of Computer Science, Chongqing University

VLM Hallucination Mitigation Faithfulness in VLMs RL for Language Model Reasoning

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have demonstrated promising capabilities for remote sensing reasoning, yet they frequently generate visually unsupported reasoning that contradicts the visual evidence. Existing post-training methods optimize primarily for final-answer correctness, providing limited supervision for identifying and correcting hallucinations that arise during the reasoning process. To address this limitation, we propose FaithRS, a hallucination-aware self-reflection framework for faithful reasoning in remote sensing. We first construct SFHD-43K, a dataset of 43K structured, fine-grained hallucination annotations, and train a detector to identify hallucination types, underlying causes, and supporting visual evidence. FaithRS then incorporates this diagnostic feedback into a Safe Detector-Guided Corrective Rollout framework, where a safety-aware gate selectively determines whether and how to revise the response. Training is driven by a composite reward that jointly optimizes answer accuracy, reasoning faithfulness, hallucination reduction, format compliance, and revision safety. Experiments with Qwen3.5 backbones (0.8B, 4B, and 9B) show consistent improvements across four remote sensing benchmarks, with gains of up to 11.64 points.

Abstract

Remote Sensing VQA

Explore similar work

Hallucination-aware Faithful Reasoning in Remote Sensing

ReDiS: Reasoning-Diagnosis Self-Evolution for Multimodal Reasoning in Remote Sensing

TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering