Fine-Grained Generative Hallucination Detection and Correction for Aerial Image Captioning
Organizations: IIT Hyderabad, India · IIT Madras, India
Abstract
Aerial imagery captured by unmanned aerial vehicles (UAVs) plays a vital role in applications such as surveillance, disaster response, and large-scale environmental monitoring. Automatic generation of textual descriptions for such imagery is essential for reducing manual inspection overhead and enabling better downstream decision-making. However, Multimodal Large Language Models (MLLMs), despite their impressive captioning capabilities, are prone to hallucination, producing descriptions that are inconsistent with the visual input. This problem is significantly amplified in aerial imagery due to its unique challenges such as extreme viewpoint variation, small objects, class imbalance, and ambiguous visual entities. Existing hallucination detection approaches either rely on computationally expensive large external models such as GPT-4, which perform poorly on aerial domains, or treat the problem as a rigid binary classification task, ignoring the inherent uncertainty in ambiguous cases. To address these limitations, we propose a lightweight, reference-free framework for hallucination detection and correction in free-form aerial image captioning. Our method leverages state-of-the-art MLLMs, specifically InternVL3 and Qwen2.5-VL, to detect hallucinated content in generated captions with minimal computational overhead. Detected hallucinations are corrected through a dedicated refinement module that improves the factual consistency of the original captions. Extensive experiments on the Labelled Illusion Dataset (LID) demonstrate that our framework significantly reduces hallucination rates while remaining practical for real-time deployment in resource-constrained environments. Code is available at: https://github.com/Debolena7/Hal_Detect.
Figures & tables
| Model | Precision | Recall | F1 | Time(s) |
|---|---|---|---|---|
| All-None Baseline | 49.11 | 38.27 | 43.01 | - |
| Zero-shot Baselines | ||||
| Internvl3-1B | 46.78 | 37.25 | 41.48 | 0.0753 |
| Internvl3-2B | 49.11 | 38.27 | 43.01 | 0.0796 |
| Internvl3-8B | 55.61 | 45.16 | 49.84 | 0.2934 |
| Qwen2.5VL-3B | 33.55 | 48.83 | 39.77 | 0.5451 |
| Model | ALOHa | CHAIRi | CHAIRs | CLIPScore | RefCLIP | BERT (Deberta) | BERT (D+IDF) | Time (s) |
|---|---|---|---|---|---|---|---|---|
| Raw Descriptions | 41.27 | 7.04 | 47.89 | 73.93 | 73.09 | 66.47 | 60.39 | - |
| Original descriptions + GT hallucinated words + instruction prompt | ||||||||
| GPT-5.2 (text-only) | 43.50 | 0.79 | 5.76 | 73.88 | 73.13 | 67.32 | 61.09 | 0.8152 |
| GPT-5.2 (text + image) | 41.64 | 1.32 | 11.69 | 74.32 | 73.49 | 66.80 | 60.74 | 2.1589 |
| Original descriptions + predicted hallucinated words + instruction prompt | ||||||||
| GPT-5.2 | 43.27 | 2.67 | 19.07 | 73.93 | 73.17 | 67.29 | 61.08 | 0.8219 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Precision | Recall | F1 |
|---|---|---|---|
| Zero-shot baselines | |||
| InternVL3-1B | 2.33 | 1.54 | 1.85 |
| InternVL3-2B | 0 | 0 | 0 |
| InternVL3-8B | 17.70 | 12.30 | 14.51 |
| Qwen2.5VL-3B | 39.14 | 57.79 | 46.67 |
| Finetuned Models | |||
| Model | Precision | Recall | F1 |
|---|---|---|---|
| HalLocalizer [ 20 ] | 76.46 | 71.97 | 74.14 |
| InternVL3-1B (ours) | 80.51 | 69.69 | 74.71 |
| InternVL3-2B (ours) | 86.53 | 80.35 | 83.33 |
| InternVL3-8B (ours) | 89.41 | 83.29 | 86.24 |
Hallucination Detection in VLMs
Hallucination detection in vision-language models identifies claims that are unsupported by images, video, or other visual evidence.
See all 77 papers