cs.CVNov 19, 2026

What Does Anti-Hallucination Training Buy? Saying Less, Cheap Pruning, and the Rest

Authors: Lingkai Bu, Qian Gao

Organizations: Key Laboratory of Computing Power Network and Information Security, Ministry of Education Shandong Computer Science Center (National Supercomputer Center in Jinan) Qilu University of Technology (Shandong Academy of Sciences), Jinan, China

Abstract

Preference training is the standard recipe for reducing object hallucination in vision-language models, and each new method reports a lower CHAIR score. We ask what that reduction consists of. Every trained system is compared with its base model's caption truncated to the same object recall, which removes the part of the gain that comes from saying less; the remainder is compared with what a deliberately weak, frozen, eleven-feature post-hoc probe fitted on 500 annotated images can reproduce. All criteria were specified in the analysis code and frozen before the outputs they judge existed. For eight public methods on LLaVA-1.5-7B, the probe reproduces at least two thirds of what training removes beyond saying less for 7 of 8 methods on COCO, 8 of 8 on AMBER and 6 of 8 on Object HalBench, with RLAIF-V the consistent exception; a later public method released for both LLaVA-1.5 and Qwen2.5-VL (SENTINEL) meets the same bar on both. On our own DPO checkpoints the accounting holds on Qwen2.5-VL but fails on InternVL2.5 on COCO (2 of 4 training designs). At matched recall, the base model plus the probe beats truncation for 8 of 9 LLaVA-1.5 systems on COCO and 9 of 9 on AMBER, on the two newer backbones, and when realized as actual sentence deletion. The same measure prices training signals: DPO on the model's own samples labelled by the probe or by an object-level CLIP score recovers 0.70 and 0.87 of the beyond-brevity gain of ground-truth-labelled DPO on Qwen2.5-VL (the probe 0.73 on InternVL2.5), both beat length-matched random labels, and CLIP labels buy more than probe labels on COCO. Human judgements show that the sentences the probe deletes are wrong more often than those truncation would delete; a human check of the CHAIR evaluator's precision falls below our pre-set bar, which we report as a limitation.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Targeting the Attention Heads Behind Object Hallucination in LLaVA

    Aug 25, 2026Armaan Sandhu, Abhilasha Senapati, Hima KammachiObject Hallucination in VLMsVLM Hallucination Mitigation

  2. HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

    Apr 7, 2026Reihaneh Zohrabi, Hosein Hasani, Akshita Gupta +3Hallucination Detection in VLMsObject Hallucination in VLMs

  3. Beyond Seeing: Fine-Grained Preference Validation and Severity-Aware Alignment for Mitigating Visual Hallucinations in Vision-Language Models

    Nov 16, 2026Abdulaziz Alqahtani, Alwaleed Alharthi, Muzammil BehzadVLM Hallucination MitigationObject Hallucination in VLMs