cs.CLNov 13, 2026

VEPO: Mitigating Multilingual Hallucinations in LVLMs via Visual Evidence-guided Preference Optimization

Authors: Chengpeng Fu, Xiaocheng Feng, Wenshuai Huo, Yichong Huang, Baohang Li, Duyu Tang, Ting Liu

Organizations: Harbin Institute of Technology, 92 Xidazhi Street, Nangang District, Harbin, Heilongjiang Province, China · Peng Cheng Laboratory, No.6001 Shahu West Road, Nanshan District, Shenzhen, Guangdong Province, China · Zhongguancun Academy, Beijing, China

Abstract

Large vision-language models (LVLMs) achieve strong multimodal performance, yet they remain prone to object hallucination---generating answers that are linguistically fluent but visually ungrounded. This issue is especially severe in low-resource languages, where weaker cross-modal alignment causes models to rely more on language priors than on image evidence. Existing remedies either require costly multilingual annotations or external translation/reward models, or intervene at inference time in ways that can distort the output language. We propose VEPO(Visual Evidence-guided Preference Optimization), a training-time method that transfers reliable visual grounding from English to other languages without external models. VEPO first localizes key visual evidence using English queries via mid-layer cross-modal attention, then constructs preference pairs that share the same correct answer but differ in evidence sufficiency (full image vs.\ key-region masked image). Direct preference optimization on these pairs encourages low-resource inputs to prefer answers supported by complete visual evidence. Experiments on multilingual POPE and MME with Chinese, French, and Hindi show consistent hallucination reduction and perception gains, with the largest improvements on the lowest-resource language. Analyses further confirm that VEPO sharpens question-relevant attention and that evidence-guided masking outperforms random masking.

Figures & tables

Explore similar work

CardsList
  1. Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation

    May 24, 2026Yangneng Chen, Jing LiVision-Language ModelsVLM Adaptation

  2. FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models

    Jun 28, 2026Yichen Guo, Kai Tang, Fenglai Lin +5Vision-Language ModelsLarge Vision-Language Models

  3. NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors

    Feb 25, 2026Lingfeng Ren, Weihao Yu, Runpeng Yu +1Object Hallucination in VLMsVLM Hallucination Mitigation