Large vision-language models (LVLMs) achieve strong multimodal performance, yet they remain prone to object hallucination---generating answers that are linguistically fluent but visually ungrounded. This issue is especially severe in low-resource languages, where weaker cross-modal alignment causes models to rely more on language priors than on image evidence. Existing remedies either require costly multilingual annotations or external translation/reward models, or intervene at inference time in ways that can distort the output language. We propose VEPO(Visual Evidence-guided Preference Optimization), a training-time method that transfers reliable visual grounding from English to other languages without external models. VEPO first localizes key visual evidence using English queries via mid-layer cross-modal attention, then constructs preference pairs that share the same correct answer but differ in evidence sufficiency (full image vs.\ key-region masked image). Direct preference optimization on these pairs encourages low-resource inputs to prefer answers supported by complete visual evidence. Experiments on multilingual POPE and MME with Chinese, French, and Hindi show consistent hallucination reduction and perception gains, with the largest improvements on the lowest-resource language. Analyses further confirm that VEPO sharpens question-relevant attention and that evidence-guided masking outperforms random masking.
Figures & tables
Figure 1: Multilingual performance gap of Qwen-VL-Chat on POPE-COCO: non-English, especially low-resource, languages lag behind English.
Figure 2: Constructing a key-evidence masked image from English attention. Mid-layer attention from the last query token ranks image patches; the top- ρ patches are blacked out to form the rejected image, while the original image remains the chosen image.
Figure 3: Comparison between VEPO (left) and standard multimodal DPO (right). VEPO contrasts evidence-sufficient vs. evidence-missing images under the same correct answer; standard DPO contrasts different answers under the same image. The training examples illustrated in the figure are presented in Chinese, and the semantic content of the questions is consistent with that shown in Figure 2 .
Lang.
Method
COCO
AOKVQA
GQA
Avg.
Random
Popular
Adversarial
Random
Popular
Adversarial
Random
Popular
Adversarial
Original Models
Zh
LLaVA-1.5
0.786
0.764
0.731
0.745
0.711
0.693
0.752
0.702
0.692
0.731
+VEPO
0.818
0.815
0.766
0.783
0.764
0.720
0.787
0.731
0.707
0.766
Fr
LLaVA-1.5
0.765
0.798
0.743
0.744
0.750
0.710
0.744
0.722
0.706
0.742
+VEPO
0.822
0.858
0.784
0.798
0.814
0.745
0.798
0.783
0.743
0.794
Table 1: POPE F1 of VEPO on original and SFT models across Chinese (Zh), French (Fr), and Hindi (Hi). Best results within each language block are in bold.
Method
COCO
AOKVQA
GQA
Avg.
rand.
pop.
adv.
rand.
pop.
adv.
rand.
pop.
adv.
LLaVA-1.5
0.786
0.764
0.731
0.745
0.711
0.693
0.752
0.702
0.692
0.731
+Random Mask
0.793
0.789
0.743
0.753
0.731
0.706
0.767
0.719
0.704
0.745
+VEPO
0.818
0.815
0.766
0.783
0.764
0.720
0.787
0.731
0.707
0.766
Table 2: Chinese POPE: evidence-guided masking vs. random masking under the same occlusion ratio ρ=0.2 .
Method
rand.
pop.
adv.
Avg.
LLaVA-1.5
0.345
0.332
0.337
0.338
VEPO (Hi only)
0.309
0.499
0.409
0.406
VEPO (Hi+En)
0.632
0.669
0.661
0.654
SFT
0.706
0.711
0.709
0.709
SFT+VEPO (Hi only)
0.686
0.724
0.689
0.700
SFT+VEPO (Hi+En)
0.772
0.783
0.752
0.769
Table 3: Hindi COCO-POPE: mixing English preference data stabilizes VEPO.
Figure 4: Overall MME scores before and after VEPO across languages.
Figure 5: Fine-grained MME results: gains are concentrated on perception-oriented subtasks.
Figure 6: Attention change after VEPO: red/blue indicate increased/decreased attention on image patches.
Figure 7: Effect of layer depth and token selection on English-attention masks used for VEPO.
Large Vision-Language Models (LVLMs) extend large language models with visual understanding, but remain vulnerable to hallucination, where outputs are fluent yet inconsistent with images. Recent studies link this issue to language bias-the tendency of LVLMs to over-rely on text while neglecting visual inputs. Yet most analyses remain empirical without uncovering its underlying cause. In this paper, we provide a systematic study of language bias and identify its root in modality misalignment during training. Our analysis shows that both Visual Instruction Tuning (VIT) and Direct Preference Optimization (DPO) often prioritize textual improvements, which may cause LVLMs to overly lean toward language modeling rather than balanced multimodal understanding. To address this, we propose two simple yet effective methods: Language Bias Regularization (LBR) which mitigates language bias through regularization during instruction tuning, and Language Bias Penalty (LBP), which penalizes language bias in the DPO training process. Extensive experiments across diverse models and benchmarks demonstrate the effectiveness of our approach. LBR consistently improves performance on over ten general benchmarks, while LBP significantly reduces hallucination and improves trustworthiness. Together, these methods not only mitigate language bias but also advance the overall alignment of LVLMs, all without introducing any additional data or auxiliary models. Our code is publicly available at https://github.com/lab-klc/LVLM-Language-Bias.
Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.
Yichen Guo, Kai Tang, Fenglai Lin +5
1Nanyang Technological University · 2State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 3Tsinghua University
Object hallucination is a critical issue in Large Vision-Language Models (LVLMs), where outputs include objects that do not appear in the input image. A natural question arises from this phenomenon: Which component of the LVLM pipeline primarily contributes to object hallucinations? The vision encoder to perceive visual information, or the language decoder to generate text responses? In this work, we strive to answer this question through designing a systematic experiment to analyze the roles of the vision encoder and the language decoder in hallucination generation. Our observations reveal that object hallucinations are predominantly associated with the strong priors from the language decoder. Based on this finding, we propose a simple and training-free framework, No-Language-Hallucination Decoding, NoLan, which refines the output distribution by dynamically suppressing language priors, modulated based on the output distribution difference between multimodal and text-only inputs. Experimental results demonstrate that NoLan effectively reduces object hallucinations across various LVLMs on different tasks. For instance, NoLan achieves substantial improvements on POPE, enhancing the accuracy of LLaVA-1.5 7B and Qwen-VL 7B by up to 6.45 and 7.21, respectively. The code is publicly available at: https://github.com/lingfengren/NoLan.
Lingfeng Ren, Weihao Yu, Runpeng Yu +1
National University of Singapore, Singapore · Peking University Shenzhen Graduate School, China