VLM Hallucination Mitigation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

11 papers in the last four weeks, down 15% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 131

All topics

FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
Jun 28, 2026Yichen Guo, Kai Tang, Fenglai Lin +5Vision-Language Models Large Vision-Language Models
1Nanyang Technological University · 2State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 3Tsinghua University
Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding
Jun 25, 2026Liu Yu, Can Chen, Ping Kuang +3Object Hallucination in VLMs Faithfulness in VLMs
School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China · School of Computer Science, University of Auckland, Auckland, New Zealand
Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models
Jun 25, 2026Shravan Venkatraman, Ritesh Thawkar, Omkar Thawakar +4Multimodal Large Language Models VLM Reasoning
Mohamed bin Zayed University of Artificial Intelligence · Aalto University · Australian National University +1
TAVR-VLM: Risk-Conditioned Causal Grounding for Hallucination-Resistant Report Generation
Jun 25, 2026Zhixiang Lu, Xiwei Liu, Sifan Song +5Radiology Report Generation Medical VLMs
Xi’an Jiaotong-Liverpool University · University of Liverpool · Mohamed bin Zayed University of Artificial Intelligence +2
Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs
Jun 24, 2026Xi Xiao, Chen Liu, Chih-Ting Liao +9Multimodal Large Language Models Visual Grounding
UAB · Yale · UNSW +6
Enhancing Brain MRI Anomaly Detection and Reasoning with ROI Rethink and Synthetic Data
Jun 24, 2026Shangkun Li, Jie Xu, Yi Guo +2Medical Image Anomaly Detection VLM Reasoning
College of Biomedical Engineering, Fudan University, Shanghai, China
Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs
Jun 24, 2026Yunhun Nam, Jongheon JeongSynthetic Data Generation Object Hallucination in VLMs
Korea University
E-MRL: Cross-view Aligned Evidence-driven Multimodal Reinforcement Learning for Reliable 3D Tumor Analysis
Jun 22, 2026Sijing Li, Zhongwei Qiu, Zhuoya Wang +6Medical Image Grounding VLM Hallucination Mitigation
Zhejiang University · DAMO Academy, Alibaba Group · Hupan Lab +3
Mind the Heads: Topological Representation Alignment for Multimodal LLMs
Jun 22, 2026Davide Caffagni, Alberto Compagnoni, Federico Melis +5Cross-Modal Alignment Multimodal Large Language Models
University of Modena and Reggio Emilia · University of Pisa · AMD Silo AI
CFPO: Counterfactual Policy Optimization for Multimodal Reasoning
Jun 22, 2026Zhangyuan Yu, Wanran Sun, Guangjing Yang +2Multimodal Reasoning VLM Hallucination Mitigation
Beijing University of Posts and Telecommunications, Beijing, China.
Improving Reasoning in Vision-Language Models via Perception Verified Self-Training
Jun 20, 2026Sourabh Sharma, Sonam Gupta, SadbhawnaVLM Reasoning VLM Hallucination Mitigation
Malaviya National Institute of Technology Jaipur, India · IBM Software Innovation Lab, India
Spectral Query-Key Product Weight Steering for Training-Free VLM Hallucination Mitigation
Jun 18, 2026Karn Tiwari, Varnith Chordia, Prathosh A PObject Hallucination in VLMs VLM Hallucination Mitigation
1Indian Institute of Science, Bengaluru · 2Snap Research
Language-Instructed Vision Embeddings for Controllable and Generalizable Perception
Jun 17, 2026Chengzhi Mao, Xudong Lin, Wen-Sheng ChuVisual Representation Learning Vision Foundation Model Adaptation
Google
Hallucination Detection and Correction in Medical VLMs via Counter-Evidence Verification
Jun 17, 2026Nan Zhou, Ke Zou, Meng Liu +5Hallucination Detection in VLMs Vision-Language Models
the College of Computer Science, Sichuan University. · Yong Loo Lin School of Medicine, National University of Singapore. · National Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing2026 Institute of Technology. +2
CaVe-VLM-CoT: An Interpretable Vision-Language Model Framework
Jun 16, 2026Sneha Rao, Shaina Raza, Dhanesh RamachandramVLM Evaluation Vision-Language Grounding
Vector Institute, Toronto, ON M5G 0C6, Canada
MODE-RAG: Manifold Outlier Diagnosis and Energy-based Retrieval-Augmented Generation Evaluation
Jun 16, 2026Zehang Wei, Jiaxin Dai, Jiamin Yan +1Multimodal RAG Multi-Agent LLM Systems
School of AI and Automation, Huazhong University of Science and Technology, China · School of Computer Science & Tech, Huazhong University of Science and Technology
Semantic Flip: Synthetic OOD Generation for Robust Refusal in Embodied Question Answering and Spatial Localization
Jun 15, 2026Dongbin Na, Chanwoo Kim, Giyun Choi +1Synthetic Data Generation Embodied QA
RGA Inc.
Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model
Jun 15, 2026Jian Xu, Delu Zeng, John Paisley +1Hallucination Detection in VLMs Active Perception
RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1
The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineages
Jun 14, 2026Miso Choi, Seonga Choi, Mincheol Kwon +3LLM Grounding LLM Interpretability
Korea University, Seoul, Republic of Korea · Kakao Mobility Corp., Seongnam, Republic of Korea
Mitigating Visual Hallucinations in Multimodal Systems through Retrieval-Augmented Reliability-Aware Inference
Jun 14, 2026Pratheswaran Hariharan, Haiping Xu, Donghui YanProbability Calibration Selective Prediction
Computer and Information Science University of Massachusetts, Dartmouth, MA · Mathematics and Data Science University of Massachusetts, Dartmouth, MA
Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs
Jun 10, 2026Shayan Mohammadizadehsamakosh, Pritam Sarkar, Leonid Sigal +2Medical VLMs Medical Image Grounding
York University · Vector Institute · University of British Columbia +1
MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models
Jun 10, 2026Yuansheng Gao, Wenbin Xing, Jiahao Yuan +4Video-Language Models Multimodal Foundation Models
Zhejiang University · Sun Yat-sen University · East China Normal University
Disentangling Hallucinations: Orthogonal Semantic Projection for Robust Interpretability
Jun 8, 2026Emirhan Bilgiç, Baptiste Caramiaux, Zhi Yan +1VLM Interpretability Multimodal Model Interpretability
U2IS, ENSTA, Institut Polytechnique de Paris, Palaiseau · ISIR, Université Sorbonne, Pierre et Marie Curie, Paris · AMIAD, Pôle Recherche, Palaiseau
MAGIS: Evidence-Based Multi-Agent Reasoning for Interpretable Strabismus Clinical Decision-Making
Jun 8, 2026Xikai Tang, Yifan Wang, Jiafan Zhuang +12Medical Diagnosis Medical VLMs
School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China · Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen, 518000, China · Joint Shantou International Eye Center of Shantou University and The Chinese University of Hong Kong, Shantou, 515041, China +5
P $^2$ -DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
Jun 2, 2026Ruipeng Zhang, Zhihao Li, Haozhang Yuan +2VLM Robustness Direct Preference Optimization
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology · Pazhou Lab, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China
Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation
Jun 2, 2026Ruipeng Zhang, Zhihao Li, C. L. Philip Chen +1VLM Hallucination Large Vision-Language Models
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China · Pazhou Lab, Guangzhou, China.
Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation
Jun 2, 2026Boyuan Xiao, Bohong Chen, Yumeng Li +3Vision-Language-Action Models VLMs for Robotics
State Key Lab of CAD&CG, Zhejiang University, China · Baiont Quant, China · Zhejiang Key Laboratory of Intelligent Medical Decision Support, China
Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decoding
May 31, 2026Yingxuan Zhuang, Jingxiao Yang, Miao Pan +7Object Hallucination in VLMs VLM Hallucination Mitigation
Ningbo Global Innovation Center, Zhejiang University
Ask4VG: Risk-Aware Question Selection for Reducing Prior-Driven Answers in Medical VQA
May 31, 2026Xiaorong Zhu, Qiang Li, Zibo Xu +2Medical VQA VLM Hallucination Mitigation
School of Microelectronics, Tianjin University, Tianjin 300072, China · DISI, University of Trento, Trento, Italy · School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue
May 30, 2026Yue Jiang, Xue Jiang, Lihua Zhang +6Multimodal Large Language Models Multimodal Hallucination
College of Intelligent Robotics and Advanced Manufacturing, Fudan University · Southern University of Science and Technology · TMLR Group, Hong Kong Baptist University +3

Jun 28, 2026, cs.AIJ/K move · Enter open · S save

FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models

Yichen Guo, Kai Tang, Fenglai Lin, Yiding Sun+4

1Nanyang Technological University · 2State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 3Tsinghua University

Vision-Language Models Large Vision-Language Models Object Hallucination in VLMs VLM Interpretability Faithfulness in VLMs VLM Hallucination Mitigation

Open paper AskPDF arXiv

Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.

Momentum

Related topics

Latest papers 131