Faithfulness in VLMs

VLM: Vision-Language Model

This field studies whether vision-language models use image or video evidence to make decisions, rather than relying on language priors or producing plausible but unsupported explanations. Researchers test this with counterfactual image edits, modality ablations, grounding and attribution measures, and benchmarks for hallucination, spatial reasoning, and evidence use. Methods include process-level supervision and grounding rewards, as well as inference-time decoding or causal interventions that strengthen visual evidence and suppress conflicting language pathways. A central challenge is that accurate answers and fluent reasoning do not prove faithful evidence use, while reducing hallucinations can also make responses less informative.

Latest papers

84 papers19 papers in the last four weeks, up 111% on the four weeks before. 0.1% of all new papers.

EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensation
Aug 29, 2026Sihang Jia, Shuliang Liu, Songbo Yang +1VLM Hallucination Mitigation Large Vision-Language Models
The Hong Kong University of Science and Technology (Guangzhou)
Evaluating Explanation-Driven Vision-Language Reasoning via Generation Order Interventions
Aug 26, 2026Siting Liang, Luca Rippe, Omar Adjali +1VLM Reasoning Reasoning for VLMs
German Research Center for Artificial Intelligence · University of Oldenburg
V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning
Aug 26, 2026Shulin Tian, Minglun Li, Yuhao Dong +6Faithfulness in VLMs RL for VLMs
S-Lab, Nanyang Technological University · A*STAR · Independent Researcher +1
SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answering
Aug 22, 2026Long Shu, Shuochen Liu, Wei Chen +4Knowledge-Based VQA Multimodal RAG
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China China · NIO China
Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment
Aug 20, 2026Yunseo Lee, Hyun Jun Kim, Heeseung Shin +1Medical VLMs Vision-Language Alignment
Department of Statistics and Data Science, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea · Department of Smart Cities, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea · Department of Applied Statistics, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea
MR-IQA-2: Faithful Image Quality Reflection via Fine-Grained Credit Assignment
Aug 19, 2026Yuan li, Youyuan Lin, Chenhui Chu +1Image Quality Assessment Faithfulness in VLMs
Graduate School of Informatics, Kyoto University
Diagnosing Dense Same-Class Attribute Misbinding in Large Vision-Language Models
Aug 17, 2026Yuanzhi Xu, Qian Gao, Jun Fan +4VLM Evaluation Large Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences), Jinan, China · China Telecom Digital Intelligence Technology Co., Ltd., Jinan, China · Shenyang Aerospace University, Shenyang, China +1
LookBack: Where and How to Score LVLM Responses via Visual Reference Usage
Aug 12, 2026Beomsik Cho, Jinhyeong Kim, Dongseok Lee +1VLM Evaluation Hallucination Detection in VLMs
Yonsei University
Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression
Aug 3, 2026Long Qian, Jiaqi Wei, Bingke Zhu +2Efficient VLM Inference Coreset Selection
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Engineering, Cardiff University, Cardiff, United Kingdom · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China +1
DiffuseAgent-MI: Distributionally-Grounded,Tool-Integrated Self-Evolving Agents for Faithful Visual Reasoning
Aug 1, 2026An Lanji, Dawei Liu, Jin Li +3AI Agent Reliability VLM Interpretability
University of Electronic Science and Technology of China, Chengdu, China.
Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMs
Jul 31, 2026Mingyu Wang, Weilin Jin, Wenbo Li +5Hallucination Detection in VLMs Attention Head Analysis
Peking University, Beijing, China · Joy Future Academy, Beijing, China
FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Self-Verification
Jul 30, 2026Haoqing Wang, Xingrun Xing, Ziheng Li +2Tool Use in VLMs VLM Reasoning
Samsung Research, Beijing, China · Peking University
Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models
Jul 29, 2026Supratik Bhowal, Subhrajyoti Basu, Aritra Gir Mahanta +1Medical VQA Medical VLMs
Department of Computer Science (AIML) IEM Kolkata, School of UEMK Kolkata, India · Department of Computer Science and Business System Heritage Institute of Technology Kolkata, India · Department of Computer Science and Engineering Indian Institute of Information Technology, Kalyani Kalyani, India
Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning
Jun 29, 2026Peng, Lee, Yin Zhang +9RL for VLMs Vision-Language Grounding
GMLab · Hong Kong Polytechnic University · XPENG Robotics
FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
Jun 28, 2026Yichen Guo, Kai Tang, Fenglai Lin +5Vision-Language Models Large Vision-Language Models
1Nanyang Technological University · 2State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 3Tsinghua University
Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding
Jun 25, 2026Liu Yu, Can Chen, Ping Kuang +3Object Hallucination in VLMs Faithfulness in VLMs
School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China · School of Computer Science, University of Auckland, Auckland, New Zealand
CaVe-VLM-CoT: An Interpretable Vision-Language Model Framework
Jun 16, 2026Sneha Rao, Shaina Raza, Dhanesh RamachandramVLM Evaluation Vision-Language Grounding
Vector Institute, Toronto, ON M5G 0C6, Canada
Mirage Probes: How Vision Models Fake Visual Understanding
Jun 11, 2026Daniel Ben-Levi, Judah Goldfeder, Weiliang Zhao +5Vision-Language Models Emotional Hallucination in VLMs
Columbia University · Intuit · Technion +2
YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models
May 29, 2026Ting Chen, Geng Li, Guohao Chen +5Efficient VLM Inference Object Hallucination in VLMs
Guangdong University of Technology · Nanyang Technological University · Shenzhen TENCLASS Technology Co., Ltd.
The Abstraction Gap in Vision-Language Causal Reasoning
May 27, 2026Chinh Hoang, Mohammad Rashedul HasanVLM Evaluation Causal Reasoning in Language Models
Department of Electrical and Computer Engineering, University of Nebraska–Lincoln, Lincoln, Nebraska, USA.
Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
May 21, 2026Changyuan Tian, Zhicong Lu, Huaxing Liu +7Multimodal Large Language Models Multimodal Reasoning
AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University +1
Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?
May 20, 2026Tianyi Zhang, Mahtab Bigverdi, Ranjay KrishnaVLM Evaluation VLM Interpretability
University of Washington
Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating
May 20, 2026Zhe Cheng, Wenyu Chen, Fode Zhang +1VLM Robustness Hallucination in Language Models
Center of Statistical Research, School of Statistics and Data Science, Southwestern University of Finance and Economics, Chengdu, China. · Department of Biomedical Engineering, College of Design and Engineering, National University of Singapore, Singapore.
When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
May 7, 2026Harshvardhan Saini, Samyak Jha, Yiming Tang +1VLM Hallucination Mitigation Vision-Language Models
Indian Institute of Technology Dhanbad · National University of Singapore
Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoning
Apr 28, 2026Yashwant Pravinrao Bangde, Debaditya RoyVLM Inference Object Hallucination in VLMs
Department of Computer Science and Engineering Indian Institute of Technology Kharagpur Kharagpur, India 721302
Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation
Apr 27, 2026Yubo Jiang, Xin Yang, Abudukelimu Wuerkaixi +7Hallucination in Language Models Object Hallucination in VLMs
School of Astronautics, Beihang University, Beijing 102206, China · Longcat Interaction Team, Meituan, Beijing 100102, China · Tianmushan Laboratory, Beihang University, Hangzhou 311115, China
Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding
Apr 22, 2026Yubo Jiang, Yitong An, Xin Yang +7Vision-Language Models Hallucination in Language Models
School of Astronautics, Beihang University, Beijing 102206, China · Longcat Interaction Team, Meituan, Beijing 100102, China · Tianmushan Laboratory, Beihang University, Hangzhou 311115, China
Do Thought Streams Matter? Evaluating Reasoning in Gemini Vision-Language Models for Video Scene Understanding
Apr 13, 2026Shivam Sharma, Sankalp Nagaonkar, Ashish Choithani +1VLM Reasoning VLM Evaluation
VideoDB∗
Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
Apr 10, 2026Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2Vision-Language Grounding Visual Evidence Sensitivity in VLMs
Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.
Visually-Guided Policy Optimization for Multimodal Reasoning
Apr 10, 2026Zengbin Wang, Feng Xiong, Liang Lin +5VLM Reasoning RL for VLMs
AMAP, Alibaba Group · SYSU · BUPT

Aug 29, 2026J/K move · Enter open · S save

EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensation

Sihang Jia, Shuliang Liu, Songbo Yang, Xuming Hu

The Hong Kong University of Science and Technology (Guangzhou)

VLM Hallucination Mitigation Large Vision-Language Models Object Hallucination in VLMs Vision-Language Grounding Faithfulness in VLMs

Open paper AskPDF arXiv

Large vision-language models (LVLMs) frequently generate content unsupported by visual inputs. Preliminary experiments show that visual evidence is primarily incorporated into answer-side representations in early-to-middle decoder layers, while its direct influence progressively weakens in later layers. This attenuation suggests that visual evidence acquired earlier may be insufficiently utilized during subsequent generation. Based on this observation, we propose EviAnchor, a training-free and single-branch inference framework that preserves and reactivates visual evidence throughout generation. EviAnchor introduces Regional Evidence Anchor (REA) slots to progressively aggregate dense visual tokens into spatially structured representations. It then strengthens the current decision state's access to these visual anchors through decision-conditioned evidence routing, mitigating excessive dependence on textual context. Finally, the model resumes its native Transformer computation to integrate the retrieved visual evidence with question semantics and generation history. Experiments across POPE, CHAIR, and MMHal-Bench demonstrate consistent improvements in visual grounding.