Faithfulness in VLMs

VLM: Vision-Language Model

This field studies whether vision-language models use image or video evidence to make decisions, rather than relying on language priors or producing plausible but unsupported explanations. Researchers test this with counterfactual image edits, modality ablations, grounding and attribution measures, and benchmarks for hallucination, spatial reasoning, and evidence use. Methods include process-level supervision and grounding rewards, as well as inference-time decoding or causal interventions that strengthen visual evidence and suppress conflicting language pathways. A central challenge is that accurate answers and fluent reasoning do not prove faithful evidence use, while reducing hallucinations can also make responses less informative.

Latest papers

84 papers19 papers in the last four weeks, up 111% on the four weeks before. 0.1% of all new papers.

Uncovering Shortcut Amplification in Video RLVR through the Lens of Perceptual Uncertainty Collapse
Oct 27, 2026Jiafeng Liang, Xianhao Ou, Tao Ren +6RL for Video Reasoning Reinforcement Learning with Verifiable Rewards
Harbin Institute of Technology · Peking University · Huawei Technologies Co., Ltd +2
PRISM-BENCH: Diagnosing Grounded and Faithful Multimodal Reasoning Beyond Final-Answer Accuracy
Oct 27, 2026Vansh Bhatia, Mehdi Rezagholizadeh, Parsa Ashrafi Fashi +3MLLM Evaluation Multimodal Model Evaluation
Advanced Micro Devices, Inc. · University of Toronto
Conviction Without Evidence: Verdict Stability Is Independent of Visual Grounding in Medical Vision--Language Models
Oct 25, 2026Federico Felizzi, Francesco Andrea Causio, Bianca Destro Castaniti +17Medical VLMs Visual Evidence Sensitivity in VLMs
SIIAM, Rome, Italy · Dept. of Computer Science, University of Pennsylvania, USA · UCSC, Rome, Italy +4
Of Chains and Gains: What Reinforcement Learning changes on Reasoning-based Image Quality Assessment
Oct 23, 2026Yixuan Li, Yuxuan Jiang, Hanwei Zhu +4Image Quality Assessment Reinforcement Learning with Verifiable Rewards
University of Bristol, UK · Jiangxi University of Finance and Economics, China · China University of Mining and Technology, China +1
Where Do Long Captions Fail? Position-Aware Diagnosis and Reinforcement Learning for Detailed Image Captioning
Oct 22, 2026Yuanze Hu, Zhichao Yang, Junwei Jing +9Image Captioning Faithfulness in VLMs
Beihang University · Kuaishou Technology · Fudan University +1
Diagnostic Loss: Asymmetric Supervision for Radiology Report Generation
Oct 22, 2026Ignacy Kolton, Wenxuan Li, Zongwei Zhou +4Radiology Report Generation Faithfulness in VLMs
Jagiellonian University · Johns Hopkins University · GAMMED +2
NavBind: Diagnosing Instruction-Action Binding at Route Forks in Navigation VLAs
Oct 21, 2026Josh Qixuan Sun, Xiaoying Xing, Yurun Chen +3Vision-Language-Action Models Vision-Language Navigation
University of Waterloo · Northwest University · Shanghai Jiao Tong University +1
What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models
Oct 19, 2026Wenhao Zhang, Zhongliang Zhou, Yi Ping +2Medical VLMs VLM Evaluation
School of Data Science, University of Virginia, Charlottesville, VA 22903, USA · Merck & Co., Inc., Rahway, NJ 07065, USA
Shift-Reliance Alignment: Ranking Vision-Language Models by the Cues They Trust
Oct 15, 2026Faisal Mohammad, Mohamed YounisVLM Evaluation Object Hallucination in VLMs
University of Maryland, Baltimore County · Johns Hopkins University Applied Physics Laboratory
RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark
Oct 13, 2026Mahsa Khoshnoodi, Sarah Adel BargalVideo-Language Models Video-Language Model Evaluation
Department of Computer Science, Georgetown University
Hallucination-aware Faithful Reasoning in Remote Sensing
Oct 13, 2026Yi Liu, Jing Zhang, Xiang Feng +5VLM Hallucination Mitigation Faithfulness in VLMs
School of Computer Science, Wuhan University · Zhongguancun Academy · School of Computer Science, Chongqing University
Teaching Vision-Language Models to Decline: Support Completion Against Mirage
Oct 12, 2026Alfaj Uddin Ahmed, Md. Shaown Miah, Md Arid Hasan +2Abstention in VLMs VLM Hallucination Mitigation
Department of Biomedical Engineering, Purdue University, West Lafayette, IN, USA · Department of Computer Science, Luddy School of Informatics, Computing, and Engineering, Indiana University Indianapolis, Indianapolis, IN, USA · Department of Computer Science, University of Toronto, Toronto, ON, Canada
Attention Sinks in VLMs Are Borrowed, Not Seen
Oct 11, 2026Dengjia Zhang, Alexander Martin, Pengyu Yan +6Attention Intervention in VLMs VLM Interpretability
Johns Hopkins University · University of Pittsburgh · University at Buffalo +2
When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors
Oct 9, 2026Shubh Garg, Chen ShaniFaithfulness in VLMs VLM Evaluation
Thapar Institute of Engineering and Technology · Tel Aviv University
CV-OPD: Contrastive Vision On-policy Distillation
Oct 9, 2026Haowei Wang, Hao Lin, Qi Zhu +4VLM Distillation Vision-Language Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · Moxin Technology
Visual Evidence Under Cross-Examination: Evaluating and Controlling Decision-Level Evidence Use in Vision-Language Models
Oct 7, 2026Huiyao Zhang, Jin Bai, Zilong Su +6VLM Evaluation Faithfulness in VLMs
Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences · University of Chinese Academy of Sciences · University of Science and Technology of China
SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs
Oct 5, 2026Rafael Teixeira Sousa, Vinícius Paulo Lopes de Oliveira, Elisa Ayumi Masasi de Oliveira +5VLM Evaluation Spatial Reasoning Benchmarks
Universidade Federal de Mato Grosso (UFMT) · Universidade Federal de Goiás (UFG) · Advanced Knowledge Center for Immersive Technologies (AKCIT)
Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally
Oct 2, 2026Arun Josephraj Arokiaraj, Chenyi Huang, Zekun Wu +1VLM Robustness Adversarial Attacks on VLMs
Department of Computer Science, University College London · Holistic AI
Tracing the Evidence: Faithful Token Attribution Through Vision-Language Reasoning
Sep 29, 2026Bowen Yuan, Danny Wang, Ruihong Qiu +2Large Vision-Language Models VLM Interpretability
The University of Queensland
Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Models
Sep 28, 2026Xingming Long, Jie Zhang, Yuecong Min +2Credit Assignment in RL Step-Level Credit Assignment in RL
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhongguancun Academy
When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Context
Sep 28, 2026Yuxing Cheng, Yuan Wu, Yi ChangVLM Evaluation Counterfactual Evaluation of VLMs
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMs
Sep 22, 2026Dingyang Lin, Yingfeng Luo, Chenglong Wang +4Visual Evidence Sensitivity in VLMs VLM Robustness
School of Computer Science and Engineering, Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China
Taming CoT Obfuscation in VLMs: From Mechanistic Evidence to Activation-Level Enforcement
Sep 21, 2026Xutao Mao, Jianing Zhu, Jinman Zhao +4RL for VLMs Faithfulness in VLMs
City University of Hong Kong · Hong Kong Baptist University · University of Toronto +2
Benchmarking the Explanatory Quality of Open-Weight Vision-Language Models in Face Recognition
Sep 18, 2026Laurent Colbois, Sébastien MarcelVLM Evaluation Face Recognition
Idiap Research Institute, Switzerland · Université de Lausanne, Switzerland
EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing
Sep 16, 2026Sihao Ding, Santosh Vasa, Aditi Ramadwar +1VLM Evaluation VLM Robustness
Mercedes-Benz Research & Development North America
I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models
Sep 7, 2026Aydin Javadov, Daniel Schoess, Florian von WangenheimVLM Evaluation VLM Interpretability
ETH Zurich
Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models
Sep 6, 2026Bayar Menzat, Maximilian Süss, Ruizhi Wang +3CoT Faithfulness Faithfulness in VLMs
Vienna University of Technology, Austria · University of Oxford, UK · Imperial College London, UK +1
Grounded and Faithful P&ID Reasoning: Constraining Vision-Language Models with Recovered Evidence Graphs
Sep 5, 2026Prathamesh Gadekar, Sagar Srinivas Sakhinana, Venkataramana RunkanaVLM Reasoning Faithfulness in VLMs
Tata Research Development and Design Center, Pune, India 411057
Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methods
Sep 1, 2026Mehrdad Fazli, Sina Mansouri, Mohit Marvania +1VLM Evaluation Faithfulness in VLMs
Department of Computer Science, George Mason University Fairfax, VA 22030, USA
Visual Attention Faithfulness in Vision-Language Models is Heterogeneous
Sep 1, 2026Xurui Song, Weishi Wang, Zhongqi Yue +5VLM Evaluation Visual Question Answering
SAP · Microsoft · College of Computing and Data Science, Nanyang Technological University, Singapore

Oct 27, 2026J/K move · Enter open · S save

Uncovering Shortcut Amplification in Video RLVR through the Lens of Perceptual Uncertainty Collapse

Jiafeng Liang, Xianhao Ou, Tao Ren, Rongchuan Mu+5

Harbin Institute of Technology · Peking University · Huawei Technologies Co., Ltd+2

RL for Video Reasoning Reinforcement Learning with Verifiable Rewards Visual Evidence Sensitivity in VLMs Faithfulness in VLMs Confidence Estimation for VLMs Self-Distillation

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective post-training paradigm for enhancing the video reasoning capability of Large Multimodal Models (LMMs). Despite its empirical effectiveness, we show that outcome-level optimization can introduce an undesirable shortcut amplification effect, whereby models increasingly favor text-prior answers even when they are not supported by the visual evidence. To characterize this behavior, we investigate the training dynamics of video RLVR through the lens of perceptual uncertainty, analyzing how predictive uncertainty evolves under video-conditioned and text-only contexts. Our analysis identifies a perceptual uncertainty collapse phenomenon, in which RL training continuously sharpens the model's answer distribution while the uncertainty reduction attributable to visual evidence progressively diminishes. A further gradient-direction analysis reveals that such confidence enhancement becomes increasingly less aligned with visually grounded uncertainty reduction and is instead dominated by text-prior compression. Motivated by these findings, we propose Bifocal Self-distillation Policy Optimization (BSPO), a self-distillation framework that regulates policy updates with two complementary distributional signals. Specifically, BSPO derives empirical reasoning principles from intra-group correct and incorrect rollouts to identify reasoning-sensitive tokens, and introduces shortcut-aware visual perturbation to penalize predictions that remain invariant when task-relevant visual evidence is weakened. By adaptively reweighting the original RL advantage, BSPO preserves the task-solving benefits of RLVR while promoting updates that are both reasoning-sensitive and visually grounded. Extensive experiments validate the effectiveness of BSPO in improving video reasoning performance and strengthening visual faithfulness.