Faithfulness in VLMs

VLM: Vision-Language Model

This field studies whether vision-language models use image or video evidence to make decisions, rather than relying on language priors or producing plausible but unsupported explanations. Researchers test this with counterfactual image edits, modality ablations, grounding and attribution measures, and benchmarks for hallucination, spatial reasoning, and evidence use. Methods include process-level supervision and grounding rewards, as well as inference-time decoding or causal interventions that strengthen visual evidence and suppress conflicting language pathways. A central challenge is that accurate answers and fluent reasoning do not prove faithful evidence use, while reducing hallucinations can also make responses less informative.

Latest papers

84 papers19 papers in the last four weeks, up 111% on the four weeks before. 0.1% of all new papers.

Skill-Conditioned Visual Geolocation for Vision-Language Models
Apr 10, 2026Chenjie Yang, Yutian Jiang, Yutong Deng +1VLM Reasoning Vision-Language Models
Southwest Jiaotong University Chengdu, China · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China · Zhejiang University Hangzhou, China
Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimization
Apr 9, 2026Sai Srinivas Kancheti, Aditya Kanade, Rohit Sinha +2Visual Spatial Reasoning Multimodal CoT Reasoning
CSE Dept., IIT Hyderabad, Hyderabad, India · Microsoft Research, Bengaluru, India
When to Call an Apple Red: Humans Follow Introspective Rules, VLMs Don't
Apr 7, 2026Jonathan Nemitz, Carsten Eickhoff, Junyi Jessy Li +3Faithfulness in VLMs VLM Calibration
University of Tübingen · The University of Texas at Austin · Brown University
Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward
Apr 6, 2026Shizhan Gong, Minda Hu, Qiyuan Zhang +2VLM Reasoning Faithfulness in VLMs
The Chinese University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding
Apr 6, 2026Haruka Kawasaki, Ryota Tanaka, Kyosuke NishidaLarge Vision-Language Models Document Image Understanding
Human Informatics Labs., NTT, Inc.
Don't Blink: Evidence Collapse during Multimodal Reasoning
Apr 5, 2026Suresh Raghu, Satwik PandeySelective Prediction for VLMs Vision-Language Models
Independent Researcher, New Delhi, India
Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions
Mar 30, 2026Doan Nam Long Vu, Simone BalloccuMedical VLMs Prompt Sensitivity
Natural Language Processing for Expert Domains (ExpNLP), Technical University of Darmstadt
CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models
Mar 30, 2026Kesheng Chen, Yamin Hu, Qi Zhou +2Emotional Hallucination in VLMs VLM Hallucination
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China
ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better
Mar 27, 2026Mriganka Nath, Anurag Das, Jiahao Xie +1VLM Hallucination Mitigation Test-Time Adaptation of VLMs
Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
Learning to Rank Caption Chains for Video-Text Alignment
Mar 26, 2026Ansel Blume, Burak Uzkent, Shalini Chaudhuri +1Vision-Language Alignment Preference Optimization
University of Illinois Urbana-Champaign · Amazon Prime Video
MIRAGE: The Illusion of Visual Understanding
Mar 23, 2026Mohammad Asadi, Jack W. O'Sullivan, Fang Cao +5VLM Evaluation Faithfulness in VLMs
Department of Electrical Engineering, Stanford University, CA, USA · Division of Cardiology, Department of Medicine, Stanford University, CA, USA · Department of Biomedical Data Science, Stanford University, CA, USA +3
Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models
Mar 22, 2026Binesh Sadanandan, Vahid BehzadanVLM Evaluation Medical VLMs
SAIL Lab, University of New Haven, West Haven, CT, USA
INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs
Mar 12, 2026Junqi Yang, Yuecong Min, Jie Zhang +2Video-Language Model Evaluation Video-Language Models
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, UCAS
Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppression
Mar 11, 2026Hamidreza Dastmalchi, Aijun An, Ali Cheraghian +1VLM Hallucination VLM Hallucination Mitigation
York University Toronto, Canada · Macquarie University Sydney, Australia · Northern Illinois University Illinois, USA
GeoSolver: Scaling Test-Time Reasoning in Remote Sensing with Fine-Grained Process Supervision
Mar 10, 2026Lang Sun, Ronghao Fu, Zhuoran Duan +3VLM Reasoning Process Reward Models
College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University
Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge
Mar 7, 2026Shuai Lu, Meng Wang, Jia Guo +6Vision-Language Models Medical VLMs
Beijing Institute of Technology, Beijing, China · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · National University of Singapore, Singapore +2
Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models
Mar 6, 2026Md Ashikur Rahman, Md Arifur Rahman, Niamul Hassan Samin +2Vision-Language Grounding VLM Robustness
The KOW Company · University of Dhaka · American International University - Bangladesh (AIUB)
PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignment
Feb 28, 2026Yantao Li, Qiang Hui, Chenyang Yan +8Multimodal Reasoning Faithfulness in VLMs
National Key Laboratory for Novel Software Technology, Nanjing University · Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom
Causal Decoding for Hallucination-Resistant Multimodal Large Language Models
Feb 24, 2026Shiwei Tan, Hengyi Wang, Weiyi Qin +3Object Hallucination in VLMs VLM Hallucination Mitigation
Rutgers University · Meta Ranking AI
TAG: Thinking with Action Unit Grounding for Facial Expression Recognition
Feb 21, 2026Haobo Lin, Tianyi Bai, Jiajun Zhang +5Facial Expression Recognition VLM Reasoning
PKU · JLU · Shanghai AI Lab +2
Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain
Feb 19, 2026Seulbi Lee, Sangheum HwangVision-Language Models Large Vision-Language Models
Department of Data Science, Seoul National University of Science and Technology, Seoul, Republic of Korea. · Currently with Tomocube, Daejeon, Republic of Korea.
Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs
Jan 20, 2026Yujin Jo, Sangyoon Bae, Taesup KimVLM Hallucination Mitigation Object Hallucination in VLMs
Seoul National University
More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
Oct 6, 2025Xurui Song, Shuo Huai, JingJing Jiang +2VLMs for Autonomous Driving Reasoning for VLA Models
S-Lab, Nanyang Technological University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore
TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Grounding
Aug 11, 2025Chaohong Guo, Xun Mo, Yongwei Nie +3Temporal Video Grounding Vision-Language Models
South China University of Technology · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Bytedance Inc.

Apr 10, 2026J/K move · Enter open · S save

Skill-Conditioned Visual Geolocation for Vision-Language Models

Chenjie Yang, Yutian Jiang, Yutong Deng, Chenyu Wu

Southwest Jiaotong University Chengdu, China · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China · Zhejiang University Hangzhou, China

VLM Reasoning Vision-Language Models Faithfulness in VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) have shown a promising ability in image geolocation, but they still lack structured geographic reasoning and the capacity for autonomous self-evolution. Existing methods predominantly rely on implicit parametric memory, which often exploits outdated knowledge and generates hallucinated reasoning. Furthermore, current inference is a "one-off" process, lacking the feedback loops necessary for self-evolution based on reasoning outcomes. To address these issues, we propose GeoSkill, a training-free framework based on an evolving Skill-Graph. We first initialize the graph by refining human expert trajectories into atomic, natural-language skills. For execution, GeoSkill employs an inference model to perform direct reasoning guided by the current Skill-Graph. For continuous growth, an Autonomous Evolution mechanism leverages a larger model to conduct multiple reasoning rollouts on image-coordinate pairs sourced from web-scale data and verified real-world reasoning. By analyzing both successful and failed trajectories from these rollouts, the mechanism iteratively synthesizes and prunes skills, effectively expanding the Skill-Graph and correcting geographic biases without any parameter updates. Experiments demonstrate that GeoSkill achieves promising performance in both geolocation accuracy and reasoning faithfulness on GeoRC, while maintaining superior generalization across diverse external datasets. Furthermore, our autonomous evolution fosters the emergence of novel, verifiable skills, significantly enhancing the system's cognition of real-world geographic knowledge beyond isolated case studies.