VLM Evaluation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

70 papers in the last four weeks, up 67% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 649

All topics

Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methods
Sep 1, 2026Mehrdad Fazli, Sina Mansouri, Mohit Marvania +1VLM Evaluation Faithfulness in VLMs
Department of Computer Science, George Mason University Fairfax, VA 22030, USA
MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models
Sep 1, 2026Tawsif Tashwar Dipto, Mehedi Ahamed, Radib Bin Kabir +5VLM Evaluation Multilingual VLM Evaluation
1Islamic University of Technology · 2South East University · 3Vector Institute +3
FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making
Sep 1, 2026Vahid Reza Khazaie, Ahmed Y. Radwan, Shaina RazaVLM Evaluation
Vector Institute, Toronto, Canada
InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations
Sep 1, 2026Maeve Hutchinson, Syed Mahbubul Huq, Mohammad Albinhassan +3VLM Evaluation Claim Verification
City, University of London · Imperial College London · The Alan Turing Institute
IntroConformal: Conformal Factuality Guarantees for Large Vision-Language Models via Introspective Signals
Sep 1, 2026Md. Atabuzzaman, Christian Alexander, Chris ThomasVLM Evaluation Large Vision-Language Models
Department of Computer Science Virginia Tech
Reliability Challenges in Diffusion Vision-Language Models
Sep 1, 2026Md. Atabuzzaman, Chris ThomasVLM Evaluation VLM Robustness
Department of Computer Science Virginia Tech
A multicenter benchmark and clinically structured metric for coronary CTA report generation
Sep 1, 2026Zhiyu Ye, Yue Sun, Limiao Zou +7VLM Evaluation Radiology Report Generation
State Key Laboratory of Biomedical Imaging Science and System, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · University of Chinese Academy of Sciences, Beijing, China +1
The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence
Sep 1, 2026Genpei ZhangVLM Evaluation Large Vision-Language Models
University of Wisconsin–Madison
Visual Attention Faithfulness in Vision-Language Models is Heterogeneous
Sep 1, 2026Xurui Song, Weishi Wang, Zhongqi Yue +5VLM Evaluation Visual Question Answering
SAP · Microsoft · College of Computing and Data Science, Nanyang Technological University, Singapore
Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structures
Sep 1, 2026Can Polat, Mustafa Kurban, Erchin Serpedin +1VLM Evaluation VLM Reasoning
Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA · Department of Electrical and Computer Engineering, Texas A&M University at Qatar, Doha, Qatar · Department of Prosthetics and Orthotics, Ankara University, Ankara, Turkey +1
You Cannot Photograph the Same Street Twice: Reliability Limits in Vision-Language Measurement of Urban Change
Sep 1, 2026Kaizhen TanVLM Evaluation Vision-Language Models
New York University
Beyond Blind Compliance: Benchmarking Task Verification in OCR Reasoning
Aug 31, 2026Yue Zhou, Yuan Wu, Yi ChangVLM Evaluation Large Vision-Language Models
School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · 3International Center of Future Science, Jilin University
Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy
Aug 31, 2026Shmuel Berman, Jia DengVLM Evaluation Memory-Augmented VLMs
Department of Computer Science Princeton University
Do VLMs Share Safety Neurons Across Modalities?
Aug 31, 2026Jiaxuan Li, Jiahao Zhang, Duc Minh Vo +3VLM Evaluation VLM Interpretability
SB Intuitions Corp., Tokyo, Japan
Fine-Grained Multi Image Object Hallucination Benchmark
Aug 31, 2026Joonki Min, Chaeyun Kim, Hyungwook Choi +4VLM Evaluation Object Hallucination in VLMs
Seoul National University · AIM Intelligence
Lot Machine: Multimodal Lot Extraction from Auction Catalogs
Aug 31, 2026Mathias Zinnen, Alisha Mund, Sabine Lang +3VLM Evaluation Document Image Understanding
Pattern Recognition Lab, FAU Erlangen · Department of Digital Humanties and Social Studies, FAU Erlangen
Frontier vision-language models have overtaken young adults at detecting AI-generated portraits -- but not their calibration
Aug 31, 2026Sunwhi Kim, Sunyul Kim, Meounggun Jo +1VLM Evaluation Vision-Language Models
Hwasung Medi-Science University, Dept. of Bio-Healthcare · Hwasung Medi-Science University, Dept. of Bio-Healthcare, Republic of Korea · Yonsei University, Graduate School of Engineering, Dept. of Artificial Intelligence +5
Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models
Aug 30, 2026Aditi Sarker, Nazreen Shah, Rafi Ibn Sultan +3VLM Evaluation VLM Unlearning
Department of Computer Science, Wayne State University, MI, USA · Department of Electronics & Communications Engineering, IIIT Delhi, India · Institute for AI and Data Science (AIDaS), Wayne State University, MI, USA
MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models
Aug 27, 2026Ashshak Sharifdeen, Shihab Aaqil Ahamed, Ufaq Khan +6VLM Evaluation VLM Robustness
1Mohamed bin Zayed University of AI · 2Sabaragamuwa University of Sri Lanka · 3Digital Platform Development, SLT PLC
Do Vision-Language Models Agree on the Affective Qualities of Shape? A Cross-Model Audit for Generative Design Interfaces
Aug 26, 2026Luca Bux, Thiago Rios, Ingo Scholtes +2VLM Evaluation Vision-Language Models
Honda Research Institute Europe GmbH Offenbach am Main, Germany · Julius-Maximilians-Universität Würzburg Würzburg, Germany
When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs
Aug 25, 2026Zhengxiang Wang, Owen RambowVLM Evaluation Large Vision-Language Models
Department of Linguistics & IACS Stony Brook University
How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures
Aug 13, 2026Paul Osemudiame Oamen, Owusu-Banahene Osei, Ananya Mukherjee +4VLM Evaluation VLM Robustness
The Aberdeen NLP Research Group, University of Aberdeen, UK · International Institute of Information Technology Hyderabad, India · University of Technology Nuremberg, Germany +1
TRAPSBench: Vision-Language Models Encode but Fail to Express Epistemic Restraint
Aug 13, 2026Fnu Pramono, John Cai, Sourabh KulkarniVLM Evaluation Vision-Language Models
Meta Superintelligence Labs · Reflection AI
Can Vision-Language Models Assess Proxemic Risk from Egocentric Robot Images?
Aug 12, 2026Vladyslava Rudas, Dmytro KuzmenkoVLM Evaluation Robot Safety
Department of Computer Science, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine · Department of Multimedia Systems, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine · Department of Computer Science, University of Turin, Turin, Italy
LookBack: Where and How to Score LVLM Responses via Visual Reference Usage
Aug 12, 2026Beomsik Cho, Jinhyeong Kim, Dongseok Lee +1VLM Evaluation Hallucination Detection in VLMs
Yonsei University
How China-Origin Vision-Language Models Move from Refusal to Reframing in State Alignment
Aug 12, 2026Guang Yang, Fengchen Liu, Alex Wang +2VLM Evaluation Refusal Behavior in Language Models
University of California, Los Angeles · University of California, Berkeley · Stanford University
JieZi: A Large-Scale Expert-Audited Dataset and Benchmark for Ancient Chinese Character Exegesis
Aug 12, 2026Ran Li, Huiguo He, Jiahuan Cao +3VLM Evaluation VLM Reasoning
South China University of Technology Guangzhou, China
VLMs Win a Systematic Evaluation of Underwater Image Reconstruction
Aug 11, 2026Sara Aghajanzadeh, Yingxue Wang, Ieva Bagdonaviciute +1VLM Evaluation Underwater Image Enhancement
University of Illinois Urbana-Champaign
Can Frontier LLMs Match Natively Multimodal Embeddings? A Comparison on Hard-Negative Text-to-Image Retrieval
Aug 11, 2026Archan Dutta, Vyanktesh KanungoVLM Evaluation LLM Reranking
Westcliff University
CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering
Aug 11, 2026Mouxiao Huang, Qiangyu Yan, Borui Jiang +1VLM Evaluation Image Captioning
Huawei Technologies

Sep 1, 2026, cs.CVJ/K move · Enter open · S save

Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methods

Mehrdad Fazli, Sina Mansouri, Mohit Marvania, Ziwei Zhu

Department of Computer Science, George Mason University Fairfax, VA 22030, USA

VLM Evaluation Faithfulness in VLMs Multimodal Model Evaluation VLM Hallucination Mitigation

Open paper AskPDF arXiv

Recent inference-time hallucination mitigation methods for large vision-language models (LVLMs) report strong gains on hallucination benchmarks. However, it remains unclear whether lower hallucination scores reflect improved multimodal grounding or more conservative generation. We evaluate six mitigation methods across three LVLMs and four benchmarks, including hallucination-focused evaluation and the diverse capability benchmark MMStar. Our analysis reveals two consistent patterns. First, hallucination reduction is often coupled with reduced informativeness: methods that lower hallucination rates also reduce object recall, visual coverage, or response detailedness. Second, improvements on hallucination benchmarks do not reliably transfer to broader multimodal capabilities, with methods showing inconsistent or degraded performance on fine-grained perception and reasoning tasks. Our findings suggest that current evaluation protocols may overestimate progress by rewarding conservative generation. We argue that hallucination mitigation should be evaluated as a faithfulness--informativeness--capability trade-off rather than through hallucination scores alone.

Momentum

Related topics

Latest papers 649