Vision-Language Models

Also known as VLM

Follow this topic Ask about this topic

Momentum

55 papers in the last four weeks, up 120% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 491

All topics

The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation
May 7, 2026Hoin Jung, Xiaoqian WangMultimodal RAG Multimodal Grounding
Elmore Family School of Electrical and Computer Engineering Purdue University West Lafayette, IN 47907
How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study
May 6, 2026Junran Wang, Xinjie Shen, Zehao Jin +1VLM Evaluation Vision-Language Models
Georgia Institute of Technology
DiCLIP: Diffusion Model Enhances CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation
May 6, 2026Zhiwei Yang, Pengfei Song, Yucong Meng +3Vision-Language Models Weakly Supervised Image Segmentation
Academy for Engineering and Technology, Fudan University, Shanghai 200433, China · Shandong Computer Science Center, China · Digital Medical Research Center, School of Basic Medical Science, Fudan University, and also with Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China +1
Memory-Efficient Continual Learning with CLIP Models
May 5, 2026Ryan King, Gang Li, Bobak Mortazavi +1Class-Incremental Learning Contrastive Learning
Department of Computer Science Texas A&M University College Station, TX 77843 · Texas A&M University College Station, TX 77843
Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks
May 5, 2026JuneHyoung Kwon, MiHyeon Kim, Eunju Lee +3VLM Evaluation VLM Unlearning
Department of Artificial Intelligence, Chung-Ang University · KT Corporation · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
Neuroscience-Inspired Analyses of Visual Interestingness in Multimodal Transformers
May 5, 2026Mathis Immertreu, Fitim Abdullahu, Thomas Kinfe +3Transformer Interpretability Vision-Language Models
Cognitive Computational Neuroscience Group, Patter Recognition Lab, University Erlangen-Nürnberg, Germany · IDS Institut für Data Science, ZHAW School of Engineering, Winterthur, Switzerland · Mannheim Center for Neuromodulation and Neuroprosthetics, University Hospital Mannheim, Heidelberg University, Germany +3
MHPR: Multidimensional Human Perception and Reasoning Benchmark for Large Vision-Languate Models
May 5, 2026Kangkang Wang, Qinting Jiang, Wanping Zhang +2VLM Evaluation Vision-Language Models
AI Compute Group, Baidu · Tsinghua University
MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models
May 2, 2026Yin Zhang, Jiaxuan Zhao, Zonghan Wu +5Vision-Language Models Reinforcement Learning
School of Mathematics, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Shanghai Advanced Institute of Finance (SAIFS), East China Normal University, Shanghai, China +4
Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
May 2, 2026Muyang Li, Yucheng Liu, Jianbo Ma +3Vision-Language Models Vision-Language Alignment
1Sydney AI Centre, The University of Sydney · 2Dolby Laboratories · 3TMLR Group, Hong Kong Baptist University
Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs
May 1, 2026Siyuan Huang, Xiaoye Qu, Yafu Li +6Memory-Augmented VLMs VLM Robustness
Shanghai AI Laboratory · Shanghai Jiao Tong University · The Chinese University of Hong Kong +3
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3
Leveraging Vision-Language Models as Weak Annotators in Active Learning
May 1, 2026Phuong Ngoc Nguyen, Kaito Shiku, Ryoma Bise +2Vision-Language Models Active Learning
Kyushu University, Japan.
Online Self-Calibration Against Hallucination in Vision-Language Models
May 1, 2026Minghui Chen, Chenxu Yang, Hengjie Zhu +3Vision-Language Models Hallucination in Language Models
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions
Apr 30, 2026Ce Chen, Yi Ren, Yuanming Li +5Vision-Language Models Change-Point Detection
University of Melbourne · 2HeyGen Research · 3Nanyang Technological University
The Effects of Visual Priming on Cooperative Behavior in Vision-Language Models
Apr 30, 2026Kenneth J. K. OngVLM Evaluation VLM Robustness
1ST Engineering, Singapore.
Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining
Apr 30, 2026Hyeonseo Jang, Jaebyeong Jeon, Joong-Won Hwang +1Deep Learning Optimization Vision-Language Models
1Yonsei University · 2ETRI
Revealing the Impact of Visual Text Style on Attribute-based Descriptions Produced by Large Visual Language Models
Apr 30, 2026Xiaomeng Wang, Martha Larson, Zhengyu ZhaoVLM Evaluation Vision-Language Models
Radboud University · Nijmegen, Netherlands · Xi’an Jiaotong University +1
Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis
Apr 30, 2026David Fernandez, Pedram MohajerAnsari, Amir Salarpour +1Adversarial Attacks on VLMs Vision-Language Models
School of Computing, Clemson University, USA
COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts
Apr 30, 2026Bingli Wang, Huanze Tang, Haijun Lv +5Vision-Language Models Vision-Language Grounding
1Southeast University · 2Shanghai AI Laboratory
A Multistage Extraction Pipeline for Long Scanned Financial Documents: An Empirical Study in Industrial KYC Workflows
Apr 29, 2026Yuxuan Han, Yuanxing Zhang, Yushuo Wang +3Vision-Language Models Document Information Extraction
OCBC, Singapore
Delineating Knowledge Boundaries for Honest Large Vision-Language Models
Apr 29, 2026Junru Song, Yimeng Hu, Yijing Chen +4VLM Robustness Vision-Language Models
Shandong University
Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models
Apr 28, 2026Chengsheng Zhang, Chenghao Sun, Xinyan Jiang +2Vision-Language Models Hallucination in Language Models
University of Science and Technology of China · Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China · University of Chinese Academy of Sciences, Beijing, China
VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
Apr 28, 2026Divake Kumar, Sina Tayebati, Devashri Naik +2VLM Evaluation Vision-Language Models
University of Illinois at Chicago · 2AI Labs at Capital One
A systematic evaluation of vision-language models for observational astronomical reasoning tasks
Apr 27, 2026Wenke Ren, Hengxiao Guo, Wenwen Zuo +1VLM Evaluation Vision-Language Models
Shanghai Astronomical Observatory, Chinese Academy of Sciences, Shanghai, China · Harvard University, Boston, Massachusetts, USA
PhysNote: Self-Knowledge Notes for Evolvable Physical Reasoning in Vision-Language Model
Apr 27, 2026Sinin Zhang, Yunfei Xie, Yuxuan Cheng +2Vision-Language Models VLM Reasoning
The Chinese University of Hong Kong, Shenzhen · Rice University · City University of Hong Kong +1
Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs
Apr 27, 2026Byeonggeuk Lim, JungMin Yun, Junehyoung Kwon +2Vision-Language Models Hallucination in Language Models
Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University · Department of Artificial Intelligence, Chung-Ang University
SycoPhantasy: Quantifying Sycophancy and Hallucination in Small Open Weight VLMs for Vision-Language Scoring of Fantasy Characters
Apr 27, 2026Arya Shah, Deepali Mishra, Chaklam SilpasuwanchaiVLM Evaluation Vision-Language Models
IIT Gandhinagar Gandhinagar, India · IIT Kanpur Kanpur, India · Asian Institute of Technology Bangkok, Thailand
SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
Apr 27, 2026Zi-Hao Bo, Yaqian Li, Anzhou Hou +6Vision-Language Models Efficient VLM Inference
Li Auto Inc.
PhySE: A Psychological Framework for Real-Time AR-LLM Social Engineering Attacks
Apr 25, 2026Tianlong Yu, Yang Yang, Ziyi Zhou +5Vision-Language Models Cybersecurity
1Hubei University · 2Apple Inc · 3Huazhong University of Science and Technology
SketchVLM: Vision language models can annotate images to explain thoughts and guide users
Apr 23, 2026Brandon Collins, Logan Bolton, Hung Huy Nguyen +3Vision-Language Models LLM-Assisted Annotation
Auburn University · Adobe Research

May 7, 2026, cs.CLJ/K move · Enter open · S save

The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation

Hoin Jung, Xiaoqian Wang

Elmore Family School of Electrical and Computer Engineering Purdue University West Lafayette, IN 47907

Multimodal RAG Multimodal Grounding Vision-Language Models Multimodal Robustness Multimodal Understanding

Open paper AskPDF arXiv

While Multimodal Large Language Models (MLLMs) are increasingly integrated with Retrieval-Augmented Generation (RAG) to mitigate hallucinations, the introduction of external documents can conceal severe failure modes at the instance level. We identify and formalize the phenomenon of recorruption, where the introduction of even perfectly accurate "oracle" context causes a capable model to abandon an initially correct prediction. Through a mechanistic diagnosis of internal attention matrices, we show that recorruption is driven by a two-fold attentional collapse: (1) visual blindness, characterized by the systemic suppression of visual attention mass (

M_{vis}

) and sharpness (

S_{vis}

), and (2) a structural positional bias that forces the model to prioritize boundary tokens over semantic relevance. Our analysis reveals an Illusion of Success, demonstrating that many seemingly correct RAG outcomes are merely positional coincidences where the model's textual copying bias happens to align with the ground-truth location. To address these vulnerabilities, we propose Bottleneck Attention Intervention for Recovery (BAIR), a parameter-free, inference-time framework that restores visual saliency and applies position-aware penalties to textual distractors. Across medical factuality, social fairness, and geospatial benchmarks, BAIR successfully restores multimodal grounding and improves diagnostic reliability without requiring model retraining or fine-tuning.

Momentum

Related topics

Latest papers 491