Vision-Language Models

Also known as VLM

Vision-language models study how systems can connect images or videos with language to recognize, describe, retrieve, and reason about visual content. Common approaches include contrastive image-text encoders such as CLIP, generative models that connect vision encoders to language models, and instruction tuning, sometimes combined with retrieval or tools. Researchers evaluate them on image-text retrieval, visual question answering, OCR and document understanding, visual grounding, and video benchmarks. A central challenge is that fluent answers do not guarantee visual grounding: hallucinations, reliance on textual shortcuts, weak compositional and temporal reasoning, and the costs of efficient deployment remain important concerns.

Latest papers

738 papers77 papers in the last four weeks, up 20% on the four weeks before. 0.4% of all new papers.

Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
Apr 16, 2026Danae Sánchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras +1CoT Faithfulness Vision-Language Models
University of Copenhagen, Department of Computer Science · University of Sheffield, School of Computer Science
AIM: Asymmetric Information Masking for Visual Question Answering Continual Learning
Apr 16, 2026Peifeng Zhang, Zice Qiu, Donghua Yu +4Visual Question Answering Vision-Language Models
Sun Yat-Sen University · National Supercomputing Center in Shenzhen · Tsinghua University
Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models
Apr 16, 2026Haoyi Sun, Xiaoxiao Wang, Ning Mao +5Cross-Modal Knowledge Distillation Vision-Language Models
Li Auto Inc.
SatBLIP: Context Understanding and Feature Identification from Satellite Imagery with Vision-Language Learning
Apr 15, 2026Xue Wu, Shengting Cao, Shenglin Li +1Remote Sensing Image Captioning Remote Sensing Image Understanding
The University of Alabama Tuscaloosa, Alabama, USA · Knox College Galesburg, Illinois, USA
BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation
Apr 15, 2026Baoyou Chen, Hanchen Xia, Peng Tu +5Vision-Language Models Diffusion VLMs
Shanghai Academy of AI for Science · Fudan University · Shanghai Innovation Institute +1
Why MLLMs Struggle to Determine Object Orientations
Apr 14, 2026Anju Gopinath, Nikhil Krishnaswamy, Bruce DraperRepresentation Geometry in VLMs Vision-Language Models
Department of Computer Science, Colorado State University, Fort Collins, CO, USA
MODIX: A Training-Free Multimodal Information-Driven Positional Index Scaling for Vision-Language Models
Apr 14, 2026Ruoxiang Huang, Zhen YuanVision-Language Models VLM Adaptation
Peking University, China
From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception
Apr 14, 2026Jilong Zhu, Yang FengVision-Language Models Fine-Grained Visual Perception
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · University of Chinese Academy of Sciences, Beijing, China
ViLL-E: Video LLM Embeddings for Retrieval
Apr 13, 2026Rohit Gupta, Jayakrishnan Unnikrishnan, Fan Fei +3Video-Language Models Vision-Language Models
Benchmarking Deflection and Hallucination in Large Vision-Language Models
Apr 13, 2026Nicholas Moratelli, Christopher Davis, Leonardo F. R. Ribeiro +2VLM Evaluation Vision-Language Models
University of Modena and Reggio Emilia · Amazon AGI · University of Cambridge
Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models
Apr 13, 2026Yvon Apedo, Martyna Poreba, Michal Szczepanski +1Vision-Language Models Efficient VLM Inference
Universit´e Paris-Saclay, Univ Evry, IBISC, 91020, Evry-Courcouronnes, France · Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Apr 13, 2026Imanol Miranda, Ander Salaberria, Eneko Agirre +1Compositional Reasoning in VLMs Vision-Language Models
HiTZ Center – Ixa, University of the Basque Country (UPV/EHU)
What Do Vision-Language Models Encode for Personalized Image Aesthetics Assessment?
Apr 13, 2026Koki Ryu, Hitomi YanakaPersonalized Image Aesthetics Assessment Vision-Language Models
The University of Tokyo · Riken · Tohoku University
Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Models
Apr 13, 2026Sameera Horawalavithana, Lauren Phillips, Ian Stewart +2Vision-Language Models VLM Reasoning
Pacific Northwest National Laboratory, Richland, WA 99352, USA
SVSR: A Self-Verification and Self-Rectification Paradigm for Multimodal Reasoning
Apr 11, 2026Zhe Qian, Nianbing Su, Zhonghua Wang +6Multimodal Reasoning Vision-Language Models
South China Agricultural University · Monash University · University of Glasgow +5
Counting to Four is still a Chore for VLMs
Apr 11, 2026Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van VoAttention Intervention in VLMs Vision-Language Models
MBZUAI
VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images
Apr 10, 2026Guanyu Zhou, Yida Yin, Wenhao Chai +3Visual Question Answering Synthetic Data Augmentation
Princeton University · New York University
UIPress: Bringing Optical Token Compression to UI-to-Code Generation
Apr 10, 2026Dasen Dai, Shuoqi Li, Ronghao Chen +3Token Compression Visual Token Compression
The Chinese University of Hong Kong China · Peking University China · University of Chinese Academy of Sciences China +2
Do Vision Language Models Need to Process Image Tokens?
Apr 10, 2026Sambit Ghosh, R. Venkatesh Babu, Chirag AgarwalVision-Language Models VLM Reasoning
IBM · Indian Institute of Science · University of Virginia
Visually-Guided Policy Optimization for Multimodal Reasoning
Apr 10, 2026Zengbin Wang, Feng Xiong, Liang Lin +5VLM Reasoning RL for VLMs
AMAP, Alibaba Group · SYSU · BUPT
Skill-Conditioned Visual Geolocation for Vision-Language Models
Apr 10, 2026Chenjie Yang, Yutian Jiang, Yutong Deng +1VLM Reasoning Vision-Language Models
Southwest Jiaotong University Chengdu, China · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China · Zhejiang University Hangzhou, China
Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Models
Apr 10, 2026Binesh Sadanandan, Vahid BehzadanMedical VLMs VLM Evaluation
SAIL Lab, University of New Haven, West Haven, CT, USA
Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Models
Apr 9, 2026Sumra Khan, Sagar Chhabriya, Aizan Zafar +5VLM Hallucination Mitigation Medical VLMs
Department of Computer Science, Salim Habib University, Karachi, Pakistan · Computer Science, Institute of Business Administration Sukkur, Pakistan · Center for Research in Computer Vision, University of Central Florida, USA +2
EXAONE 4.5 Technical Report
Apr 9, 2026Eunbi Choi, Kibong Choi, Sehyun Chun +55Vision-Language Models Large Vision-Language Models
Can Vision Language Models Judge Action Quality? An Empirical Evaluation
Apr 9, 2026Miguel Monte e Freitas, Rui Henriques, Ricardo Rei +1Action Quality Assessment Vision-Language Models
Sword Health · Instituto Superior T´ecnico, Universidade de Lisboa · INESC-ID
Small Vision-Language Models are Smart Compressors for Long Video Understanding
Apr 9, 2026Junjie Fei, Jun Chen, Zechun Liu +13Long-Video Understanding Video Token Compression
Meta AI · King Abdullah University of Science and Technology (KAUST)
AgriChain Visually Grounded Expert Verified Reasoning for Interpretable Agricultural Vision Language Models
Apr 9, 2026Hazza Mahmood, Yongqiang Yu, Rao AnwerCoT Supervision Plant Disease Classification
Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates
Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM
Apr 8, 2026Chengyue Wu, Shiyi Lan, Yonggan Fu +9Vision-Language Models Diffusion VLMs
The University of Hong Kong · NVIDIA · MBZUAI +1
ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding
Apr 8, 2026Xuanle Zhao, Xinyuan Cai, Xiang Cheng +2Vision-Language Models VLM Reasoning
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Is CLIP Cross-Eyed? Revealing and Mitigating Center Bias in the CLIP Family
Apr 7, 2026Oscar Chew, Hsiao-Ying Huang, Kunal Jain +3Contrastive Learning for VLMs Vision-Language Models
Texas A&M University · National Taiwan University · ASUS +1

Apr 16, 2026J/K move · Enter open · S save

Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models

Danae Sánchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras, Desmond Elliott

University of Copenhagen, Department of Computer Science · University of Sheffield, School of Computer Science

CoT Faithfulness Vision-Language Models VLM Interpretability VLM Reasoning Visually Grounded Reasoning Reasoning for VLMs

Open paper AskPDF arXiv

Recent advances in vision language models (VLMs) offer reasoning capabilities, yet how these unfold and integrate visual and textual information remains unclear. We analyze reasoning dynamics in 18 VLMs covering instruction-tuned and reasoning-trained models from two different model families. We track confidence over Chain-of-Thought (CoT), measure the corrective effect of reasoning, and evaluate the contribution of intermediate reasoning steps. We find that models are prone to answer inertia, in which early commitments to a prediction are reinforced, rather than revised during reasoning steps. While reasoning-trained models show stronger corrective behavior, their gains depend on modality conditions, from text-dominant to vision-only settings. Using controlled interventions with misleading textual cues, we show that models are consistently influenced by these cues even when visual evidence is sufficient, and assess whether this influence is recoverable from CoT. Although this influence can appear in the CoT, its detectability varies across models and depends on what is being monitored. Reasoning-trained models are more likely to explicitly refer to the cues, but their longer and fluent CoTs can still appear visually grounded while actually following textual cues, obscuring modality reliance. In contrast, instruction-tuned models refer to the cues less explicitly, but their shorter traces reveal inconsistencies with the visual input. Taken together, these findings indicate that CoT provides only a partial view of how different modalities drive VLM decisions, with important implications for the transparency and safety of multimodal systems.