Vision-Language Models

Also known as VLM

Vision-language models study how systems can connect images or videos with language to recognize, describe, retrieve, and reason about visual content. Common approaches include contrastive image-text encoders such as CLIP, generative models that connect vision encoders to language models, and instruction tuning, sometimes combined with retrieval or tools. Researchers evaluate them on image-text retrieval, visual question answering, OCR and document understanding, visual grounding, and video benchmarks. A central challenge is that fluent answers do not guarantee visual grounding: hallucinations, reliance on textual shortcuts, weak compositional and temporal reasoning, and the costs of efficient deployment remain important concerns.

Latest papers

738 papers77 papers in the last four weeks, up 20% on the four weeks before. 0.4% of all new papers.

VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding
Mar 7, 2026Xueqing Yu, Bohan Li, Yan Li +1Abstention in VLMs VLM Evaluation
Peking University · ByteDance
Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigation
Mar 7, 2026Xiaochen Yang, Hao Fang, Jiawei Kong +3VLM Hallucination Mitigation Vision-Language Models
Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology · Beijing Institute of Technology +1
Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders
Mar 6, 2026Boqiang Zhang, Lei Ke, Ruihan Yang +5Vision-Language Models Vision-Language Pretraining
GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training
Mar 6, 2026Yingtai Li, Shuai Ming, Qiuli Wang +13Medical Imaging Foundation Models Radiology VLMs
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui, 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, 215123, China · Department of Radiology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, USTC, Hefei, Anhui, 230001, China +7
Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement
Mar 6, 2026Yakov Pyotr ShkolnikovRepresentation Geometry in VLMs Vision-Language Models
Do 3D Large Language Models Really Understand 3D Spatial Relationships?
Mar 6, 2026Xianzheng Ma, Tao Sun, Shuai Chen +73D Spatial Reasoning 3D VQA
VGG, University of Oxford · AVL, University of Oxford · Stanford University +2
Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models
Mar 6, 2026Nicoleta-Nina Basoc, Adrian Cosma, Emilian RadoiVLM Robustness Vision-Language Models
National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania · IDSIA, The Dalle Molle Institute for Artificial Intelligence, Lugano, Switzerland
DeepSight: Bridging Depth Maps and Language with a Depth-Driven Multimodal Model
Mar 6, 2026Hao Yang, Hongbo Zhang, Yanyan Zhao +13D VLMs Vision-Language Models
Harbin Institute of Technology, Harbin, China
Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving
Mar 6, 2026Nikos Theodoridis, Reenu Mohandas, Ganesh Sistu +3Vision-Language Models VLMs for Autonomous Driving
Department of Electronic and Computer Engineering University of Limerick · Valeo Vision Systems
DARK: Diagonal-Anchored Repulsive Knowledge Distillation for Vision-Language Models under Extreme Compression
Mar 5, 2026Numan Saeed, Asif Hanif, Fadillah Adamsyah Maani +2VLM Distillation Knowledge Distillation
Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
VisionPangu: A Compact and Fine-Grained Multimodal Assistant with 1.7B Parameters
Mar 5, 2026Jiaxin Fan, Wenpo SongVision-Language Models Multimodal Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University
Differentially Private Multimodal In-Context Learning
Mar 5, 2026Ivoline C. Ngong, Zarreen Reza, Joseph P. NearPrivacy Protection in VLMs Multimodal ICL
University of Vermont, Burlington, VT, USA · Independent Researcher
When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models
Mar 4, 2026Qianpu Chen, Derya Soydaner, Rob SaundersVLM Robustness VLM Evaluation
LIACS (Leiden Institute of Advanced Computer Science), Leiden University, Leiden, The Netherlands
From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoning
Mar 4, 2026Ruilin Luo, Chufan Shi, Yizhen Zhang +10Multimodal Reasoning VLM Reasoning
1Tsinghua University · 3Qwen Team, Alibaba Group · University of South California +3
Benchmarking Compact VLMs for Clip-Level Surveillance Anomaly Detection Under Weak Supervision
Mar 3, 2026Kirill Borodin, Kirill Kondrashov, Nikita Vasiliev +4Vision-Language Models Weakly Supervised Video Anomaly Detection
Faculty of Information Technology, Moscow Technical University of Communication and Informatics, Moscow 111024, Russia
Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models
Mar 3, 2026Haruto Yoshida, Keito Kudo, Yoichi Aoki +4Vision-Language Models Large Vision-Language Models
Tohoku University · Human Informatics Labs., NTT, Inc.
OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning
Mar 3, 2026Zhengwei Yang, Andi Long, Hao Li +3Vision-Language Models Large Vision-Language Models
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University · Harbin Institute of Technology
Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMs
Mar 3, 2026Zhikang Xu, Qianqian Xu, Zitai Wang +4OOD Detection Vision-Language Models
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences +2
VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning
Mar 1, 2026Mingkang Dong, Hongyi Cai, Jie Li +4Vision-Language Models Data Selection
SJTU · Universiti Malaya · CMU +3
GameVerse: Can Vision-Language Models Learn from Video-based Reflection?
Mar 1, 2026Kuan Zhang, Dongchen Liu, Qiyue Zhao +5Vision-Language Models VLM Evaluation
College of AI, Tsinghua University, China.
ClinCoT: Clinical-Aware Visual Chain-of-Thought for Medical Vision Language Models
Mar 1, 2026Xiwei Liu, Yulong Li, Xinlin Zhuang +5Vision-Language Models Medical VLMs
Mohamed bin Zayed University of Artificial Intelligence · East China Normal University
Unified Vision-Language Modeling via Concept Space Alignment
Mar 1, 2026Yifu Qiu, Paul-Ambroise Duquenne, Holger SchwenkVision-Language Models Vision-Language Alignment
University of Edinburgh · FAIR at Meta
TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications
Feb 28, 2026Feibo Jiang, Siwei Tu, Li Dong +5Joint Source-Channel Coding Vision-Language Models
Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University, Changsha, China. · School of Information Science and Engineering, Hunan Normal University, Changsha, China. · Changsha Social Laboratory of Artificial Intelligence, Hunan University of Technology and Business, Changsha, China. +5
U-VLM: Hierarchical Vision Language Modeling for Report Generation
Feb 28, 2026Pengcheng Shi, Minghui Zhang, Kehan Song +3Radiology Report Generation Vision-Language Models
Medical Image Insights Co. Ltd., Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models
Feb 28, 2026April FuCross-Layer Attention Self-Correction in VLMs
Independent Researcher
CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answering
Feb 27, 2026Yuyang Hong, Jiaqi Gu, Yujin Lou +7Knowledge-Based VQA Knowledge Conflicts in Language Models
School of Artificial Intelligence, UCAS · MAIS, Institute of Automation · Alibaba Cloud Computing
Half-Truths Break Similarity-Based Retrieval
Feb 27, 2026Bora Kargi, Arnas Uselis, Seong Joon OhContrastive Learning for VLMs Vision-Language Models
University of Tübingen, Tübingen AI Center · Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)
Cytoarchitecture in Words: Weakly Supervised Vision-Language Modeling for Human Brain Microscopy
Feb 26, 2026Matthew Sutton, Katrin Amunts, Timo Dickscheid +1Medical VLMs Vision-Language Models
Institute of Neuroscience and Medicine (INM-1), Research Centre Jülich, Jülich, Germany · Helmholtz AI, Research Centre Jülich, Jülich, Germany · C´ecile & Oskar Vogt Institute for Brain Research, University Hospital Düsseldorf, Düsseldorf, Germany +1
Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models
Feb 26, 2026Jonathan Steinberg, Oren GalVision-Language Models VLM Interpretability
Swarms & AI Lab (SAIL), University of Haifa
Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentation
Feb 24, 2026Asim Unmesh, Kaki Ramesh, Mayank Patel +2Temporal Action Segmentation Zero-Shot Learning
Purdue University, USA · Birla Institute of Technology and Science (BITS) Hyderabad, India

Mar 7, 2026J/K move · Enter open · S save

VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding

Xueqing Yu, Bohan Li, Yan Li, Zhenheng Yang

Peking University · ByteDance

Abstention in VLMs VLM Evaluation Video-Language Model Evaluation Vision-Language Models Long-Video Understanding

Open paper AskPDF arXiv

Recent Vision-Language Models (VLMs) have made remarkable progress in multimodal understanding tasks, yet their evaluation on long video understanding remains unreliable. Due to limited frame inputs, key frames necessary for answering the question may be missing from the model's input. However, models that truthfully refuse to answer under such uncertainty are marked as incorrect, while those that guess may coincidentally produce the correct answer and thus obtain deceptively higher accuracy, leading to misleading evaluation results and encouraging models to guess rather than respond honestly. To address this issue, we introduce VirtueBench, a benchmark explicitly designed to assess model trustworthiness under uncertainty. VirtueBench constructs multiple frame-sampling levels for each video and provides ground truths that distinguish between answerable and unanswerable cases. Evaluations on 25 open-source and commercial VLMs reveal distinct refusal behaviors across different model families, with refusal accuracy ranging from over 70% in the best models to nearly 0% in the worst. Moreover, most models exhibit a substantial drop in refusal when the prompt does not explicitly require them to do so. These findings highlight the need for developing trustworthy VLMs for multimodal understanding, guided by benchmarks and leaderboards that emphasize reliability and trustworthiness.