Vision-Language Models

Also known as VLM

Vision-language models study how systems can connect images or videos with language to recognize, describe, retrieve, and reason about visual content. Common approaches include contrastive image-text encoders such as CLIP, generative models that connect vision encoders to language models, and instruction tuning, sometimes combined with retrieval or tools. Researchers evaluate them on image-text retrieval, visual question answering, OCR and document understanding, visual grounding, and video benchmarks. A central challenge is that fluent answers do not guarantee visual grounding: hallucinations, reliance on textual shortcuts, weak compositional and temporal reasoning, and the costs of efficient deployment remain important concerns.

Latest papers

738 papers77 papers in the last four weeks, up 20% on the four weeks before. 0.4% of all new papers.

A systematic evaluation of vision-language models for observational astronomical reasoning tasks
Apr 27, 2026Wenke Ren, Hengxiao Guo, Wenwen Zuo +1VLM Evaluation Vision-Language Models
Shanghai Astronomical Observatory, Chinese Academy of Sciences, Shanghai, China · Harvard University, Boston, Massachusetts, USA
PhysNote: Self-Knowledge Notes for Evolvable Physical Reasoning in Vision-Language Model
Apr 27, 2026Sinin Zhang, Yunfei Xie, Yuxuan Cheng +2Vision-Language Models VLM Reasoning
The Chinese University of Hong Kong, Shenzhen · Rice University · City University of Hong Kong +1
Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs
Apr 27, 2026Byeonggeuk Lim, JungMin Yun, Junehyoung Kwon +2Vision-Language Models Hallucination in Language Models
Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University · Department of Artificial Intelligence, Chung-Ang University
SycoPhantasy: Quantifying Sycophancy and Hallucination in Small Open Weight VLMs for Vision-Language Scoring of Fantasy Characters
Apr 27, 2026Arya Shah, Deepali Mishra, Chaklam SilpasuwanchaiVLM Evaluation Vision-Language Models
IIT Gandhinagar Gandhinagar, India · IIT Kanpur Kanpur, India · Asian Institute of Technology Bangkok, Thailand
SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
Apr 27, 2026Zi-Hao Bo, Yaqian Li, Anzhou Hou +6Vision-Language Models Efficient VLM Inference
Li Auto Inc.
PhySE: A Psychological Framework for Real-Time AR-LLM Social Engineering Attacks
Apr 25, 2026Tianlong Yu, Yang Yang, Ziyi Zhou +5Vision-Language Models Cybersecurity
1Hubei University · 2Apple Inc · 3Huazhong University of Science and Technology
SketchVLM: Vision language models can annotate images to explain thoughts and guide users
Apr 23, 2026Brandon Collins, Logan Bolton, Hung Huy Nguyen +3Vision-Language Models LLM-Assisted Annotation
Auburn University · Adobe Research
Source-Modality Monitoring in Vision-Language Models
Apr 23, 2026Etha Tianze Hua, Tian Yun, Ellie PavlickVision-Language Models Multimodal Understanding
Department of Computer Science, Brown University
R-CoV: Region-Aware Chain-of-Verification for Alleviating Object Hallucinations in LVLMs
Apr 22, 2026Jiahao Xie, Alessio Tonioni, Nathalie Rauschmayr +2Hallucination Detection in VLMs Vision-Language Models
Max Planck Institute for Informatics, SIC · VIA Research Center · Google
Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding
Apr 22, 2026Yubo Jiang, Yitong An, Xin Yang +7Vision-Language Models Hallucination in Language Models
School of Astronautics, Beihang University, Beijing 102206, China · Longcat Interaction Team, Meituan, Beijing 100102, China · Tianmushan Laboratory, Beihang University, Hangzhou 311115, China
Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation
Apr 22, 2026Xingyu Zhu, Junfeng Fang, Shuo Wang +4Vision-Language Models VLM Adaptation
MoE Key Lab of BIPC, University of Science and Technology of China · National University of Singapore
Detecting AI-Generated Content on Social Media with Multi-modal Language Models
Apr 21, 2026Chenyang Yang, Shen Yan, Yibo Yang +13Vision-Language Models AI-Generated Image Detection
Carnegie Mellon University · Meta
PLaMo 2.1-VL Technical Report
Apr 21, 2026Tommi Kerola, Yuya Masuda, Takashi Masuko +5Visual Question Answering Vision-Language Models
Preferred Networks, Inc. (PFN)
Disparities In Negation Understanding Across Languages In Vision-Language Models
Apr 21, 2026Charikleia Moraitaki, Sarah Pan, Skyler Pulling +3VLM Evaluation Vision-Language Models
Massachusetts Institute of Technology
ESsEN: Training Compact Discriminative Vision-Language Transformers in a Low-Resource Setting
Apr 20, 2026Clayton Fields, Casey KenningtonVision-Language Models Efficient ViTs
Department of Computer Science · Boise State University
Multilingual Training and Evaluation Resources for Vision-Language Models
Apr 20, 2026Daniela Baiamonte, Elena Fano, Matteo Gabburo +3VLM Evaluation Vision-Language Models
Aithlas
Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting
Apr 20, 2026Hyeonseo Jang, Hyuk Kwon, Kibok LeeClass-Incremental Learning Vision-Language Models
Yonsei University
Mitigating Multimodal Hallucination via Phase-wise Self-reward
Apr 20, 2026Yu Zhang, Chuyang Sun, Kehai Chen +3Vision-Language Models Hallucination in Language Models
Harbin Institute of Technology · Peng Cheng Laboratory · Shenzhen, China
RS-HyRe-R1: A Hybrid Reward Mechanism to Overcome Perceptual Inertia for Remote Sensing Images Understanding
Apr 19, 2026Gaozhi Zhou, Hu He, Peng Shen +8Vision-Language Models Visual Question Answering
School of Mechanical and Electrical Engineering, Central South University, Changsha 410083, China · School of Geosciences and Info-Physics, Central South University, Changsha 410083, China
More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage
Apr 19, 2026Wei HeVLM Evaluation Figurative Language Understanding
IDSAI, University of Exeter Exeter, UK
Towards Joint Quantization and Token Pruning of Vision-Language Models
Apr 19, 2026Xinqing Li, Xin He, Xindong Zhang +3Vision-Language Models VLM Quantization
VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · OPPO Research Institute +3
LLaTiSA: Towards Difficulty-Stratified Time Series Reasoning from Visual Perception to Semantics
Apr 19, 2026Yueyang Ding, HaoPeng Zhang, Rui Dai +4Vision-Language Models Time Series Reasoning
Amap, Alibaba Group
PestVL-Net: Enabling Multimodal Pest Learning via Fine-grained Vision-Language Interaction
Apr 19, 2026Xueheng Li, Tao Hu, Ke Cao +5Agricultural Image Analysis Vision-Language Models
Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences · University of Science and Technology of China · 3Zhongke Hefei Institute of Technology Innovation Engineering
Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability
Apr 19, 2026Lijie ZhouCross-Modal Learning VLM Robustness
School of Computer Science University of Nottingham Ningbo China Ningbo, China
DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models
Apr 18, 2026JiYang Wang, Jiawei Chen, Mengqi Xiao +3VLM Evaluation Hallucination Detection in VLMs
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy
Channel Attention-Guided Cross-Modal Knowledge Distillation for Referring Image Segmentation
Apr 18, 2026Chen YangCross-Modal Learning Cross-Modal Knowledge Distillation
School of Information Science and Technology, TaiShan University, Taian, China
Information Router for Mitigating Modality Dominance in Vision-Language Models
Apr 17, 2026Seulgi Kim, Mohit Prabhushankar, Ghassan AlRegibVLM Robustness Vision-Language Models
OLIVES at the Center for Signal and Information Processing CSIP, School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA
Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
Apr 17, 2026Yige Xu, Yongjie Wang, Zizhuo Wu +3Vision-Language Models VLM Adaptation
College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China
AstroVLM: Expert Multi-agent Collaborative Reasoning for Astronomical Imaging Quality Diagnosis
Apr 17, 2026Yaohui Han, Tianshuo Wang, Zixi Zhao +6Vision-Language Models VLM Reasoning
The Chinese University of Hong Kong · Central South University · Huawei Technologies Co., Ltd +1
VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models
Apr 16, 2026Huawei Ji, Yuanhao Sun, Yuan Jin +4Vision-Language Models Efficient VLM Inference
Shanghai Jiao Tong University, Shanghai, China · University of Edinburgh, Edinburgh, UK

Apr 27, 2026J/K move · Enter open · S save

A systematic evaluation of vision-language models for observational astronomical reasoning tasks

Wenke Ren, Hengxiao Guo, Wenwen Zuo, Xiaoman Zhang

Shanghai Astronomical Observatory, Chinese Academy of Sciences, Shanghai, China · Harvard University, Boston, Massachusetts, USA

VLM Evaluation Vision-Language Models Vision-Language Grounding VLM Reasoning

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly proposed as general-purpose tools for scientific data interpretation, yet their reliability on real astronomical observations across diverse modalities remains untested. We present AstroVLBench, a benchmark of over 4,100 expert-verified instances across five tasks spanning optical imaging, radio interferometry, multi-wavelength photometry, time-domain light curves, and optical spectroscopy. We systematically evaluate six state-of-the-art frontier models and find that performance is strongly modality-dependent and uncorrelated with general benchmark rankings. Mechanistic cross-model ablations reveal three concrete bottlenecks. First, prompt grounding exhibits marked model and task dependence: physical guidance yields more balanced classifications on optical imaging, whereas phenomenological prompts better navigate undersampled time-domain data. Second, presenting one-dimensional measurements as numerical tables rather than rendered plots consistently improves light-curve classification across all models by up to 23 percentage points, demonstrating that plot rendering can obscure physically relevant signal. Third, an expert audit of optical spectroscopy reveals a notable "right-answer-wrong-reason" phenomenon: across evaluated models, an average of 26% of correct predictions are accompanied by physically inconsistent or invalid reasoning chains, showing that accuracy alone is insufficient for trustworthy scientific deployment. AstroVLBench provides the first systematic, multi-modal baselines for VLMs in observational astronomy and identifies the representation, grounding, and reasoning bottlenecks that must be addressed before VLMs can be trusted as autonomous scientific interpreters.